AVP – Data Engineering Lead

JOB PURPOSE
The AVP – Data Engineer (AI Architect) is a hands-on practitioner responsible for architecting, building, and scaling the organization’s data engineering and AI platforms. This role combines deep technical execution with architectural ownership to deliver secure, scalable, and production-grade data pipelines and AI solutions, including Generative AI and LLM-powered systems.
You will actively design and implement data and AI architectures, write production-quality code, prototype new capabilities, and work closely with data engineers, data scientists, and product teams to operationalize analytics and AI use cases at enterprise scale.
ROLES AND RESPONSIBILITIES
Architect, build, and optimize scalable data pipelines using Python, Spark, and Databricks for batch, streaming, and real-time workloads.
Design, develop, and maintain ELT/ETL pipelines, data transformations, and feature engineering workflows.
Design, prototype, test, and productionize AI and Generative AI solutions, moving beyond PoCs to enterprise-ready deployments.
Build and deploy cloud-native data and AI solutions on Microsoft Azure, leveraging services such as Azure OpenAI, Azure Databricks, AKS, App Services, Azure Functions, and Azure Storage.
Develop containerized data and AI services using Docker and Kubernetes, following DevOps and CI/CD best practices.
Design and implement secure APIs and microservices to expose data and AI capabilities to downstream applications.
Implement monitoring and observability for data pipelines and AI systems, including latency, data quality, model performance, drift, and cost metrics.
Optimize cloud and AI workloads for performance, scalability, and cost efficiency.
Actively contribute code, perform design and code reviews, and troubleshoot complex production issues across data and AI systems.
Serve as a hands-on technical mentor, leading by example within data engineering and AI teams.
Influence technical standards and long-term architecture while remaining deeply involved in day-to-day delivery.
RELATED YEARS OF EXPERIENCE
7+ years of progressive, hands-on experience in data engineering, AI engineering, or cloud platform architecture
Proven experience architecting and operating enterprise-scale data platforms and AI solutions, from ingestion to consumption, with high availability and security.
Demonstrated success taking AI and Generative AI solutions from experimentation to production, including scaling, monitoring, and continuous improvement.
Strong experience with lakehouse architectures and modern data platform patterns
Hands-on expertise with relational databases (PostgreSQL, MySQL, SQL Server) and NoSQL / distributed data stores (MongoDB, Cassandra, Cosmos DB).
Experience with cloud-native data platforms such as Azure Synapse, Snowflake, or BigQuery, with a strong preference for Azure-based ecosystems.
Practical experience designing real-time and event-driven architectures, using Kafka, Spark Structured Streaming, Event Hubs, or Flink.
FIELD OF EXPERIENCE
Data Engineering, AI Architect
TECHNICAL AND INTERPERSONAL SKILLS
Expert-level Python for data engineering, AI pipelines, and backend services.
Strong practical experience with Databricks and Azure services, including: Azure OpenAI, Azure Kubernetes Service, Azure App Services & Functions, Azure API Management etc.
Familiarity with vector databases and search platforms (Azure AI Search, Pinecone, FAISS etc.).
Experience with data orchestration and workflow management (Airflow, Azure Data Factory, N8N).
Strong understanding of CI/CD, containerization and cloud-native engineering (Docker, Kubernetes)
Ability to move seamlessly between architecture design and hands-on implementation.
Strong communication skills and ability to explain complex systems to non-technical stakeholders.
QUALIFICATION
Bachelor’s degree in Computer Science, Data Engineering, AI, or a related field
Master’s degree and/or cloud certifications (Azure Solutions Architect, Azure Data Engineer, Azure AI Engineer, Databricks) are a plus.