Data Engineer
Wait ā Check First
- Check if your CV is ATS-ready for Experts Plus Recruitment Services
- Get AI-rewritten bullet points
- Download Gulf-ready CV
60 seconds. $3.99 one-time.
As a Data Engineer, your main goal is to build and maintain the systems that process and store data for the Events & Exhibitions ecosystem. You will take scattered, vendor-specific data (from registration systems, apps, marketing tools) and transform it into a unified, AI-ready dataset using a Medallion Architecture on Azure.
Think of it as organizing raw data into a structured pipeline thatās ready for analysis and machine learning.
Requirements
1. Data Ingestion & API Integration (Bronze Layer)
ā¢ Build and manage robust ETL/ELT pipelines using Azure Data Factory to ingest data from 3rd-party vendors (REST APIs, Webhooks, SFTP).
ā¢ Ensure raw data is landed securely in Azure Data Lake Gen2 (Bronze Layer) without data loss.
ā¢ Implement error-handling and logging to monitor the health of real-time and batch ingestion jobs.2. Transformation & Modeling (Silver & Gold Layers)
ā¢ Utilize PySpark (Azure Databricks/Synapse) and SQL to clean, deduplicate, and standardize data in the Silver Layer.
ā¢ Execute Identity Resolution logic to stitch together visitor and exhibitor profiles from multiple touchpoints into a "Golden Record."
ā¢ Develop optimized data sets in the Gold Layer for high-performance reporting and predictive AI models.3. Infrastructure & Performance Optimization
ā¢ Optimize SQL queries and Spark jobs to reduce Azure compute costs and minimize data latency.
ā¢ Maintain the Data Dictionary and technical documentation to ensure the "Engine Room" logic is transparent and scalable.
ā¢ Implement data masking and security protocols to ensure GDPR and internal compliance.4. Business Enablement
ā¢ Support the Senior Data Manager in building the Semantic Layer that feeds our Power BI "Data Window."
ā¢ Collaborate with the Events Tech team to troubleshoot data discrepancies between front-end apps and back-end tables.Technical Requirements
ā¢ Experience: 3ā5 years in Data Engineering with a focus on Cloud environments.
ā¢ Core Azure Stack: Proven expertise in Azure Data Factory, Azure Synapse Analytics, and Data Lake Gen2.
ā¢ Coding: High proficiency in SQL (complex joins/optimizations) and Python/PySpark.
ā¢ Architectural Knowledge: Practical experience with the Medallion Architecture (Bronze/Silver/Gold).
ā¢ Integration: Strong experience working with REST APIs and JSON/XML data formats.Benefits
Requirements
- ā¢3-5 years in Data Engineering with a focus on Cloud environments
- ā¢Proven expertise in Azure Data Factory, Azure Synapse Analytics, and Data Lake Gen2
- ā¢High proficiency in SQL and Python/PySpark
- ā¢Practical experience with the Medallion Architecture
- ā¢Strong experience working with REST APIs and JSON/XML data formats
- ā¢Experience building and managing robust ETL/ELT pipelines
- ā¢Implement error-handling and logging for ingestion jobs
- ā¢Optimize SQL queries and Spark jobs
Nice to Have
- ā¢Experience with Salesforce data
- ā¢Experience with Events Exhibitions data
- ā¢Familiarity with Power BI
Responsibilities
- ā¢Build and manage ETL/ELT pipelines using Azure Data Factory
- ā¢Clean, deduplicate, and standardize data using PySpark and SQL
- ā¢Execute Identity Resolution logic
- ā¢Develop optimized data sets in the Gold Layer
- ā¢Optimize SQL queries and Spark jobs for cost and latency
- ā¢Maintain Data Dictionary and technical documentation
- ā¢Implement data masking and security protocols
- ā¢Support building the Semantic Layer for Power BI
Related Jobs
- Check if your CV is ATS-ready for Experts Plus Recruitment Services
- Get AI-rewritten bullet points
- Download Gulf-ready CV
60 seconds. $3.99 one-time.