Azure Data Engineer

ob Summary

We are seeking an experienced Azure Data Engineer to design, develop, and maintain scalable data pipelines and modern data platforms on Microsoft Azure. The ideal candidate will have strong expertise in Azure data services, ETL/ELT development, SQL, Spark, and cloud-based data engineering solutions.

Key Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using Azure data services.
• Build and maintain enterprise data lakes and data warehouses on Azure.
• Develop data ingestion frameworks from structured, semi-structured, and unstructured data sources.
• Work with business analysts, data scientists, and application teams to understand data requirements.
• Optimize SQL queries and Spark jobs for performance and scalability.
• Implement data quality, governance, security, and monitoring processes.
• Automate deployment using Azure DevOps and CI/CD pipelines.
• Monitor production data pipelines and troubleshoot performance issues.
• Ensure compliance with data governance and security standards.
• Participate in code reviews and implement best practices.

Required Skills

• 7+ years of experience in Data Engineering.
• Strong experience with Microsoft Azure.
• Hands-on experience with:
  • Azure Data Factory (ADF)
  • Azure Databricks
  • Azure Synapse Analytics
  • Azure Data Lake Storage (ADLS Gen2)
  • Azure SQL Database
  • Azure Blob Storage
• Strong programming experience in Python and PySpark.
• Advanced SQL development and query optimization.
• Experience building ETL/ELT pipelines.
• Knowledge of Spark optimization techniques.
• Experience with Delta Lake.
• Familiarity with Azure DevOps and CI/CD pipelines.
• Experience with Git version control.
• Understanding of data modeling (Star Schema, Snowflake Schema).
• Strong analytical and problem-solving skills.

Preferred Skills

• Experience with Microsoft Fabric.
• Knowledge of Apache Kafka or Event Hubs.
• Experience with Snowflake or Databricks Lakehouse.
• Experience working with Power BI.
• Azure certifications are preferred.
• Experience in Agile/Scrum environments.

Technical Skills

• Cloud: Microsoft Azure
• Data Integration: Azure Data Factory (ADF)
• Processing: Azure Databricks, Apache Spark, PySpark
• Storage: ADLS Gen2, Blob Storage
• Database: Azure SQL, SQL Server
• Analytics: Azure Synapse Analytics
• Programming: Python, SQL
• Version Control: Git
• CI/CD: Azure DevOps
• Data Formats: Parquet, Delta Lake, JSON, CSV
• Methodologies: Agile, Scrum