Azure Data Engineer

Job Title: Azure Data Engineer
Location: Weehawken  new Jersey

Employment Type: Contract Long Term

W2 Only

Job Summary

We are looking for an experienced Azure Data Engineer to design, develop, implement, and maintain scalable data pipelines and data integration solutions using Microsoft Azure services. The ideal candidate should have strong expertise in data engineering, ETL/ELT processes, SQL, Python, and Azure data services to support business intelligence, analytics, and reporting requirements.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF).
• Build and optimize ETL/ELT workflows to ingest, transform, and process structured and unstructured data.
• Work with Azure Databricks, Apache Spark, and PySpark for large-scale data processing.
• Develop complex SQL queries, stored procedures, views, and performance-optimized database solutions.
• Work with Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Synapse Analytics, and other Azure data services.
• Implement data quality checks, data validation, error handling, and monitoring mechanisms.
• Develop batch and, where required, real-time data processing solutions.
• Integrate data from multiple sources, including databases, APIs, files, and cloud applications.
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
• Implement data security, access controls, and governance best practices.
• Collaborate with data architects, data analysts, BI developers, and business stakeholders.
• Participate in code reviews, testing, deployment, documentation, and production support.
• Use Git and CI/CD pipelines to manage version control and deployments.

Required Skills and Qualifications

• 3+ years of experience in data engineering or a related role.
• Strong hands-on experience with Microsoft Azure data services.
• Good experience with Azure Data Factory (ADF), including pipelines, linked services, datasets, triggers, and integration runtimes.
• Strong knowledge of SQL, query optimization, joins, aggregations, and data modeling.
• Experience with Python and PySpark for data transformation and processing.
• Hands-on experience with Azure Data Lake Storage Gen2.
• Experience with Azure Databricks and Apache Spark.
• Knowledge of ETL/ELT concepts, data warehousing, and dimensional modeling.
• Understanding of incremental data loading, CDC, and pipeline orchestration.
• Experience with Git, Azure DevOps, or similar version control and CI/CD tools.
• Strong analytical, problem-solving, and communication skills.

Preferred Skills

• Experience with Azure Synapse Analytics or Microsoft Fabric.
• Knowledge of Delta Lake and Lakehouse architecture.
• Experience with streaming technologies such as Azure Event Hubs or Kafka.
• Familiarity with data governance, Microsoft Purview, and data security.
• Experience with Power BI data integration and reporting requirements.
• Knowledge of Infrastructure as Code and automated deployment practices.
• Microsoft Azure Data Engineer certification, such as DP-203, or equivalent practical experience.

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent professional experience.

Success Criteria

• Reliable and efficient data pipelines.
• High-quality, validated, and consistent data.
• Improved pipeline performance and reduced processing costs.
• Secure and well-documented data solutions.
• Timely delivery of data for analytics and business reporting.