Senior Azure Data Engineer

Senior Azure Data Engineer

Location: Atlanta, GA (Open for Remote)

Type: [Full-time / Contract] | Start: Immediate

We're looking for a hands-on Senior Azure Data Engineer to build and run production data pipelines on Azure. This role focuses on building complex data pipelines using Azure Data Factory and Azure Databricks, along with data validation using Python, PySpark, Pandas, and Great Expectations.

What You'll Do:

• Analyze incoming data files (CSV, JSON, Parquet) and their schemas to plan ingestion and transformation.
• Design and build lakehouse-style Medallion (Bronze, Silver, Gold) data pipelines, using Azure Data Factory for data movement and Azure Databricks for complex data processing.
• Build and maintain complex, multi-step pipelines in Azure Data Factory.
• Use ADF to trigger and manage Azure Databricks Jobs, including passing parameters, tracking job status, and handling failures and retries.
• Write data transformation and validation logic in Python and PySpark.Set up automated data quality checks using Great Expectations and Pandas, and flag bad data before it reaches downstream users.
• Troubleshoot and tune pipelines in production.

Core Skills Required-

1. 7+ years in data engineering, with recent senior-level work.
2. Strong hands-on experience with Azure Data Factory and its integration with Azure Databricks Jobs.Strong Python, PySpark, and SQL skills.
3. Hands-on experience building data validation frameworks with
4. Great Expectations and Pandas.
5. Solid cloud data engineering background. Azure preferred; AWS or Google Cloud Platform considered.

Big Plus-

1. Local to Atlanta, GA / Open for remote too
2. Experience on State or Federal government projects.
3. Azure Data Engineer (DP-203) or Databricks certification.