Data Engineer

Azure Databricks / Synapse Data Engineer

Job Summary

Seeking an experienced Data Engineer to develop and optimize data pipelines, lakehouse solutions, and data warehouse workloads using Azure Databricks and Azure Synapse Analytics. The role focuses on batch and streaming data integration, data modeling, orchestration, performance tuning, and secure data governance for sensitive health data in compliance with HIPAA and agency policies.

Key Responsibilities

• Build and maintain batch and streaming pipelines using Azure Databricks, PySpark, Spark SQL, and Delta Lake.

• Implement medallion architecture (Bronze/Silver/Gold) and incremental data ingestion.

• Develop and optimize data models, SQL queries, and warehouse workloads in Azure Synapse Analytics.

• Orchestrate workflows using Azure Data Factory and Databricks Workflows.

• Implement data governance, security, and access controls using Unity Catalog, Entra ID, and Key Vault.

• Manage CI/CD, source control, monitoring, troubleshooting, and pipeline performance optimization.

• Collaborate with architects, analysts, data scientists, and project managers on requirements and technical solutions.

• Maintain technical documentation and support data quality, governance, and operational improvements.

Required Skills & Qualifications

• 4+ years of relevant hands-on data engineering experience or equivalent education and experience.

• Strong experience with PySpark, Spark SQL, Delta Lake, Databricks notebooks, and medallion architecture.

• Hands-on experience with Unity Catalog, Delta Lake transactions, schema evolution, and Spark performance tuning.

• Experience with Azure Synapse Analytics, MPP architecture, dimensional modeling, star schemas, SCD Type 1/2, and SQL optimization.

• Proficiency in Azure Data Factory, Databricks Workflows, ADLS Gen2, and pipeline orchestration.

• Experience with Azure DevOps/GitHub, CI/CD, managed identities, security controls, and Azure monitoring tools.

• Knowledge of Microsoft lakehouse patterns and Synapse–Databricks integration.

• Strong written and verbal communication skills.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional.

• Microsoft Certified: Azure Data Engineer Associate (DP-203), if already held.

• Experience with Power BI connectivity to Synapse, including Import vs. DirectQuery and dashboard performance optimization.