Azure Databricks Engineer

Azure Databricks Engineer

Location: Newtown Sqaure, PA

W2 POSITION

Job Summary

We are looking for an experienced Azure Databricks Engineer to design, develop, and maintain scalable data engineering solutions on Microsoft Azure. The ideal candidate will have strong hands-on experience with Azure Databricks, Apache Spark, Python/SQL, and modern data lake architectures.

Key Responsibilities:

• Design and develop scalable data pipelines using Azure Databricks, Apache Spark, Python, and SQL.
• Build and optimize ETL/ELT pipelines for batch and real-time data processing.
• Develop data solutions using Azure Data Lake Storage (ADLS Gen2) and Delta Lake.
• Implement Delta Lake tables, partitioning, schema evolution, and performance optimization.
• Develop notebooks, workflows, jobs, and reusable frameworks in Azure Databricks.
• Integrate Databricks with Azure services such as Azure Data Factory, Azure Key Vault, Azure Synapse Analytics, and Azure Event Hubs.
• Perform data transformation, cleansing, validation, and quality checks.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Implement CI/CD pipelines using Azure DevOps and Git.
• Monitor production pipelines, troubleshoot failures, and resolve data-quality issues.
• Implement security, access control, and governance following enterprise standards.
• Collaborate with data architects, analysts, developers, and business stakeholders.

Required Skills:

• 10 15+ years of experience in data engineering or a related field.
• Strong hands-on experience with Azure Databricks.
• Strong knowledge of Apache Spark / PySpark.
• Proficiency in Python and SQL.
• Experience with Azure Data Lake Storage Gen2.
• Strong understanding of Delta Lake and lakehouse architecture.
• Experience with Azure Data Factory (ADF).
• Experience with Git and Azure DevOps CI/CD.
• Good understanding of data warehousing and ETL/ELT concepts.
• Experience troubleshooting and optimizing distributed data-processing workloads.

Preferred Skills:

• Experience with Azure Synapse Analytics.
• Knowledge of Unity Catalog and Databricks governance.
• Experience with streaming technologies such as Event Hubs, Kafka, or Structured Streaming.
• Experience with Terraform or infrastructure-as-code.
• Knowledge of data modeling and dimensional modeling.
• Experience implementing data quality and observability frameworks.
• Familiarity with Power BI and downstream analytics platforms.