Databricks Data Engineer with Azure

Position: Databricks Data Engineer with Azure
Location: Houston, TX (4 days hybrid onsite)

Job Summary:
We are looking for an experienced Azure + Databricks Data Engineer to design, develop, and maintain scalable data pipelines and data solutions using Microsoft Azure and Databricks. The ideal candidate should have strong hands-on experience with PySpark, SQL, Azure Data Factory, and Delta Lake.

Key Responsibilities:

• Design and develop scalable ETL/ELT data pipelines using Azure and Databricks.
• Build data processing solutions using Databricks, Apache Spark, and PySpark.
• Develop and optimize SQL queries and data transformations.
• Create and manage pipelines using Azure Data Factory (ADF).
• Work with Azure Data Lake Storage (ADLS/ADLS Gen2) for data ingestion and storage.
• Implement data solutions using Delta Lake and Databricks.
• Perform data integration from multiple sources and ensure data quality and reliability.
• Optimize Spark jobs, Databricks notebooks, and data pipelines for performance.
• Work with Unity Catalog for data governance and access management.
• Support deployment and CI/CD processes using Azure DevOps.
• Troubleshoot data pipeline failures and resolve production issues.
• Collaborate with data architects, analysts, developers, and business teams.

Required Skills:

• Strong experience as a Data Engineer with Azure and Databricks.
• Hands-on experience with Databricks, Apache Spark, and PySpark.
• Strong SQL skills.
• Experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage (ADLS/ADLS Gen2).
• Strong understanding of Delta Lake and data lake architecture.
• Experience developing ETL/ELT pipelines.
• Knowledge of Databricks Unity Catalog and data governance.
• Experience with Azure DevOps and CI/CD.
• Good understanding of data modeling, data integration, and data quality.

Preferred Skills:

• Experience with Terraform or Infrastructure as Code.
• Knowledge of Azure Event Hubs/Kafka.
• Experience with Delta Live Tables (DLT).
• Familiarity with cloud security and Azure IAM.
• Experience working in Agile development environments.