Azure DataBricks Engineer

Azure Databricks Engineer

Location: Atlanta, GA (Onsite)
Employment Type: Full-Time
Experience Required: 6+ Years Overall | 3+ Years Azure Databricks Experience

About the Role

We are seeking a highly skilled Senior Azure Databricks Engineer to join our Data & Analytics team in Atlanta. The ideal candidate will have extensive experience designing, developing, and optimizing enterprise-scale data solutions using Azure Databricks, Apache Spark, and Microsoft Azure services.

This role will be responsible for building scalable data pipelines, implementing Lakehouse architectures, enabling real-time and batch data processing, and driving best practices for performance, governance, and security.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and support ETL/ELT frameworks for large-scale structured and unstructured data.
• Develop batch and real-time data processing solutions.
• Implement and maintain Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Integrate data solutions with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hub, and other Azure services.
• Optimize Spark workloads for performance, reliability, and cost efficiency.
• Implement data quality checks, monitoring, and alerting processes.
• Collaborate with business stakeholders, architects, analysts, and data scientists to deliver data solutions.
• Establish and enforce data governance, security, and compliance standards.
• Support CI/CD pipelines and automated deployment processes.
• Troubleshoot and resolve production issues related to data pipelines and Databricks workloads.
• Document technical solutions, architectural decisions, and operational procedures.

Required Qualifications

• Bachelor’s degree in computer science, Information Technology, Engineering, or a related field.
• 6+ years of experience in Data Engineering.
• 3+ years of hands-on experience with Azure Databricks.
• Strong experience with:
  • Azure Databricks
  • Apache Spark
  • PySpark
  • Python
  • SQL
  • Delta Lake
• Experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Strong understanding of Data Warehousing, Data Lakes, and Lakehouse architectures.
• Experience working with large-scale datasets and distributed computing environments.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration abilities.

Preferred Qualifications

• Experience with Azure Synapse Analytics.
• Experience working with Event Hub or Kafka.
• Knowledge of Unity Catalog and data governance frameworks.
• Experience with Delta Live Tables (DLT).
• Experience with Terraform, ARM Templates, or Infrastructure as Code (IaC).
• Experience with Azure DevOps or GitHub Actions.
• Knowledge of Power BI and modern analytics platforms.
• Cloud certifications such as:
  • Microsoft Azure Data Engineer Associate (DP-203)
  • Databricks Certified Data Engineer Associate/Professional

Technical Skills

Required

• Azure Databricks
• Apache Spark
• PySpark
• Python
• SQL
• Azure Data Factory
• ADLS Gen2
• Delta Lake
• Git

What We're Looking For

The ideal candidate has hands-on experience building modern cloud-based data platforms and can demonstrate expertise in:

• Lakehouse Architecture
• Spark Performance Tuning
• Enterprise Data Engineering
• Data Pipeline Development
• Real-Time Data Processing
• Data Governance and Security
• CDC (Change Data Capture) Implementations
• Production Support and Troubleshooting
• Azure Cloud Data Services
• Cost Optimization and Scalability

Why Join Us?

• Work on enterprise-scale cloud data modernization initiatives.
• Collaborate with experienced data and cloud professionals.
• Exposure to cutting-edge Azure and Databricks technologies.
• Opportunity to drive impactful business solutions through data.
• Long-term growth and career advancement opportunities.