Databricks Data Engineer

Job Title: Databricks Data Engineer (Mid-Level)

Position Summary
We are seeking two mid-level Databricks Data Engineers to design, build, and optimize scalable data pipelines and data solutions on the Databricks platform. The role involves developing ETL workflows with PySpark and SQL, managing Delta Lake architectures, and working within Azure or AWS cloud environments.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines in Databricks using PySpark and SQL.
• Build and manage Delta Lake tables, including schema evolution, ACID transactions, time travel, and data versioning.
• Optimize Spark jobs for performance and cost through partitioning, caching, cluster configuration, and query tuning.
• Ingest data from structured and unstructured sources, including APIs, databases, and cloud storage.
• Implement data quality checks, validation rules, and error handling within pipelines.
• Work with cloud services on Azure (Data Factory, ADLS Gen2, Synapse) or AWS (S3, Glue, Redshift, Lambda).
• Implement medallion architecture (Bronze, Silver, Gold layers).
• Schedule and orchestrate workflows using Databricks Workflows, Airflow, or Azure Data Factory.
• Collaborate with data analysts, data scientists, and business stakeholders to deliver data requirements.
• Maintain documentation for pipelines, data models, and processes.
• Follow version control and CI/CD practices for data engineering code.

Required Qualifications

• 3–6 years of experience in data engineering.
• 2+ years of hands-on experience with Databricks.
• Strong proficiency in PySpark and Spark SQL.
• Hands-on experience with Delta Lake.
• Advanced SQL skills, including complex joins, window functions, and query optimization.
• Experience building and maintaining production ETL pipelines.
• Experience with at least one cloud platform: Azure or AWS.
• Working knowledge of Python.
• Experience with Git and CI/CD tools (Azure DevOps, GitHub Actions, or Jenkins).
• Understanding of data modeling concepts (star schema, dimensional modeling).

Preferred Qualifications

• Databricks certification (Data Engineer Associate or Professional).
• Azure (DP-203) or AWS (Data Analytics / Data Engineer) certification.
• Experience with Unity Catalog for data governance.
• Experience with streaming data (Structured Streaming, Kafka, Event Hubs, or Kinesis).
• Experience with Delta Live Tables.
• Familiarity with Terraform or other infrastructure-as-code tools.