DataBricks Lead Engineer

Job Title: Lead DataBricks Engineer

Job Summary: We are looking for an experienced Lead DataBricks Engineer to lead the design, development, and implementation of scalable data engineering solutions using Databricks and modern cloud data technologies. The ideal candidate will have strong hands-on experience with Databricks, Apache Spark, PySpark, SQL, Delta Lake, and cloud platforms, along with the ability to provide technical leadership to the data engineering team.

Key Responsibilities:

• Lead the design and development of scalable data pipelines using Databricks and Apache Spark.
• Develop and optimize ETL/ELT pipelines using PySpark and SQL.
• Design and implement data solutions using Delta Lake and Databricks Lakehouse architecture.
• Provide technical leadership, code reviews, and mentoring to data engineers.
• Work closely with architects, business stakeholders, and cross-functional teams to define data requirements.
• Optimize Spark jobs, queries, and data pipelines for performance and cost.
• Implement data quality, monitoring, security, and governance best practices.
• Integrate Databricks with cloud data services and enterprise data platforms.
• Troubleshoot production data pipeline issues and drive root-cause resolution.
• Participate in architecture discussions and contribute to technical standards and best practices.

Required Skills:

• 10+ years of experience in Data Engineering or related roles.
• 4+ years of hands-on experience with Databricks.
• Strong experience with Apache Spark / PySpark.
• Advanced SQL skills.
• Strong understanding of Delta Lake and Lakehouse architecture.
• Experience with AWS, Azure, or Google Cloud Platform.
• Experience building enterprise-scale ETL/ELT data pipelines.
• Strong understanding of data warehousing, data modeling, and distributed computing.
• Experience with Git and CI/CD.
• Excellent communication and technical leadership skills.

Preferred Skills:

• Databricks certifications.
• Experience with Azure Data Factory / AWS Glue / Airflow.
• Experience with Unity Catalog and Databricks Workflows.
• Knowledge of Terraform and infrastructure automation.
• Experience with streaming technologies such as Kafka or Structured Streaming.