Lead Databricks Engineer

We are seeking a Lead Databricks Engineer with strong hands-on experience in Databricks, Apache Spark, PySpark, SQL, and modern cloud data engineering. The ideal candidate will lead the design and development of scalable data platforms, mentor engineers, and work closely with architects and business stakeholders to deliver high-performance data solutions.

Key Responsibilities

• Lead the design, development, and implementation of Databricks-based data platforms and pipelines.
• Develop scalable ETL/ELT pipelines using Databricks, Apache Spark, PySpark, and SQL.
• Design and implement Delta Lake / Lakehouse architectures.
• Build batch and near-real-time data processing solutions.
• Optimize Spark jobs, SQL queries, clusters, and data pipelines for performance and cost.
• Develop reusable data engineering frameworks and standards.
• Work with cloud storage, data warehouses, APIs, and enterprise data sources.
• Implement data quality, validation, monitoring, and error-handling frameworks.
• Use Databricks Workflows, notebooks, Delta Live Tables/Lakeflow, and CI/CD practices.
• Implement security, access control, governance, and data lineage.
• Collaborate with Data Architects, Data Scientists, Analysts, and application teams.
• Provide technical leadership, code reviews, mentoring, and troubleshooting support.