Databricks Lead Engineer - Hybrid - Contract

A hands-on Databricks Lead Engineer is required to lead the delivery of a modern enterprise data platform for a leading financial services organisation.

The role will take technical ownership of Databricks lakehouse solutions, covering ingestion through to curated serving layers. Responsibilities will include implementing Medallion Architecture, Unity Catalog, Declarative Pipelines and scalable Databricks-native ingestion patterns while mentoring engineers and driving production-quality delivery.

• Up to £575 per day
• Inside IR35
• Initial 6-month contract with potential extension
• Hybrid working with three days per week on-site in London
• ASAP start
• Technical leadership of a strategic Databricks implementation
• Opportunity to shape engineering standards, reusable patterns and delivery practices

• At least two years of recent, hands-on Databricks experience
• Evidence of delivering Databricks solutions into production environments
• An active Databricks Data Engineering, Machine Learning or Generative AI certification
• Strong experience implementing Medallion Architecture across Bronze, Silver and Gold layers
• Hands-on Unity Catalog experience covering catalog design, access controls, lineage and secure data sharing
• Experience developing Declarative Pipelines with Expectations, including failure handling and observability
• Knowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patterns
• Production-level PySpark and SQL development skills
• Strong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUM
• Experience with Databricks Workflows, orchestration and CI/CD
• Understanding of monitoring, alerting, replay, backfill and operational support strategies
• Ability to mentor engineers and act as the technical escalation point
• Experience with RBAC, data masking, Photon, cluster optimisation or cost management would be beneficial
• Exposure to Mosaic AI, Vector Search, model serving, RAG or Lakebase would be advantageous