Databricks Lead Engineer

££575 GBP

Hybrid WORKING

Location: Central London, Greater London - United Kingdom Type: Contract

NEW CONTRACT ROLE - DATABRICKS LEAD ENGINEER

A new opportunity has just landed for an experienced, hands-on Databricks Lead Engineer to join a major data transformation programme.

We're looking for someone with strong recent experience delivering modern, production-grade data platforms on Databricks. This is a hands-on technical leadership role, so we're looking for someone who can lead engineering delivery and design solutions, rather than simply work from an existing blueprint.

Location: London - 3 days per week on-site

Rate: Up to £575 per day

Contract: Inside IR35

Duration: 6 months initial contract with potential extension

Start: ASAP

Key requirements

• 2+ years of recent, hands-on Databricks experience delivering production solutions
• Active/current Databricks certification - Data Engineer, Machine Learning or Generative AI
• Strong experience designing and implementing Databricks Lakehouse architectures
• Hands-on implementation of Medallion Architecture - Bronze, Silver and Gold layers
• Experience with Unity Catalog, including governance, access control, lineage and secure data sharing
• Strong experience building Declarative Pipelines / DLT, including Expectations, data quality, failure handling and observability
• Experience with Databricks-native ingestion approaches such as AutoLoader and/or Lakeflow Connect
• Strong PySpark and SQL skills
• Delta Lake experience, including MERGE, schema evolution, optimisation and performance tuning
• Experience with Databricks Workflows and production pipeline orchestration
• Strong understanding of CI/CD, testing, monitoring, alerting, incident triage and operational best practice
• Ability to mentor engineers and act as the technical escalation point within the delivery team

Nice-to-have experience includes

• Mosaic AI, Vector Search, model serving or RAG pipelines
• Lakeflow ingestion connectors
• Row-level security, column masking and sensitive data handling through Unity Catalog
• Databricks performance optimisation, including Photon, cluster sizing, partitioning and shuffle/skew mitigation
• Cost optimisation and understanding of Databricks compute and DBU drivers

This is a great opportunity for a technically strong Databricks specialist who wants to take ownership of engineering delivery and help shape a modern enterprise data platform.

If you're interested, or know someone who could be a good fit, please get in touch.

#Databricks #DataEngineering #DataEngineer #Lakehouse #PySpark #DeltaLake #UnityCatalog #DataJobs #ContractJobs #LondonTech #InsideIR35