Quick overview
Job description
Databricks Technical Lead l Contract | 12 Months | Outside IR35 l Up to £1,000 per day | Remote first, occasional on-site (Midlands & South East)
The role
Method Resourcing have been asked by one of our consulting partners to find a hands-on Databricks Technical Lead to design and deliver a new Azure Databricks platform for a major life sciences organisation replacing its enterprise data and analytics stack.
The programme uses AI and LLM tooling to reverse-engineer existing pipelines, business logic and reports into documented, tested code. You'll mentor three senior data engineers, own the technical architecture and stay hands-on in development.
What you'll do
- Architect and build the platform on Azure Databricks (Delta Lake, Unity Catalog, medallion architecture)
- Use AI agents and LLMs to reverse-engineer legacy reports, queries and transformations, and build repeatable AI-assisted processes for code generation, testing and migration
- Build automated testing and reconciliation to validate migrated data against existing reporting
- Integrate SAP, Azure Data Lake, Salesforce and other systems, and design gold-layer models for Power BI
- Set engineering standards (Workflows, CI/CD, IaC, governance) and mentor the team through to UAT and cutover
What you'll need
- Extensive hands-on experience delivering and technically leading enterprise solutions on Azure Databricks
- Demonstrable use of AI, LLMs or agents to reverse-engineer data pipelines or reporting logic
- Strong Python, PySpark and SQL
- Deep knowledge of Delta Lake, Unity Catalog, Workflows and medallion architecture
- Automated validation, reconciliation and testing experience
- Azure data engineering (ADLS Gen2, ADF)
- Databricks CI/CD (Terraform, Asset Bundles or Azure DevOps)
- SAP ECC or S/4HANA integration experience
- Dimensional modelling and Power BI semantic model familiarity
- Proven team leadership while staying hands-on
Desirable
- Databricks certification (Data Engineer Professional or Solutions Architect)
- Platform migration, carve-out or separation experience
- Reusable AI agent frameworks for data engineering
- Strong governance and data quality experience
- Consulting background
Contract
- outside IR35
- £900-£1,000 per day
- Remote with some travel required
- 12 month contract
If this is you, send your CV to and if suitable, i'll be in touch. As you can appreciate, this role will receive a high volume of candidates so I will do my best to respond if possible.
Databricks Technical Lead l Contract | 12 Months | Up to £1,000 per day | Remote first, occasional on-site (Midlands & South East)
RSG Plc is acting as an Employment Business in relation to this vacancy.
Similar jobs
- LR
Lead Data EngineerRecently posted
Lorien Resourcing Limited · South East London, London · On-site · Posted 17 Nov 2026 🇬🇧
AWSMLOpsMLflow+10TechnologySalary not listed - SC

Data EngineerRecently posted
Syntax Consultancy Limited · Milford, Belper · Remote · Posted 17 Nov 2026 🇬🇧
SQLT-SQLApache Spark+8Technology£40k/yr - GG
Databricks EngineerRecently posted
Gazelle Global Consulting Ltd · Farringdon, Central London · On-site · Posted 17 Nov 2026 🇬🇧
UnityEngineeringSalary not listed - CY
PostgreSQL DBARecently posted
Cyberteam · Manchester, Greater Manchester · On-site · Posted 17 Nov 2026 🇬🇧
GCPMongoDBGit+7TechnologySalary not listed - HD
Data Warehouse EngineerRecently posted
Hays DT - Midlands · Birmingham · Hybrid · Posted 17 Nov 2026 🇬🇧
SQLTeamCityGit+16Technology£63.0k/yr - A-
Data Engineer – ICT Level 5Recently posted
AFBI - Agri-Food and Biosciences Institute · Belfast, Northern Ireland · Hybrid · Posted 17 Nov 2026 🇬🇧
Technology£37.7k - £39.0k/yr - MG
Python Data EngineerRecently posted Auto Apply
Man Group · London · On-site · Posted 17 Nov 2026 🇬🇧
OracleSQLGit+12TechnologySalary not listed - TA
Data EngineerRecently posted
Talent · United Kingdom · Hybrid 🇬🇧
Stakeholder ManagementSQLApache Spark+8TechnologySalary not listed - OF

Data EngineerRecently posted
Outcomes First Group · manchester · Hybrid 🇬🇧
Stakeholder ManagementSQLApache Spark+5Technology£49k/yr - BH
Hybrid Data Engineer: SQL Pipelines & QualityRecently posted
Believe Housing Limited · bishop auckland · Hybrid · Posted 17 Nov 2026 🇬🇧
SQLSQL ServerTechnologySalary not listed - SE

Junior Data Engineer – Live Service & Incident Support (SC)Recently posted
Searchability · shrewsbury · Hybrid · Posted 17 Nov 2026 🇬🇧
SQLJavaTechnologySalary not listed - NO

Lead Quantexa Data Engineer - Hybrid (Birmingham)Recently posted
Noir · birmingham · Hybrid · Posted 17 Nov 2026 🇬🇧
ScalaGitDatabricks+6TechnologySalary not listed