Senior Data Engineer - Databricks

Role Overview

We are seeking a hands-on Senior Data Engineer with strong Databricks expertise to support the modernisation and optimisation of enterprise data platforms.

The successful candidate will work on migrating and improving data pipelines, enhancing processing performance, implementing automated data quality checks, and safely retiring redundant workloads. The role requires strong Python, PySpark and SQL skills, alongside practical experience delivering reliable, scalable data solutions into production.

Candidate must have Active Security Clearance (SC)

Key Responsibilities

• Data Pipeline Development and Migration
• Assess existing Databricks notebooks, SQL queries, PySpark code, schedules and dependencies.
• Identify opportunities to migrate, simplify, optimise or retire existing data processing workloads.
• Develop and enhance production-grade ETL/ELT pipelines using Python, PySpark, SQL and Databricks.
• Handle incremental data loads, late-arriving data, deletes, backfills and safe reruns while preserving business logic.
• Ensure data processing remains accurate, reliable and maintainable throughout migration and modernisation activities.

Databricks Performance and Cost Optimisation

• Investigate slow-running or expensive workloads using Spark UI, query plans and job metrics.
• Optimise joins, partitioning, data skew, shuffles, file sizes and compute configurations.
• Reduce unnecessary full refreshes and redundant processing.
• Monitor and compare execution time, reliability and compute costs before and after optimisation.

Data Quality and Automated Testing

• Build reusable, automated data quality and reconciliation frameworks.
• Implement checks covering schemas, data types, record counts, duplicates, null values, key integrity and business rules.
• Develop unit, integration and regression tests to validate data pipelines and transformations.
• Produce clear test results, identify discrepancies and support timely resolution of data quality issues.
• Validate incremental processing, deletes, reruns and relevant data protection requirements.

Production Deployment and Release Management

• Collaborate with architects, analysts, QA teams and platform engineers to define acceptance criteria.
• Manage code changes through Git, code reviews and CI/CD pipelines.
• Validate data outputs and complete reconciliation and parallel testing where required before production release.
• Maintain technical documentation, data mappings, test evidence, release notes and operational support guides.
• Support production troubleshooting, deployment activities and operational handover.

Pipeline Decommissioning

• Identify unused, duplicated or superseded data pipelines and scheduled jobs.
• Review dependencies with relevant stakeholders before making changes.
• Plan and execute controlled retirement activities, including monitoring, retention requirements and rollback procedures.
• Validate that retired workloads no longer generate unnecessary processing or costs.

Essential Skills and Experience

• Strong hands-on experience with Databricks, including notebooks, scheduled jobs and Delta Lake.
• Excellent Python, PySpark and SQL skills.
• Proven experience building, maintaining and supporting production ETL/ELT pipelines.
• Experience with incremental data processing, data transformations and complex data workflows.
• Strong understanding of Apache Spark performance tuning and troubleshooting.
• Practical experience developing automated data quality checks, testing frameworks and source-to-target reconciliation.
• Experience with Git, code reviews, CI/CD and production release processes.
• Knowledge of cloud object storage, identity management and secrets management on AWS or Azure.
• Ability to investigate data processing failures, identify root causes and implement effective solutions.
• Experience delivering data migration, optimisation or pipeline retirement initiatives into production.
• Strong analytical, problem-solving and stakeholder communication skills.