Data Engineer - AWS - SC Cleared

Are you an experienced Data Engineer? Ready and available to support the delivery of an AWS-based data solution for a leading MSC working with a Government end client. You will be working closely with the Technical Lead to deliver reliable, production-ready data solutions and build scalable data pipelines, transforming complex datasets, and implementing data-matching and record-linkage solutions.

Key Responsibilities

• Design, develop and maintain scalable ETL/ELT pipelines using Spark and Python Scripting.
• Ingest, transform and validate data from multiple sources and formats.
• Implement data matching, deduplication, and record-linkage processes.
• Develop automated testing, monitoring, and data quality controls.
• Optimise data pipelines for performance, scalability, and cost.
• Apply engineering best practices, including Git, CI/CD and secure development.
• Collaborate with technical teams and stakeholders within an Agile environment.

Essential Skills & Experience

• Active SC clearance
• Strong hands-on experience with Spark, Python and SQL.
• Experience working within Agile/Scrum delivery teams.
• Proven experience building production-grade data pipelines.
• Strong understanding of ETL/ELT, data modelling and data quality.
• Experience with data matching, record linkage or complex data transformations.
• Experience with automated testing, Git and CI/CD.

This is a fully remote role but will require "key touch points" in the programme, as well as laptop collection & ID verification.

You must have or previously have held SC clearance. If this is you? If so, apply immediately.

This role is inside IR35