Data Engineer - SC Cleared - AWS

Data Engineer - AWS/Python/Kafka - SC Cleared

We are looking for experienced Data Engineers to support a large-scale central government data and analytics programme.

You will design, build and support secure, scalable data pipelines and services, working across cloud infrastructure, streaming technologies and production data environments.

Due to the nature of the roles, active SC is required.

Key responsibilities

• Design, build and maintain data pipelines and data-processing services.
• Develop production-quality solutions using Python and SQL.
• Build and support Kafka-based streaming and event-driven architectures.
• Work with AWS services and cloud-native data platforms.
• Manage infrastructure through Terraform/Infrastructure as Code.
• Support data profiling, mapping and transformation.
• Re-engineer existing pipelines as upstream systems or requirements change.
• Contribute to monitoring, incident investigation and production support.
• Work with engineering, architecture and delivery teams to improve data services.

Essential experience

• Strong Python engineering experience.
• Strong AWS experience.
• Apache Kafka/streaming data.
• Terraform.
• Strong SQL.
• ETL/ELT and data pipeline development.
• CI/CD and automated deployment.
• Experience supporting production data services.