Senior Data Engineer

Clearance required - active DV clearance
Client sites in Wiltshire full time. Potentially other UK site travel as well.
Opportunity to condense the hours of work.
This role is inside IR35.

What you will do as a Senior Data Engineer
Design, build and maintain secure, scalable data solutions across the full life cycle, including acquisition, integration, storage, processing and analysis.
Develop and maintain data ingestion and transformation pipelines using Apache NiFi, Python and SQL.
Design data models, schemas and data contracts/Interface Control Documents (ICDs) to support reliable integration between systems.
Integrate data from REST APIs and other source systems, working with Parquet and S3-compatible object storage.
Implement data validation, assurance, metadata management and traceability to ensure data is accurate, complete and fit for purpose.
Develop tools for data-driven insights and task automation, and research and adopt appropriate emerging technologies.
Collaborate with customer and internal stakeholders to define data, interface and tool requirements.
Work closely with DevOps Engineers, Data Scientists, Analysts and Cyber Consultants to deliver integrated solutions.
Take ownership of deliverables and ensure the quality, security and maintainability of outputs.
What you'll bring
Proven experience in data engineering, with the ability to work independently and take ownership of customer-facing deliverables.
Strong Python and SQL skills, with experience building, testing and optimising data pipelines.
Hands-on experience with Apache NiFi for data ingestion, routing, transformation and orchestration.
Experience of data modelling and schema design, including defining data contracts or Interface Control Documents (ICDs).
Experience working with Parquet and S3-compatible object storage.
Experience integrating REST APIs and troubleshooting data exchanges between systems.
Experience deploying or supporting data solutions within OpenShift or Kubernetes environments.
Experience using GitLab CI/CD to automate build, test and deployment activities.
Understanding of data validation and assurance approaches, including quality controls and reconciliation.
Understanding of metadata management, data lineage and traceability.
Strong problem-solving skills and the ability to collaborate across multidisciplinary teams.
Clear communication skills for engaging customer stakeholders and sharing technical decisions and insights.
Core Areas (must have):
Proven experience delivering data engineering solutions in a project or production environment.
Strong Python and SQL skills.
Hands-on experience with Apache NiFi and data pipeline orchestration.
Experience with data modelling and schema design.
Experience with parquet and S3-compatible object storage.
Experience with REST API integration.
Experience with OpenShift or Kubernetes.
Experience with GitLab CI/CD practices and pipelines.
Experience developing Data contracts or Interface Control Documents (ICDs).
Experience with data validation and assurance.
Experience with metadata management, lineage and traceability.
Critical thinking, problem-solving and effective stakeholder communication.
Desirable:
Degree in STEM, Data, AI, or Programming-related field
Experience influencing technical decisions and supporting capability growth.
Experience with big data tools eg Spark, Hadoop, NiFi
Exposure to Ansible and Terraform
Understanding of ML algorithms and deployment