Senior Software Engineer

Lead Backend Software Engineer – Data Services Federated Data Services | Distributed Systems | Apache Spark | DataFusion | Python | Kubernetes<br />Location: London (2 days per week onsite)<br />Rate: £700–£800 per day (Outside IR35)<br />Duration:

Initial 6-month contract with likely extension Overview<br />We're looking for an experienced Lead Backend Software Engineer to join a high-profile programme building secure, cloud-native data services that will support data-driven research and innovation at national scale.<br />This is a founding engineering role, offering the opportunity to shape the architecture, engineering standards, and core data services from the ground up.

Working closely with the Engineering Manager and a small team of senior engineers, you'll establish the technical foundations for a complex federated data platform operating within a highly regulated environment.<br />We're looking for someone who enjoys building from first principles, thrives in ambiguity, and can balance strategic architectural thinking with hands-on software engineering.

This role is centred around designing and building distributed data services rather than traditional application development.

Role &amp; Responsibilities Lead the design and development of scalable, cloud-native data services supporting a federated platform.<br />Define architectural patterns, engineering standards, and best practices that will scale across the wider organisation.<br />Design and build distributed server-side services using modern software engineering practices.<br />Architect and develop federated data services capable of securely querying and processing data across multiple environments.<br />Design high-performance data processing and streaming workflows using technologies such as Apache Spark, Apache DataFusion, or similar distributed query frameworks.<br />Design and implement Kubernetes-based services and containerised workloads.<br />Build secure APIs and backend services supporting authentication, authorisation, governance, and audit requirements.<br />Contribute to the design of distributed systems, data pipelines, and platform architecture.<br />Establish CI/CD pipelines, Infrastructure as Code, automated testing, and deployment practices.<br />Evaluate and integrate open-source technologies and distributed data frameworks.<br />Mentor engineers and promote engineering excellence across the team.<br />Work closely with architecture, product, security, and external stakeholders.<br />Skills &amp; Experience<br />Essential Experience as a Lead Engineer, Principal Engineer, Staff Engineer, Software Architect, or similar senior technical role.<br />Strong server-side software engineering experience, with flexibility across programming languages such as Python, Java, Go, Scala, C#, or similar.<br />Experience designing and building distributed data services or large-scale backend platforms.<br />Experience with Apache Spark, Apache DataFusion, Trino, Presto, or similar distributed data processing technologies.<br />Strong architecture and design experience across distributed systems and data services.<br />Experience designing scalable data processing or data streaming workflows.<br />Experience building cloud-native applications on AWS, Azure, or GCP.<br />Experience with Kubernetes, Docker, and containerised environments.<br />Experience building and operating CI/CD pipelines and Infrastructure as Code.<br />Strong understanding of distributed systems, scalability, resilience, and secure-by-design principles.<br />Comfortable making architectural decisions within evolving, fast-paced environments.<br />Excellent communication skills with the ability to work across technical and non-technical teams.<br />Desirable Experience building federated data services or federated data platforms.<br />Experience within healthcare, life sciences, research, government, or other highly regulated environments.<br />Knowledge of Trusted Research Environments (TREs), Secure Data Environments (SDEs), or similar secure data platforms.<br />Experience with data governance, metadata services, data catalogues, or secure data access models.<br />Exposure to event-driven architectures and streaming technologies such as Kafka, Flink, or Spark Streaming.<br />Experience evaluating or contributing to open-source technologies