Machine Learning Engineer

Our Company

We exist to bring amazing people together to explore the art of possible.

Bailey Abbott are a progressive and dynamic IT Consultancy business working across both Public and Private sectors. People are at the core of everything we do. We're all about nurturing and inspiring people - ours and our clients. We bring a positive attitude and mindset to everything we do. We confidently use our knowledge and skills to solve problems, finding a better and easier way forward.

We deliver transformational outcomes to take businesses further, fearlessly. Confidence and trust is fundamental to delivering superior outcomes and we take ownership and responsibility for our commitments.

Our People

Our staff are creative, collaborative, and passionate. We're strong advocates for change, challenging the status quo through new thinking, technology, and practices. Always exploring creative ways to simplify complexity.

Role Overview

We are seeking a Machine Learning Engineer with a strong software engineering foundation to productionise, scale, and maintain advanced analytical and predictive models. In this role, you will bridge the gap between data science experimentation and robust software engineering, designing scalable data pipelines and automated ML lifecycles within our Databricks environment using Spark, R, and modern engineering practices.

Key Responsibilities

• Production ML Pipelines: Build, test, and optimize production-grade data processing and model training pipelines in Apache Spark and Databricks.

• MLOps & Lifecycle Automation: Standardise model deployment, versioning, monitoring, and automated retraining workflows (CI/CD, MLflow).
• Code Quality & Architecture: Apply software engineering best practices-modular code architecture, automated testing, containerisation, and version control-to analytical and statistical codebases.
• R & Statistical Workloads: Optimise and scale R-based modeling workflows to run efficiently over distributed data architectures using Sparklyr/SparkR.
• Collaboration & Integration: Partner with data architects, data engineers, and domain analysts to integrate predictive outputs cleanly into downstream systems and analytical platforms.
• Performance & Reliability: Monitor pipeline latency, manage compute cluster sizing, and ensure data integrity and governance across production workspaces.

Required Skills & Experience

• Software Engineering Fundamentals: Solid foundation in core software engineering principles (design patterns, unit/integration testing, Git version control, CI/CD pipelines).
• Databricks Ecosystem: Proven hands-on experience orchestrating workloads, managing workspaces, and using MLflow / Unity Catalog within Databricks.
• Distributed Computing (Apache Spark): Deep practical understanding of Spark internals, data partitioning, query optimization, and memory management.
• R for Production / Advanced Analytics: Strong proficiency in R (including Spark integrations like sparklyr or SparkR) for statistical modeling, forecast automation, or data manipulation.
• Data Layer Experience: Comfort querying complex relational data, working with Delta Lake formats, and handling large-scale time-series or tabular datasets.

Desirable Skills

• Familiarity with Python/PySpark alongside R for cross-language workflows.
• Experience with cloud infrastructure (Azure, AWS, or GCP) and Infrastructure as Code (Terraform).
• Understanding of model drift detection, data lineage, and enterprise data governance frameworks.

Our clients are diverse and so are we. We engage with great talent from all walks of life to bring their extensive and varied experience to help promote innovation. We encourage applications from candidates from all backgrounds to further strengthen Bailey Abbott.

Bailey Abbott. Explore Possible.