Microsoft Data Engineer / Consultant

DATABRICKS CERTIFICATION IS ESSENTIAL & REQUIRED FOR THIS ROLE

IT IS ABSOLUTLEY ESSENTIAL THAT YOU HAVE SOLID CLIENT FACING DATABRICKS & DLT

We are looking for 3 Senior Azure candidates with expertise within Databricks to join our clients teams.

We have open Databricks roles with 3 different organisations.

We have 2 roles both on-site and 1 role which is hybrid.

Assignments Summary

Design, develop, and maintain data platforms and governance frameworks with a strong focus on Databricks and Fabric, ensuring robust and secure data pipelines.

Main Responsibilities

• Drive improvements of the Databricks platform setup for scalable and secure data solutions.
• Design and implement governance models, catalogs, and metadata management.
• Work with Databricks and related services including Delta Live Tables and Lakeflow connections.
• Apply data modelling principles for analytical and operational workloads, including star and snowflake schemas.
• Implement and manage Infrastructure as Code (IaC) solutions for Databricks.
• Collaborate with stakeholders to translate requirements into governed data products.

Key Requirements

• Bachelor's or Master's in Computer Science, Data Engineering, or equivalent experience.
• Proficiency in Python, SQL, and Terraform (or equivalent IaC tools).
• Fabric an advantage
• Deep expertise in the Databricks platform, including Delta Lake and Unity Catalog.
• Experience in Infrastructure as Code, automation, and CI/CD pipelines.
• Knowledge of cloud technologies, preferably AWS.
• Strong background in modeling structured and unstructured data.
• A Grenadier attitude.

Infrastructure and DevOps

• Lead and enforce the use of source control systems such as Git or Azure DevOps in line with company standards.
• Design, implement, and maintain CI/CD pipelines and environment management processes for data engineering solutions.
• Implement and maintain automated testing and data validation for pipelines.
• Lead performance testing and optimisation activities for data engineering solutions.

Testing and Quality

• Design, write, and review unit and integration tests to ensure solution quality and reliability.
• Document issues and resolutions accurately for review•
• Lead and oversee testing processes to validate data accuracy, performance, and quality.
• Ensure solutions are delivered in line with quality and documentation standards.
• Support knowledge sharing and improvement of internal frameworks and templates.

Nice to Have

• Familiarity with high-performance data processing pipelines.
• Experience with data visualisation tools.