Senior Data/Platform Engineer

Note:

We don't need an Azure Data Engineer

The candidates' hands-on experience appears to be limited to ADF or Synapse workspace analytics. We're looking for broader data platform engineering experience.

Must Have:

• Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows).
• Python/PySpark: strong programming skills for data processing, automation, and Scripting.
• Azure data stack: Synapse, Databricks, ADLS Gen2, Key Vault - including Linked Services, Managed Identity, and Spark Pool configuration.
• Azure platform fundamentals: compute, storage, networking (VNets, NSGs, Private Endpoints), identity and RBAC.
• CI/CD: GitHub Actions (with OIDC federation) and/or Azure DevOps for data and platform deployments.
• Infrastructure-as-Code: Pulumi (or Terraform/Bicep) across multiple environments.
• Scripting: PowerShell and Bash for platform automation.
• Monitoring & observability: Azure Monitor, Log Analytics, KQL.
• Big data file formats: Parquet and Delta Lake.
• Cloud-native data modelling and ETL/ELT frameworks on Azure.

Good to Have:

• Azure certifications: AZ-104, AZ-400, DP-203.
• Azure Data Factory (ADF), Microsoft Purview, Microsoft Fabric.
• Data governance: lineage, cataloguing, sensitivity labels.
• Event-driven architectures: Kafka/Azure Event Hubs.
• Delta Live Tables, MLflow.
• IDMC Secure Agent, Power Automate flows.
• Security baselines: CIS/NIST.
• Observability tooling: OpenTelemetry, Datadog.
• AI tools and their application in data engineering.
• Knowledge of the UK Insurance Market.
• Agile/Scrum delivery experience.