Data engineer

Job Title: Senior Databricks Platform & CI/CD Engineer

Job Summary

We are seeking a hands-on Senior Databricks Platform & CI/CD Engineer to support the implementation, administration, automation, governance, and operationalization of an enterprise Databricks platform.

The ideal candidate will have strong experience with Databricks, Unity Catalog, CI/CD, Azure DevOps, GitHub Enterprise, Databricks Asset Bundles (DAB), Python, SQL, and PySpark.

This role focuses on Databricks platform engineering and DevOps enablement, rather than cloud infrastructure provisioning.

Key Responsibilities

• Configure and administer Databricks workspaces.

• Implement and manage Unity Catalog governance standards.

• Configure catalogs, schemas, permissions, RBAC, and access controls.

• Design and implement CI/CD pipelines for Databricks solutions.

• Integrate Databricks with Azure DevOps and GitHub Enterprise.

• Build deployment automation using Databricks Asset Bundles (DAB).

• Automate deployments across Development, Test, UAT, and Production.

• Implement Git branching, pull request workflows, and release management.

• Develop automated validation, testing, and deployment processes.

• Create reusable deployment templates and engineering accelerators.

• Support data governance, lineage, audit logging, and security controls.

• Support PHI/PII data governance and compliance requirements.

• Monitor Databricks platform health, utilization, performance, and operational readiness.

• Support production deployments, incident resolution, and troubleshooting.

• Develop operational runbooks and monitoring/alerting processes.

• Support MLflow and AI/ML platform enablement when required.

Required Skills

• 5+ years of experience in Data Platform Engineering, DevOps Engineering, or Cloud Data Engineering.

• 3+ years of hands-on Databricks experience.

• 3+ years building and supporting CI/CD pipelines.

• Databricks

• Unity Catalog

• Delta Lake

• Databricks Workflows / Jobs

• Delta Live Tables (DLT)

• Databricks Asset Bundles (DAB)

• Databricks Repos

• Azure DevOps

• GitHub Enterprise

• Git branching strategies

• CI/CD pipeline development

• Release management

• Automated testing

• Python

• SQL

• PySpark

• RBAC and access management

• Data governance and data lineage

• Audit controls

• PHI/PII data handling

• Enterprise production support

Preferred Skills

• MLflow

• Databricks Model Serving

• Databricks Apps

• Vector Search

• Azure OpenAI

• Generative AI

• Agentic AI frameworks

• Healthcare or regulated-industry experience

• Monitoring and alerting

• Incident management

• Performance optimization