Data Engineering Lead / DevOps Lead

Job Summary The Data Engineering Lead / DevOps Lead is responsible for building and operationalizing the core foundation of the Data Intelligence Platform Enablement Framework. This role will lead the implementation of governance-as-code, infrastructure-as-code, CI/CD automation, workspace configuration standards, platform monitoring, and observability capabilities to deliver a scalable, secure, and enterprise-ready data platform. The ideal candidate is a hands-on technical leader with strong Databricks, Azure, DevOps, platform engineering, and automation expertise who can establish engineering standards and enable self-service deployment capabilities.

Key Responsibilities Lead the implementation of the Data Intelligence Platform core infrastructure and enablement framework. Design and build reusable platform services, deployment patterns, and engineering standards. Develop scalable platform capabilities supporting analytics, data products, AI, and self-service development. Establish repeatable engineering practices that support enterprise adoption and growth. Design and implement governance-as-code frameworks across Databricks and Azure environments.

Automate provisioning and enforcement of security, access controls, Unity Catalog policies, data classifications, and governance standards. Partner with Security, Architecture, and Data Governance teams to operationalize governance requirements through automation. Ensure platform controls are consistently deployed across environments. Lead infrastructure automation initiatives using Infrastructure-as-Code methodologies. Develop and maintain Terraform modules and deployment frameworks for Databricks and Azure services.

Automate environment provisioning, configuration management, and platform scaling. Standardize deployment patterns to improve consistency, reliability, and speed. Design and implement enterprise CI/CD pipelines supporting data engineering, analytics, and platform workloads. Establish automated deployment frameworks for code, infrastructure, notebooks, workflows, and platform configurations. Enable version control, automated testing, release management, and deployment governance. Improve developer productivity through automation and self-service platform capabilities.

Define workspace architecture, environment standards, and deployment strategies. Manage workspace configuration, cluster policies, compute governance, secret management, and environment isolation. Develop standards for development, testing, and production environments. Ensure platform configurations align with security, governance, and operational requirements. Implement platform-wide monitoring, observability, and operational health capabilities. Define and manage logging, alerting, dashboards, performance monitoring, and incident response processes.

Establish proactive monitoring for platform health, usage, performance, security, and cost management. Support Site Reliability Engineering (SRE) practices and operational excellence initiatives. Provide technical leadership and mentorship to platform engineers, data engineers, and DevOps resources. Establish engineering standards, deployment patterns, and operational best practices. Lead troubleshooting of complex platform, infrastructure, and automation challenges. Drive continuous improvement through automation, standardization, and platform innovation.

Required Qualifications Bachelor's degree in Computer Science, Information Technology, Engineering, or related field. 8+ years of experience in Data Engineering, Platform Engineering, DevOps, or Cloud Infrastructure. 5+ years of hands-on experience with Azure cloud technologies. 5+ years of Databricks experience. 3+ years of experience supporting Databricks platform deployments and administration. Strong hands-on experience with Unity Catalog implementation and permissions management. Strong experience with Terraform, Infrastructure-as-Code, CI/CD automation, and DevOps practices.

Strong understanding of cloud security, identity management, networking, and platform governance. Experience implementing data governance and security models. Experience implementing monitoring, observability, and operational support frameworks. Proven ability to act as a Databricks SME and partner with infrastructure teams. Proven ability to lead technical initiatives and drive engineering best practices.

Preferred Qualifications Databricks Certified Data Engineer or Databricks Platform certifications. Microsoft Azure DevOps Engineer Expert certification. Experience with Unity Catalog, Delta Lake, Azure Data Lake Storage, Azure Key Vault, and Azure Monitor. Experience implementing Governance-as-Code and Policy-as-Code frameworks. Knowledge of GitHub Actions, Azure DevOps Pipelines, Jenkins, or similar automation platforms. Experience supporting enterprise Data Intelligence, Data Product, or AI platforms.

Education: Bachelors Degree Certification: Databricks Certified Data Engineer , Databricks Platform , Microsoft Azure DevOps Engineer Expert