Quick overview
Job description
Position Overview:
We are looking for an experienced DevOps Engineer with strong expertise in Azure, Terraform, and CI/CD platforms. The ideal candidate should have foundational Data Engineering knowledge, hands-on experience across cloud platforms, and the ability to independently manage end-to-end projects while collaborating with clients and leadership. Flexibility to work partially in EST time zones is required.
Job Responsibilities:
• Designing DevOps Strategy: Plan source control workflows, branching models, environment provisioning, and release management strategies.
• Hybrid CI/CD Pipeline Management: Build, maintain, and optimize multi-stage pipelines using both GitHub Actions (Workflows, Runners, Secrets) and Azure Pipelines to automate deployments.
• Infrastructure as Code (IaC): Automate cloud infrastructure provisioning using Terraform (Modules, State Management, and Remote Backends) integrated directly into GitHub workflows.
• Databricks Platform Automation: Automate the deployment and configuration of Azure Databricks workspaces, including clusters, pools, and storage mounts via the Databricks Terraform Provider.
• Data Pipeline Orchestration (DataOps): Build CI/CD pipelines in GitHub
Actions or Azure DevOps for Databricks workflows, automating the deployment of notebooks, Delta Live Tables, and jobs across Dev, Test, and Prod environments.
• Cloud & Container Orchestration: Manage Azure services, including Azure
Kubernetes Service (AKS), Azure Container Registry (ACR), and serverless
functions.
• Source Control & Governance: Administer Git repositories via GitHub
Enterprise and Azure Repos, enforcing branch protection rules, code owners,
and pull request workflows.
• Security & DevSecOps Implementation: Integrate automated security scans
(like GitHub Advanced Security, Dependabot, and Secret Scanning), Azure
Policy, and Databricks Secret Scopes into deployment pipelines.
• Monitoring & Feedback Loops: Set up full-stack observability, logging, and
telemetry using Azure Monitor and Databricks diagnostic logs to track system
performance.
• Agile & Team Collaboration: Work closely with data engineers, developers, and QA testers, utilizing GitHub Projects or Azure Boards to track sprints and release milestones
Basic Qualifications:
-
Minimum 5 years of experience, preferably with a foundational Data Engineering background.
-
DevOps & CI/CD Platforms: GitHub (Actions, Advanced Security, Enterprise Features) and Azure DevOps.
-
Infrastructure as Code: Advanced Terraform (Modules, State Management, Workspace Provisioning).
-
Data & Analytics Platform: Azure Databricks (CLI, REST API, Workspace Management, Unity Catalog).
-
Cloud Platforms: Microsoft Azure (Compute, Networking, IAM, Azure Key Vault).
-
Scripting & Automation: PowerShell, Bash, or Python.
-
Strong DevOps engineer who is comfortable working across all three major cloud providers (AWS, Azure, and GCP), with a preference for Azure.
-
Strong hands-on expertise in Terraform is essential.
-
Must be comfortable collaborating frequently with clients and leadership.
-
Excellent communication and presentation skills are important, with the confidence to interact with stakeholders and present ideas without hesitation.
-
Should be flexible to work partially in EST time zones.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works