Quick overview
Job description
In This Role, You Will...
- Scaling self-healing data pipelines that automatically handle schema evolution, detect anomalies, execute circuit breakers and recover from failures without manual intervention.
- Define and enforce company-wide data governance, access, automated data quality testing, schema evolution policies and metadata management to maintain high-fidelity data assets.
- Implement intelligent storage lifecycle strategies, resource throttling and query optimization to minimize compute overhead while delivering high-throughput, low-latency data access for analytics and AI agents. With a focus on Databricks vs EMR (AWS) cloud optimization (cost and performance).
- Build automated CI/CD deployment templates, environment isolation, testing frameworks and version control standards, enabling rapid, reliable and zero-downtime deployments
Qualifications
- 10+ years of hands-on experience in Data Platform Engineering or Software Engineering with a proven track record of architecting and scaling production-grade data foundations.
- Deep expertise in scaling and optimization, query tuning, compute resource allocation and implementing efficient compute-storage lifecycle policies to minimize infrastructure costs.
- Experience implementing enterprise security standards, Role-Based Access Control (RBAC), Active Directory/IAM roles and fine-grained data masking, with strong familiarity supporting enterprise compliance requirements (e.g., SOX, audit trail controls, data retention policies).
- Track record of establishing enterprise data governance frameworks, automated schema evolution controls, data quality audits and near real-time observability/alerting.
- Hands-on experience building automated CI/CD pipelines, environment isolation (branch testing, rollback mechanisms), version control and automated deployment testing for data assets.
Bonus Qualifications
- Experience incorporating LLMs or GenAI directly into pipeline operations (e.g., code generation, triage and root-cause analysis, automated data reconciliation, backfills or anomaly detection).
- Experience with infrastructure-as-code (Terraform) and containerization/orchestration (EKS) for elastic compute management. GPU/CPU performance optimization & utilization.
- Previous experience in the autonomous vehicle, robotics or high-tech manufacturing sectors.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- EH

Senior Data Platform EngineerRecently posted Auto Apply
Ellipsis Health · San Francisco - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
GCPSQLGit+22Technology$150k - $170k/yr - TT
Lead Platform EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 17 Nov 2026 🇺🇸
FastAPIMicroservicesGit+10Technology$220k - $300k/yr - FA

DevOps Engineer Level 3Recently posted Auto Apply
Fantom Corporation · Fort Meade · On-site · Posted 17 Nov 2026 🇺🇸
DockerGitAnsible+12TechnologySalary not listed - VC

Senior Engineer, Dev Ops - WWE 2KRecently posted Auto Apply
Visual Concepts · California · Hybrid · Posted 16 Nov 2026 🇺🇸
GCPGitAWS+15Engineering$135.3k/yr - HA

Site Reliability Engineer, TeleradiologyRecently posted Auto Apply
Harrison.ai · Remote · Remote · Posted 16 Nov 2026 🇺🇸
GCPGrafanaRadiology+1TechnologySalary not listed - GA

Senior DevOps EngineerRecently posted Auto Apply
Gallup · Omaha Riverfront · Hybrid · Posted 16 Nov 2026 🇺🇸
DockerMySQLGit+12Technology$135k - $170k/yr - GA

DevOps EngineerRecently posted Auto Apply
Gallup · Omaha Riverfront · Hybrid · Posted 16 Nov 2026 🇺🇸
DockerMySQLGit+12Technology$90k - $135k/yr - EA

DevOpsStorage Engineer - FederalRecently posted Auto Apply
ELEVI Associates · Chantilly · Hybrid · Posted 17 Nov 2026 🇺🇸
GitAWSAnsible+18$185k - $240k/yr - EA

DevOps Engineer - FederalRecently posted Auto Apply
ELEVI Associates · Chantilly · Hybrid · Posted 17 Nov 2026 🇺🇸
DockerEncryptionOAuth+6Technology$155k - $200k/yr - ST
Applied Science Platform EngineerRecently posted Auto Apply
Standinsurance · San Francisco · On-site · Posted 17 Nov 2026 🇺🇸
RecruitingETLArticulate+6TechnologySalary not listed - SI

Staff Platform Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · Remote - No Preference · Remote · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerDynamoDB+37TechnologySalary not listed - SI

Staff Platform Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · NYC - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerDynamoDB+37TechnologySalary not listed
