Quick overview
Job description
Our mission is general causal intelligence; AI that is capable of (1) predicting the future and (2) identifying the actions to alter it.
To achieve this breakthrough, we are building a Large Physics foundation Model (LPM) because physical systems, unlike text or images, are governed by verifiable cause and effect. We believe that scaling on physics will enable an understanding of causality required to predict and control physical systems, starting with weather.
Our founding team has built and deployed AI against the physical world in robotics, drug discovery, and particle physics at institutions like DeepMind, Waymo, Cruise, Insitro, Nabla Bio, and CERN.
We look for data engineers who are excited to tackle unsolved problems. Physical observations arrive continuously, in many formats, at a scale that dwarfs what is used to train today's LLMs. Your mission is to build the data platform underneath it all — the storage, compute, and loading systems that make every dataset cheap to ingest, fast to query, and immediately available to training.
Responsibilities
Design and operate petabyte-scale storage: lakehouse architecture, file formats, and data layout optimized for both batch and real-time queries
Own the shared compute and orchestration platform (e.g. Spark, Ray, workflow scheduling) that ingestion and research pipelines run on
Optimize data strategy end to end from storage to loading, owning high-throughput data loading into training up to the tensor boundary
Build systems for cataloging, deduplication, lineage, search, and reproducibility at every stage of the data lifecycle
Implement the platform-level quality and monitoring tooling that data and research teams build their checks on
Scale infrastructure to improve engineering velocity and ensure reliability, with monitoring and alerting to match
Work across the full data lifecycle when the mission needs it — including building and operating ingestion pipelines for critical data sources directly
What we're looking for
We value a relentless approach to problem-solving, rapid execution, and the ability to quickly learn in unfamiliar domains.
Demonstrated experience building large-scale data pipelines and distributed compute systems (e.g. Spark, Ray, Beam)
Knowledge of state-of-the-art methods and tools for data ingestion, storage, and loading — including file formats and storage systems (e.g. Parquet, Zarr, Delta Lake) and how they impact performance and scalability
Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines
Understanding of how data loading throughput affects large-scale training, and experience optimizing it
Owns deliverables end-to-end, from collecting and translating requirements to autonomously driving execution
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- OR
Data Engineer & Application Development (TS/SCI w/ Poly) Auto Apply
Orbis · McLean · Hybrid · Posted 3 Nov 2026 🇺🇸
SQLGitData Pipeline+7TechnologySalary not listed - WE

Staff Data Engineer Auto Apply
Webflow · United States · Remote · Posted 20 Oct 2026 🇺🇸
SQLAirflowEMR+5Technology$212.5k - $255k/yr - OV

Staff Data Engineer Auto Apply
Overstory · United States · Remote 🇺🇸
AirflowBigQueryGoogle Cloud+2TechnologySalary not listed - LT
Staff ML Data Engineer Auto Apply
Liveview Technologies · Seattle · Hybrid 🇺🇸
ETLMLOpsCompliance+5Technology$171.9k - $221k/yr - F7
Staff Engineer, Data Platform (R5659) Auto Apply
F7d8749a 72d6 407c Bf98 292ea630c73a · San Diego · Hybrid · Posted 2 Nov 2026 🇺🇸
Talent AcquisitionData PipelinePython+3TechnologySalary not listed - F7
Data Engineer, Finance (R6329) Auto Apply
F7d8749a 72d6 407c Bf98 292ea630c73a · San Diego · Hybrid · Posted 21 Oct 2026 🇺🇸
SQLApache SparkSnowflake+4TechnologySalary not listed - ML
Senior Data EngineersRecently posted Auto Apply
Movement Labs · United States · Remote · Posted 14 Nov 2026 🇺🇸
SQLAirflowBigQuery+4$90k - $150k/yr - AG

Staff Data EngineerRecently posted Auto Apply
Agiloft · United States · Remote · Posted 13 Nov 2026 🇺🇸
Talent AcquisitionAWSAirflow+3TechnologySalary not listed - HI

Data Conversion Lead (1657)Recently posted Auto Apply
Highstreetit · United States · Hybrid · Posted 13 Nov 2026 🇺🇸
Stakeholder ManagementOracleSQL+5Salary not listed - CB

IT Data Science Principal (Data Warehouse)Recently posted Auto Apply
Cornerstone Building Brands · Cary, NC · Remote · Posted 12 Nov 2026 🇺🇸
Stakeholder ManagementPL/SQLSQL+8Technology$120k - $140k/yr - SG
Lead Data Engineer (Lakehouse / Databricks)Recently posted Auto Apply
Stafford Gray · Lansing, Michigan · Hybrid · Posted 13 Nov 2026 🇺🇸
DockerGCPSQL+15TechnologySalary not listed - MI
Data Engineering Solution Lead - RemoteRecently posted Auto Apply
Mindex · Rochester, New York · Remote · Posted 13 Nov 2026 🇺🇸
MongoDBSQLETL+5$105k - $155k/yr
