Quick overview
Job description
About the Role
Join an AI team building large physics foundation models to predict real-world systems and identify effective actions. You will develop the distributed infrastructure that supports model training and inference, with weather data as a key application area.
What You'll Do
Design, deploy, and maintain distributed machine learning training and inference clusters.
Build scalable end-to-end pipelines for petabyte-scale datasets and model training across the ML lifecycle.
Evaluate training approaches, including parallelization methods and numerical precision trade-offs across model sizes.
Profile and debug low-level GPU operations to improve performance.
Track relevant research and apply promising ideas to infrastructure and workflows.
What We're Looking For
Two to ten years of experience building large-scale infrastructure for pretrained or foundation models, including compute clusters, data infrastructure, inference, and distributed training.
Hands-on experience optimizing large-scale training, data, or inference workloads, and using distributed training frameworks such as FSDP or DeepSpeed.
Experience with low-level GPU performance analysis and debugging, including CUDA and JAX.
Familiarity with Python, C++, Linux, NVIDIA GPUs, cloud platforms such as GCP, AWS, or Azure, and containerization and orchestration tools such as Docker and Kubernetes.
Background in physical AI or AI for science, such as robotics, autonomous systems, or biology, and motivation to work on scientific or physical-world applications.
Comfort working across the ML lifecycle in a fast-paced environment, including analyzing experiments and ablation studies.
Compensation & Benefits
Base salary is $200,000 to $400,000 annually. Visa sponsorship is not available.
Location
On-site in San Francisco, California. Relocation support is provided.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- PT
Splunk Security EngineerRecently posted Auto Apply
Pt78 · Tampa · On-site · Posted 19 Nov 2026 🇺🇸
SplunkBash.NET+7Technology$145k - $155k/yr - AR

Senior Platform EngineerRecently posted Auto Apply
Artafinance · Bay Area · Hybrid · Posted 20 Nov 2026 🇺🇸
MicroservicesKafkaPython+2Technology$180k - $300k/yr - AL
Application Support Engineer (Remote) - USRecently posted Auto Apply
AlphaSights · Remote · Remote · Posted 19 Nov 2026 🇺🇸
Technology$80k/yr - IN

Service Desk AnalystRecently posted Auto Apply
Impact Networking · Lake Forest · On-site · Posted 19 Nov 2026 🇺🇸
Talent AcquisitionWorkforce PlanningMicrosoft Office+4Technology$20k/yr - TT
Senior Software Engineer, Autonomous Systems (C2/Backend/Service)Recently posted Auto Apply
True Talent Solutions Partners · Palo Alto · On-site · Posted 20 Nov 2026 🇺🇸
MicroservicesRustRobotics+4Technology$145k - $250k/yr - TT
Software Engineer, C++ SystemsRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 20 Nov 2026 🇺🇸
RustC++Kubernetes+1Technology$200k - $250k/yr - TC

Senior Platform EngineerRecently posted Auto Apply
Thyme Care · Remote · Remote · Posted 19 Nov 2026 🇺🇸
DockerRustGit+9Technology$170k - $200k/yr - TT
Software Engineer, RL EnvironmentsRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 19 Nov 2026 🇺🇸
ConcreteLLMPython+1Technology$260k - $290k/yr - TT
Lead Platform EngineerRecently posted Auto Apply
True Talent Solutions Partners · South San Francisco · On-site · Posted 19 Nov 2026 🇺🇸
DjangoFastAPIAWS+4Technology$205k - $235k/yr - TT
Technical Product ManagerRecently posted Auto Apply
True Talent Solutions Partners · Sunnyvale · On-site · Posted 19 Nov 2026 🇺🇸
SnowflakeAirflowDatabricks+4Technology$150k - $250k/yr - TT
Founding Full-Stack EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 19 Nov 2026 🇺🇸
DockerGCPAWS+7Technology$200k - $275k/yr - TT
Robotics Test EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 19 Nov 2026 🇺🇸
RoboticsC++Python+1Technology$182k - $230k/yr