Quick overview
Job description
About the Role
Join an AI team developing large foundation models for physical systems, with weather as a core application. You will build the infrastructure that enables scalable model training and inference, helping advance AI for science and physical AI.
What You'll Do
Design, deploy, and maintain distributed machine learning training and inference clusters.
Build scalable end-to-end pipelines for large datasets and model training across the machine learning lifecycle.
Test training approaches, including parallelization methods and numerical precision trade-offs across model scales.
Profile and debug low-level GPU operations to improve performance.
Follow relevant research and apply useful ideas to infrastructure and training workflows.
What We're Looking For
Two to ten years of experience building large-scale machine learning infrastructure for foundation models, including compute clusters, data infrastructure, inference, and distributed training.
Experience optimizing large-scale training, data, or inference workloads, with expertise in distributed training frameworks such as FSDP or DeepSpeed.
Proficiency with Python and C++, and experience with GPU performance optimization, CUDA, or JAX.
Familiarity with cloud platforms such as GCP, AWS, or Azure, plus Docker, Kubernetes, and scalable model serving or task management systems.
Background in physical AI or AI for science, such as robotics, biology, or other science-focused applications.
Experience across the machine learning lifecycle, including analyzing training experiments and ablation studies, and working in fast-paced environments.
Compensation & Benefits
Base salary range: $200,000 to $400,000 USD annually. Visa sponsorship is not available.
Location
On-site in San Francisco, California, United States, five days per week. Relocation support is available.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works