Quick overview
Job description
-
Training Performance at Scale: Own step time, scaling efficiency and long-run stability for foundation-model training across thousands of accelerators — from the data path to kernels, parallelism and communication.
-
Cross-Team Technical Leadership: Work directly with the modeling, infrastructure, simulation, data-mining and robotics teams on their hardest bottlenecks, and set the performance direction those teams build against.
-
Reusable by Design: Turn one-off wins into modular components that become defaults across most models and workloads, rather than a patch that helps a single job.
-
Observability at Scale: Extend performance tooling and metric coverage across the stack, while keeping instrumentation overhead low enough to stay on for production jobs fleet-wide.
-
Agentic Automation: Build agents that reproduce any job on demand, verify numerical parity and separate a real regression from run-to-run noise — so that most of the optimizations that land are found and fixed by AI rather than by hand.
-
Master's or PhD in CS, CE or EE, or equivalent practical experience, with 8+ years of industry experience, the majority of it in performance engineering.
-
Hands-on experience optimizing GPU training or inference workloads in production, at multi-node scale.
-
A track record of diagnosing performance problems in real production workloads and landing the fixes — often in systems owned by other teams — with the gain measured and verified in production.
-
In-depth understanding of computer architecture and high-performance computing.
-
Strong C++ and Python, with a solid command of data structures and algorithms.
-
Hands-on experience writing and tuning CUDA or Triton kernels.
-
Fluent in PyTorch, with concrete insight into what makes PyTorch code fast or slow — graph breaks, host-side overhead, synchronization points and memory layout.
-
Training at the scale of thousands of accelerators, including multi-node failure modes and long-running job stability.
-
Building profiling, tracing or benchmarking infrastructure that other teams depend on — CUPTI, Perfetto, or equivalent.
-
Using LLM agents to automate parts of an engineering workflow.
-
Autonomous driving, robotics, or another domain with hard latency, power and memory constraints.
-
Publications or open-source contributions in ML systems or performance.
-
A fun, supportive and engaging environment.
-
Infrastructures and computational resources to support your work.
-
Opportunity to work on cutting edge technologies with the top talents in the field.
-
Opportunity to make a significant impact on the transportation revolution by the means of advancing autonomous driving.
-
Competitive compensation package.
-
Snacks, lunches, dinners, and fun activities.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- CO

Sr. Staff ML Infra Engineer - Offline Infra & TrainingRecently posted Auto Apply
Coupang · Mountain View · Hybrid · Posted 21 Nov 2026 🇺🇸
GCPScalaAWS+12Salary not listed - PA

Robotics Machine Learning EngineerRecently posted Auto Apply
Pantograph · San Francisco · On-site · Posted 21 Nov 2026 🇺🇸
RoboticsTechnology$150k - $260k/yr - TR

Robotics Intern, Generalist Robot Learning (GRL)Recently posted Auto Apply
Tri · Los Altos · Hybrid · Posted 18 Nov 2026 🇺🇸
RoboticsComputer VisionDeep Learning+4$45 - $65/hr - TT
Senior AI/ML EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 18 Nov 2026 🇺🇸
ScalaC++Java+4Technology$180k - $275k/yr - TT
Founding Research Engineer, Robot LearningRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 18 Nov 2026 🇺🇸
RoboticsEngineering$260k - $300k/yr - AB

Member of Technical Staff, Machine LearningRecently posted Auto Apply
Abridge · SF Office · Hybrid · Posted 18 Nov 2026 🇺🇸
NLPDeep LearningPyTorch+3Technology$205k - $300k/yr - AI

Machine Learning Engineer, Assistance EngineeringRecently posted Auto Apply
Airbnb · United States · Remote · Posted 18 Nov 2026 🇺🇸
MLOpsLLMReact+4Technology$162k - $186k/yr - TT
Founding ML Research EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 18 Nov 2026 🇺🇸
Computer VisionPyTorchPython+1$200k - $260k/yr - CL

Founding Machine Learning EngineerRecently posted Auto Apply
Clera · San Francisco · On-site · Posted 18 Nov 2026 🇺🇸
SQLScikit-learnPandas+2Technology$170k - $200k/yr - AT

Senior Manager, Machine Learning & AI – FinanceRecently posted Auto Apply
Atoms · San Francisco · Hybrid · Posted 18 Nov 2026 🇺🇸
DockerFastAPISQL+12Technology$220k - $250k/yr - 1A

Senior/Staff AI/ML Engineer, Plant IntelligenceRecently posted Auto Apply
1 Arc Power · Mountain View · Hybrid · Posted 18 Nov 2026 🇺🇸
SQLLinearPython+2Technology$237k - $257.5k/yr - TT
Member of Technical Staff, ML SystemsRecently posted Auto Apply
True Talent Solutions Partners · Menlo Park · On-site · Posted 18 Nov 2026 🇺🇸
CUDADeep LearningLLM+1$180k - $260k/yr
