Quick overview
Job description
In this role, you will:
-
Design, implement, and operate cutting-edge ML training performance optimization techniques — distributed training (data/tensor/pipeline parallelism), mixed-precision, gradient checkpointing, and GPU kernels — to scale our VLM, VLA, and Foundational models training efficiency.
-
Work on RL framework and RL performance optimization for driving foundation model
-
Profile and debug training throughput bottlenecks across the compute, compiler, framework, and cluster-networking stack.
-
Collaborate closely with ML researchers, data engineers, and hardware engineers to define requirements and align on architectural decisions for large-scale training infrastructure.
-
Implement custom kernels and build benchmarks to track and validate training-performance optimizations across the stack.
Qualifications
-
4+ years of total experience, including 2+ years working on large-scale model training platforms.
-
Experience with distributed training frameworks (Megatron-LM, TorchTitan), efficiently utilizing GPU/accelerator clusters at scale.
-
Experience with RL framework such as Verl/MILES and RL algorithms(GRPO, PPO)
-
Experience with collective communication libraries (e.g., NCCL) and optimizing communication/compute overlap in multi-node training.
-
Experience using profiling tools (NVIDIA Nsight, PyTorch Profiler, or XLA/JAX profiling) to identify and resolve training bottlenecks.
-
Proficient in Python or C++.
-
Experience with ML compilers, performance tooling, or hardware/software co-design (e.g., GPU/TPU/accelerator architectures).
Note: You do not have to meet all the requirements below to be considered for this position:
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- TT
Founding Senior EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 16 Nov 2026 🇺🇸
LLMPythonTypeScript+1Engineering$175k - $225k/yr - TT
Robotics Software EngineerRecently posted Auto Apply
True Talent Solutions Partners · Watertown · On-site · Posted 16 Nov 2026 🇺🇸
DockerEmbedded SystemsRobotics+6Technology$130k - $180k/yr - TT
Forward Deployed Infrastructure EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 16 Nov 2026 🇺🇸
DockerGCPAWS+6Technology$200k - $300k/yr - TT
Senior/Staff Software EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 16 Nov 2026 🇺🇸
RustAWSERP+5Technology$180k - $270k/yr - TT
Senior Forward Deployed Engineer, OperateRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 16 Nov 2026 🇺🇸
GCPNext.jsAWS+15Engineering$170k - $205k/yr - TT
Senior Software Engineer, Distributed Systems (Core)Recently posted Auto Apply
True Talent Solutions Partners · Remote · Remote · Posted 16 Nov 2026 🇺🇸
DockerSQLFlink+2Technology$180k - $320k/yr - TT
Software Engineer, C++ SystemsRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
RustC++Kubernetes+1Technology$250k - $325k/yr - TT
Robotics Infrastructure EngineerRecently posted Auto Apply
True Talent Solutions Partners · Pittsburgh · On-site · Posted 15 Nov 2026 🇺🇸
DockerGCPAWS+5Technology$100k - $300k/yr - TT
Software Engineer, iOSRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 15 Nov 2026 🇺🇸
SwiftiOSRESTTechnology$130k - $230k/yr - TT
Founding Product EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 15 Nov 2026 🇺🇸
Next.jsHTMLFigma+4$160k - $220k/yr - TT
Founding EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
OnboardingAWSLLM+4Engineering$180k - $220k/yr - TT
GTM EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
Generative AIn8nEngineering$140k - $180k/yr
