Quick overview
Job description
About the Role
Join an early-stage AI infrastructure team building systems that improve the performance and efficiency of open-source model inference. This role spans low-level optimization and production operations, helping deliver fast, reliable inference across diverse hardware.
What You'll Do
Bring new open-source models into production with optimizations for latency and throughput.
Improve inference serving through batching, KV cache management, speculative decoding, and quantization.
Write and tune GPU kernels using CUDA, HIP, and Triton across a range of accelerator platforms.
Design, deploy, and operate heterogeneous clusters across infrastructure providers.
Run production inference across mixed hardware fleets, improving reliability, observability, and cost efficiency.
Work with customers to understand needs and solve infrastructure performance problems.
What We're Looking For
Approximately 1 to 6 years of experience in backend, infrastructure, or systems engineering, including low-level work beyond application development; strong early-career candidates are welcome.
Understanding of computer architecture, memory hierarchies, and operating system internals.
Experience operating infrastructure at scale, such as clusters, data centers, or GPU fleets.
Familiarity with machine learning inference optimization, including quantization, batching, KV caches, or speculative decoding.
Experience with CUDA, Triton, or other GPU programming, with Python and exposure to technologies such as HIP, TPU, or AWS Trainium useful.
Strong CS fundamentals, evidence of rapid learning or ownership of challenging systems, and experience using AI coding tools to improve workflows.
Comfort working directly with customers and taking broad ownership in a small team.
Compensation & Benefits
Base salary: $200,000 to $300,000 annually. Visa sponsorship is available.
Location
On-site in San Francisco, California.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works