Quick overview
Job description
About the Role
Join a small AI infrastructure team building efficient inference systems for open-source language models. You will optimize models and serving infrastructure across diverse hardware, helping improve performance, reliability, and cost per token.
What You'll Do
Ship support for new open-source models, tuning them for latency and throughput.
Optimize inference serving with techniques such as batching, KV caching, speculative decoding, and quantization.
Write and tune kernels using CUDA, HIP, and Triton across a range of accelerator platforms.
Design, deploy, and operate heterogeneous clusters across infrastructure vendors.
Run production inference across a mixed hardware fleet, improving reliability, observability, and cost efficiency.
Work with customers to understand technical needs and solve infrastructure problems.
What We're Looking For
Typically 1 to 6 years of experience in backend, infrastructure, systems, or low-level engineering, including experience at a technology company or early-stage startup.
Strong computer science fundamentals, including computer architecture, memory hierarchy, and operating system internals.
Experience owning a challenging system end to end, notable technical achievements, or evidence of rapid growth.
Systems engineering experience beyond application-level or full-stack development, with a non-research-focused background.
Familiarity with CUDA, Triton, or GPU programming; experience with HIP, TPU, or AWS Trainium is also relevant.
Knowledge of inference optimization techniques, such as quantization, batching, KV caching, and speculative decoding.
Experience deploying or operating infrastructure at scale, such as clusters or GPU fleets.
Experience using AI agents and coding tools to improve engineering workflows, and comfort communicating with customers.
Strong internships or equivalent hands-on experience are welcome, including for recent graduates.
Compensation & Benefits
Base salary: $200,000 to $300,000 annually. Visa sponsorship is available.
Location
On-site in San Francisco, California, five days per week.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- TT
Product EngineerRecently posted Auto Apply
True Talent Solutions Partners · Palo Alto · On-site · Posted 16 Nov 2026 🇺🇸
SQLAmplitudeMixpanel+3$185k - $205k/yr - TT
Lead Product EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 16 Nov 2026 🇺🇸
SourcingNext.jsFigma+1$225k - $275k/yr - TT
Software EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 15 Nov 2026 🇺🇸
AWSPythonReact Native+2Technology$193k - $258k/yr - TT
Senior Software Engineer, Core SystemsRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
DockerAWSETL+10Technology$215k - $275k/yr - TT
Founding Software EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
LLMProcurementPython+2Technology$200k - $250k/yr - TT
Lead Software Engineer, PlatformRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
RustPythonScheduling+1Technology$220k - $320k/yr - TT
Principal Software Engineer, PerceptionRecently posted Auto Apply
True Talent Solutions Partners · Alameda · On-site · Posted 15 Nov 2026 🇺🇸
RoboticsComputer VisionC+++3Technology$300k - $380k/yr - TT
Forward Deployed EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 14 Nov 2026 🇺🇸
SQLPythonMachine Learning+1Engineering$150k - $275k/yr - TC
Senior Field Operations EngineerRecently posted Auto Apply
Tract Capital Management, LP · Sparks · Hybrid · Posted 16 Nov 2026 🇺🇸
SCADATechnologySalary not listed - SE
Principal Software Engineer Auto Apply
ServiceNow · Santa Clara, California · Hybrid 🇺🇸
JavaScriptJavaTechnology$221.2k - $387.1k/yr - SP
Software Engineer, Super IntelligenceRecently posted Auto Apply
SpaceXSI · Palo Alto · Hybrid · Posted 16 Nov 2026 🇺🇸
RustTCP/IPC+++1Technology$140k - $225k/yr - AU

Sr. Robotic Software EngineerRecently posted Auto Apply
Aureliussystems · San Francisco · On-site · Posted 16 Nov 2026 🇺🇸
RoboticsC++Python+1TechnologySalary not listed