Quick overview
Job description
About the Role
You'll own our inference and model-serving infrastructure end to end. This isn't a research role. It's a build role: you're setting up and scaling the systems that let our agents actually run in production, fast and reliably, at increasing concurrency.
You report to Sofus and work closely with our ML and infra teams.
What You’ll Own
- Set up and scale inference/Ray Serve for ML and LLM model serving, integrated with our data analysis and agent workflows
- Scale agent GPU infrastructure for concurrency and efficiency across multiple agent workloads
- Optimize and improve the engine builder and model server that power scalable agent orchestration
What You Need
- Proven ability to build scalable ML/AI platforms from scratch, end-to-end, for production use cases. You've owned a zero-to-one build before, or can show you're capable of it
- Deep understanding of the inference stack: vLLM, KV cache, and the optimization layers underneath model serving
- Experience building distributed systems for AI/ML workloads at scale, connecting them to real product or vertical integrations
- 3+ years of relevant experience. We care about capability, not tenure
Nice to Have
- Ray / Ray Serve experience
- Familiarity with AIBrix
Why Join
- A rare chance to shape both company and product direction as an early team engineer
- Work alongside engineers and researchers from LinkedIn, Visa, Meta, and Branch
- Onsite culture in San Mateo, built for deep collaboration and high-velocity building
- Full benefits (medical, dental, vision, 401k)
- We sponsor H-1B visas and assist with immigration
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- TT
Senior AI/ML EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 17 Nov 2026 🇺🇸
DockerGCPLLM+7Technology$180k - $275k/yr - RG
Machine Learning EngineerRecently posted Auto Apply
RZR Global Inc. · San Francisco · Hybrid · Posted 17 Nov 2026 🇺🇸
RustSQLFlink+10TechnologySalary not listed - SI

Principal AI/ML Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · Remote - No Preference · Remote · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerGCP+23TechnologySalary not listed - SI

Principal AI/ML Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · NYC - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerGCP+23TechnologySalary not listed - SI

Principal AI/ML Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · Bay Area - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerGCP+23TechnologySalary not listed - SI

Head of AI/ML (Director/VP) - VC Backed StartupsRecently posted Auto Apply
Signalfire · Remote - No Preference · Remote · Posted 17 Nov 2026 🇺🇸
RecruitingTalent AcquisitionWorkforce Planning+22TechnologySalary not listed - SI

Head of AI/ML (Director/VP) - VC Backed StartupsRecently posted Auto Apply
Signalfire · NYC - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
RecruitingTalent AcquisitionWorkforce Planning+22TechnologySalary not listed - SI

Head of AI/ML (Director/VP) - VC Backed StartupsRecently posted Auto Apply
Signalfire · Bay Area - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
RecruitingTalent AcquisitionWorkforce Planning+22TechnologySalary not listed - PH

ML Infra Engineer, ComputeRecently posted Auto Apply
Physicalintelligence · San Francisco · On-site · Posted 16 Nov 2026 🇺🇸
GCPCapacity PlanningPyTorch+4Salary not listed - TR
Automated Driving Advanced Development Intern, Machine Learning Research Auto Apply
Tri · Los Altos · Hybrid · Posted 9 Nov 2026 🇺🇸
RoboticsAutonomous DrivingPyTorch+3Technology$65/hr - TT
Member of Technical Staff, Applied MLRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 17 Nov 2026 🇺🇸
Continuous ImprovementLLMPython+1$175k - $350k/yr - TT
Member of Technical Staff, ML SystemsRecently posted Auto Apply
True Talent Solutions Partners · Menlo Park · On-site · Posted 17 Nov 2026 🇺🇸
CUDADeep LearningLLM+2$180k - $260k/yr
