Quick overview
Job description
Help us make inference blazingly fast. If you love squeezing every last drop of performance out of GPUs, diving deep into CUDA kernels, and turning optimization techniques into production systems, we'd love to meet you.
About Inference.net
Inference.net trains and hosts specialized language models for companies that need frontier-quality AI at a fraction of the cost. The models we train match GPT-5 accuracy but are smaller, faster, and up to 90% cheaper. Our platform handles everything end-to-end: distillation, training, evaluation, and planet-scale hosting.
We are a well-funded ten-person team of engineers who work in-person in downtown San Francisco on difficult, high-impact engineering problems. Everyone on the team has been writing code for over 10 years, and has founded and run their own software companies. We are high-agency, adaptable, and collaborative. We value creativity alongside technical prowess and humility. We work hard, and deeply enjoy the work that we do. Most of us are in the office 4 days a week in SF; hybrid works for Bay Area candidates.
About the Role
You will be responsible for making our inference stack as fast and efficient as possible. Your work spans from implementing known optimization techniques to experimenting with novel approaches, always with the goal of serving models faster and cheaper at scale.
Your north star is inference performance: latency, throughput, cost efficiency, and how quickly we can bring new model architectures into production. You'll work across the full inference stack—from CUDA kernels to serving frameworks—to find and eliminate bottlenecks. This role reports directly to the founding team. You'll have the autonomy, a large compute budget, and technical support to push the limits of what's possible in model serving.
Key Responsibilities
Implement and productionize optimization techniques including quantization, speculative decoding, KV cache optimization, continuous batching, and LoRA serving
Deep dive into inference frameworks (vLLM, SGLang, TensorRT-LLM) and underlying libraries to debug and improve performance
Profile and optimize CUDA kernels and GPU utilization across our serving infrastructure
Add support for new model architectures, ensuring they meet our performance standards before going to production
Experiment with novel inference techniques and bring successful approaches into production
Build tooling and benchmarks to measure and track inference performance across our fleet
Collaborate with applied ML engineers to ensure trained models can be served efficiently
Requirements
2+ years of experience in ML systems, inference optimization, or GPU programming
Strong proficiency in Python and familiarity with C++
Hands-on experience with LLM inference frameworks (vLLM, SGLang, TensorRT-LLM, or similar)
Deep understanding of GPU architecture and experience profiling GPU workloads
Familiarity with LLM optimization techniques (quantization, speculative decoding, continuous batching, KV cache management)
Experience with PyTorch and understanding of how models execute on hardware
Track record of measurably improving system performance
Nice-to-Have
Experience with CUDA programming
Familiarity with serving non-LLM models (TTS, vision, embeddings)
Experience with distributed inference and multi-GPU serving
Contributions to open-source inference frameworks
Experience with Docker and Kubernetes
You don't need to tick every box. Curiosity and the ability to learn quickly matter more.
Compensation
We offer competitive compensation, equity in a high-growth startup, and comprehensive benefits. The base salary range for this role is $220,000 - $320,000, plus equity and benefits, depending on experience.
Equal Opportunity
Inference.net is an equal opportunity employer. We welcome applicants from all backgrounds and don't discriminate based on race, color, religion, gender, sexual orientation, national origin, genetics, disability, age, or veteran status.
If you're excited about making AI inference faster for everyone, we'd love to hear from you. Please send your resume and GitHub to amar@inference.net and/or apply here on Ashby.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- TT

Systems Reliability EngineerRecently posted Auto Apply
The Trade Desk · Boulder · Hybrid · Posted 18 Nov 2026 🇺🇸
MongoDBRubyRust+11Technology$124.9k/yr - TO
Senior Software Engineer, Care AutomationRecently posted Auto Apply
Toast · Remote · Remote · Posted 18 Nov 2026 🇺🇸
MicroservicesGraphQLJava+2Technology$125k - $254k/yr - RL

Principal Flight Software EngineerRecently posted Auto Apply
Rocket Lab Corporation · Littleton · Hybrid · Posted 19 Nov 2026 🇺🇸
RustEmbedded SystemsBackground Checks+3Technology$150k - $225k/yr - RL

Senior Flight Software Engineer IIRecently posted Auto Apply
Rocket Lab Corporation · Littleton · Hybrid · Posted 19 Nov 2026 🇺🇸
RustEmbedded SystemsBackground Checks+3Technology$131k - $190k/yr - RL

Senior Flight Software Engineer IRecently posted Auto Apply
Rocket Lab Corporation · Littleton · Hybrid · Posted 19 Nov 2026 🇺🇸
RustEmbedded SystemsC+++2Technology$150k - $220k/yr - SM

Staff Software Engineer, BackendRecently posted Auto Apply
Scout Motors · San Ramon · Hybrid · Posted 18 Nov 2026 🇺🇸
MicroservicesComplianceJava+3Technology$165k - $200k/yr - MB

Senior Software Engineer, Lab ApplicationsRecently posted Auto Apply
Manifold Bio · Boston · Hybrid · Posted 18 Nov 2026 🇺🇸
EchoFastAPIAWS+9TechnologySalary not listed - KP

Firmware Development EngineerRecently posted Auto Apply
Kairos Power · Alameda · Hybrid · Posted 19 Nov 2026 🇺🇸
Embedded SystemsFPGAVHDL+1Engineering$92.5k - $108.8k/yr - GU

Staff Software Engineer, Onboarding PlatformRecently posted Auto Apply
Gusto, Inc. · San Francisco · Hybrid · Posted 19 Nov 2026 🇺🇸
OnboardingRubyLLM+4Technology$197k - $247k/yr - DU

Senior Software Engineer, Motion Planning – DoorDash Labs, DoorDash AirRecently posted Auto Apply
DoorDash USA · San Francisco · Hybrid · Posted 19 Nov 2026 🇺🇸
Microsoft ExcelRoboticsCompliance+5Technology$168k/yr - BP

Embedded Software EngineerRecently posted Auto Apply
Base Power · Austin · On-site · Posted 19 Nov 2026 🇺🇸
RustEmbedded SystemsC+++2TechnologySalary not listed - AI

Software Engineer, AgentsRecently posted Auto Apply
Adaptive Innovations · New York · Hybrid · Posted 19 Nov 2026 🇺🇸
DjangoDockerGCP+14TechnologySalary not listed
