Why this role stands out
This role offers a unique opportunity to shape the future of AI inference by building cutting-edge systems for novel hardware, presenting significant growth and learning potential. You will thrive here if you are passionate about solving complex technical challenges in distributed systems and ML infrastructure, and you will be instrumental in optimizing performance for leading AI companies. Apply now to be at the forefront of AI innovation and contribute to groundbreaking advancements in the field.
Quick overview
Job description
About us
Gimlet is building the first multi-silicon neocloud designed for fast, efficient AI inference.
We combine large-scale compute infrastructure with an execution platform that partitions AI workloads and maps each stage to the hardware best suited to run it.
We work with foundation labs, hyperscalers, and AI-native companies, giving our team access to technical problems spanning frontier models, production infrastructure, and emerging hardware.
About the role
As a Member of Technical Staff focused on ML Systems, you will build the inference systems that execute models end-to-end in production.
You will work on the systems that determine how inference executes across that pipeline: how requests are batched and scheduled, how stages are placed and scaled, how KV cache and intermediate state move between accelerators, and how the system balances latency, throughput, and utilization across different hardware characteristics.
You will work across model serving, batching, scheduling, concurrency, KV cache management, and memory placement. You will help bring up models on novel hardware. You will support new model architectures and inference techniques, improve performance under real production workloads, and partner with compiler, kernel, networking, and distributed systems engineers to optimize the full execution path.
What success looks like
In the first 12-18 months, you will:
Improve the latency, throughput, and efficiency of production inference workloads
Design execution strategies across batching, scheduling, concurrency, and resource utilization
Improve KV cache management, memory efficiency, and execution under load
Enable new models, accelerator architectures, and inference techniques to run efficiently in production
You may be a good fit if you have
Strong software engineering fundamentals
Experience building or operating ML inference or model serving systems
Comfort reasoning about performance, memory usage, and system behavior under load
Bachelor's degree in a relevant field, or an equivalent combination of education, training, and professional experience.
Strong candidates may also have
Experience with inference runtimes such as TensorRT-LLM, vLLM, or custom serving systems
Deep understanding of modern model architectures and attention mechanisms
Experience with batching, scheduling, and concurrency control in inference systems
Familiarity with KV cache management and memory placement strategies
Experience profiling and tuning latency- and throughput-critical systems
Software development experience in Python and C++
Why join now?
Gimlet is expanding from its core technology into a production neocloud spanning new hardware, customers, and data centers.
Solve hard problems.
Own meaningful work.
Build for production.
Help define what’s next.
Agency Policy: Gimlet Labs does not accept unsolicited resumes from recruitment agencies or search firms. Any unsolicited resumes submitted without a signed agreement will be considered the property of Gimlet Labs, and no fees will be paid.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- 1A

Senior/Staff AI/ML Engineer, Plant IntelligenceRecently posted Auto Apply
1 Arc Power · Mountain View · Hybrid · Posted 18 Nov 2026 🇺🇸
SQLLinearPython+2Technology$237k - $257.5k/yr - TT
Senior AI/ML EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 17 Nov 2026 🇺🇸
DockerGCPLLM+7Technology$180k - $275k/yr - RG
Machine Learning EngineerRecently posted Auto Apply
RZR Global Inc. · San Francisco · Hybrid · Posted 17 Nov 2026 🇺🇸
RustSQLFlink+10TechnologySalary not listed - SI

Principal AI/ML Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · Remote - No Preference · Remote · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerGCP+23TechnologySalary not listed - SI

Principal AI/ML Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · NYC - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerGCP+23TechnologySalary not listed - SI

Principal AI/ML Engineer - VC Backed StartupsRecently posted Auto Apply
Signalfire · Bay Area - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
Talent AcquisitionDockerGCP+23TechnologySalary not listed - SI

Head of AI/ML (Director/VP) - VC Backed StartupsRecently posted Auto Apply
Signalfire · Remote - No Preference · Remote · Posted 17 Nov 2026 🇺🇸
RecruitingTalent AcquisitionWorkforce Planning+22TechnologySalary not listed - SI

Head of AI/ML (Director/VP) - VC Backed StartupsRecently posted Auto Apply
Signalfire · NYC - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
RecruitingTalent AcquisitionWorkforce Planning+22TechnologySalary not listed - SI

Head of AI/ML (Director/VP) - VC Backed StartupsRecently posted Auto Apply
Signalfire · Bay Area - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
RecruitingTalent AcquisitionWorkforce Planning+22TechnologySalary not listed - PH

ML Infra Engineer, ComputeRecently posted Auto Apply
Physicalintelligence · San Francisco · On-site · Posted 16 Nov 2026 🇺🇸
GCPCapacity PlanningPyTorch+4Salary not listed - TR
Automated Driving Advanced Development Intern, Machine Learning Research Auto Apply
Tri · Los Altos · Hybrid · Posted 9 Nov 2026 🇺🇸
RoboticsAutonomous DrivingPyTorch+3Technology$65/hr - TT
Member of Technical Staff, Applied MLRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 17 Nov 2026 🇺🇸
Continuous ImprovementLLMPython+1$175k - $350k/yr
