Quick overview
Job description
About the Role
Sarvam's model APIs (LLM, ASR, TTS, Vision, Translate) are how developers and enterprises ship products on our foundation models. Our OpenAI-compatible LLM APIs route every request across multiple model deployments, with streaming, failover, metering, billing, and rate-limiting all part of the same stack. Today it's FastAPI and Python; we're in the middle of rewriting it in Go.
We're hiring a Software Engineer (SDE 2) to own core parts of this platform. The features. The reliability. The performance. This is an IC role.
What You'll Do
Building and evolving the LLM API platform, from the moment a request hits the edge to the response that goes back out
The Python→Go rewrite: building services in Go alongside the existing Python stack, without compromising reliability
Streaming infrastructure: SSE for Chat Completions and Responses, long-lived connections, clean handling of cancellations and client disconnects
Routing and failover: weighted provider chains, retries with backoff, and latency-driven admission control across model deployments
Protocol compatibility: keeping the API faithful to the OpenAI spec across inference servers (vLLM, SGLang, Dynamo) that each behave slightly differently
The commercial layer: metering, billing, and rate-limiting, engineered to the same bar as the model APIs themselves
Performance and reliability for a multi-tenant platform serving long-context agentic and coding workloads
Observability (logging, metrics, tracing), so the team can debug production confidently and quickly
Integration and load testing: the harness that lets us ship fast without breaking customer-facing APIs
Close partnership with the Inference and MLOps teams on production readiness for new model launches
Design-first, documentation-first engineering culture within the team: RFCs before code, decisions written down, no tribal knowledge
What We're Looking For
3+ years building production backend systems
Strong Python in production: asyncio, FastAPI or similar, and services you've built and operated under real load
Working knowledge of Go, or a strong willingness to learn it: you'll work in both languages through the rewrite
Experience building low-latency, multi-tenant backend services
Hands-on experience with streaming systems: SSE or WebSockets, long-lived connections, backpressure
Strong PostgreSQL and Redis fundamentals: schema design, query performance, caching, and rate limiting
Comfort running services on Kubernetes in production
An ownership mindset: you test before you ship, debug production with metrics, logs, and traces, and write the RCA afterwards
Bonus Points
Experience serving or integrating LLMs in production, especially OpenAI-compatible APIs
Familiarity with inference servers (vLLM, SGLang, Dynamo) and latency metrics such as time to first token and inter-token latency
Experience building metering, billing, or usage-based pricing infrastructure
Time at an early- or growth-stage startup: comfort with ambiguity, speed, and ownership
Why Sarvam?
Sarvam is a fast-moving, high talent-density team building full-stack AI for India, working on problems that push the frontiers of AI with real population-scale impact.
Work alongside researchers, engineers, builders, and business leaders who move fast and hold each other to a very high bar
High ownership and high impact, from day one
Everything we do is AI-first, from the way we build and ship to the way we think about problems
You can work on problems that could change how an entire country learns, works, and communicates
If you want to work on problems at the frontier of AI in India, Sarvam is the place to be.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- CA

Backend EngineerRecently posted Auto Apply
Capco · Pune · Hybrid · Posted 17 Nov 2026 🇮🇳
OracleRubyRust+19TechnologySalary not listed - AT

Backend Engineer AIRecently posted Auto Apply
Atlys · Delhi · On-site · Posted 18 Nov 2026 🇮🇳
DjangoDockerFastAPI+13TechnologySalary not listed - BG
AI & Backend Software Engineer – Intellectual Property Rights(Python / FastAPI / GenAI)Recently posted Auto Apply
Bosch Group · bengaluru · Hybrid · Posted 18 Nov 2026 🇮🇳
DjangoFastAPIFlask+19TechnologySalary not listed - ZS

Staff Software Development Engineer (Backend Dev - C/Linux)Recently posted Auto Apply
Zscaler · Bengaluru · Hybrid · Posted 19 Nov 2026 🇮🇳
CZero TrustAgile+1TechnologySalary not listed - ZS

Staff Software Development Engineer (Backend Dev - C/C++)Recently posted Auto Apply
Zscaler · Mohali · Hybrid · Posted 19 Nov 2026 🇮🇳
GitCZero Trust+2TechnologySalary not listed - CA

Java Backend Developer with Kubernetes - VadodaraRecently posted Auto Apply
Capco · Pune · Hybrid · Posted 18 Nov 2026 🇮🇳
DockerMicroservicesSQL+15TechnologySalary not listed - CA

Java Backend Developer with Kubernetes - VadodaraRecently posted Auto Apply
Capco · Pune · Hybrid · Posted 18 Nov 2026 🇮🇳
DockerMicroservicesSpring+14TechnologySalary not listed - AD

Backend Engineer (Java)Recently posted Auto Apply
Adyen · Bengaluru · Hybrid · Posted 17 Nov 2026 🇮🇳
SQLJavaREST+1TechnologySalary not listed - OI

Senior Java DevRecently posted Auto Apply
Orion Innovation · Pune · Hybrid · Posted 18 Nov 2026 🇮🇳
MicroservicesSpringSpring Boot+10Salary not listed - BI

Backend Engineer E2 - EcosystemRecently posted Auto Apply
BitGo · India · Hybrid · Posted 16 Nov 2026 🇮🇳
MicroservicesMongoDBGit+7TechnologySalary not listed - AC

Senior AI Engineer – Backend Engineering (Agentic AI) Auto Apply
Acceldata · Bengaluru · On-site 🇮🇳
WorkdayDockerMicroservices+15TechnologySalary not listed - CA

Java Backend Developer - Vadodara/ PuneRecently posted Auto Apply
Capco · Pune · Hybrid · Posted 15 Nov 2026 🇮🇳
MicroservicesSpringSpring Boot+2TechnologySalary not listed
