Quick overview
Job description
Team owns the layer between clients and backend workloads: proxy and L7 services, API Gateway, and inference gateway for AI traffic. Every request to backend services passes through this layer, so latency, availability, and correctness here directly affect everything downstream. You'll build and grow the team, set its technical direction, and own these services end to end, from design through production operations.
- Proxy and L7: request routing, TLS termination, connection management, config propagation across a large fleet, and keeping tail latency predictable under load.
- API Gateway: authentication and authorization, rate limiting and quotas, traffic policies, and giving service teams a safe, self-serve way to expose APIs.
- Inference gateway: routing AI inference traffic, which behaves differently from typical web traffic. Requests are long-running and often streamed, cost varies widely by request, and backend GPU capacity is expensive and limited. That makes routing, queuing, fairness, and capacity efficiency hard problems.
- Build and grow the team: hire engineers and tech leads, set expectations, give direct feedback, and develop people over time.
- Define the team's roadmap and priorities with leadership and partner teams.
- Stay close enough to the technical work to review designs, weigh tradeoffs, and make calls when needed.
- Own the operational health of the services: on-call, incident response, postmortems, and follow-through on reliability work.
- Work closely with networking, security, platform, and ML infrastructure teams on problems that cross team boundaries.
- Experience managing engineers who build and operate distributed systems in production.
- A solid technical background in at least one of: proxies and load balancing, API gateways, service mesh, networking, or high-throughput backend services.
- Experience running latency-sensitive services, including on-call and incident management.
- Experience building something new, such as starting a team, launching a new service, or taking a product from early design to production.
- A track record of hiring well and helping engineers grow.
- Hands-on experience with Envoy, NGINX or similar.
- Familiarity with LLM inference serving (batching, streaming, GPU utilization).
- Experience building internal platforms used by many engineering teams.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- DA

Engineering Manager (Ingestion)Recently posted Auto Apply
Databricks · Bengaluru · Hybrid · Posted 20 Nov 2026 🇮🇳
Microsoft ExcelUnityEngineeringSalary not listed - DA

Engineering Manager - Core ExperiencesRecently posted Auto Apply
Databricks · Bengaluru · Hybrid · Posted 20 Nov 2026 🇮🇳
Microsoft ExcelUnityEngineeringSalary not listed - AD

Software Engineering Manager(Backend)Recently posted Auto Apply
Addepar · Pune · Hybrid · Posted 20 Nov 2026 🇮🇳
DockerSQLSpring+7TechnologySalary not listed - MO

Director - EngineeringRecently posted Auto Apply
Monks · India · Hybrid · Posted 19 Nov 2026 🇮🇳
OnboardingComplianceDigital Marketing+3EngineeringSalary not listed - C

Manager, EngineeringRecently posted Auto Apply
ConnectWise · Mumbai · Hybrid · Posted 19 Nov 2026 🇮🇳
EngineeringSalary not listed - BA

(Services) Engineering DirectorRecently posted Auto Apply
Backbase · Hyderabad · Hybrid · Posted 19 Nov 2026 🇮🇳
EngineeringSalary not listed - NR

Manager, Software EngineeringRecently posted Auto Apply
New Relic · Bengaluru · Hybrid · Posted 17 Nov 2026 🇮🇳
GCPAWSEncryption+15TechnologySalary not listed - SE
Sr Mgr, Software Engrg Mgmt - MoveworksRecently posted Auto Apply
ServiceNow · Bangalore · On-site · Posted 17 Nov 2026 🇮🇳
PythonC++Go+3TechnologySalary not listed - NI
Senior Manager, EngineeringRecently posted Auto Apply
NielsenIQ · Chennai, TN · Hybrid · Posted 17 Nov 2026 🇮🇳
SAFeStakeholder ManagementContinuous Improvement+1EngineeringSalary not listed - SC

Software Engineer (Backend) - WWE ChampionsRecently posted Auto Apply
Scopely · Bengaluru · Hybrid · Posted 19 Nov 2026 🇮🇳
DjangoDockerDynamoDB+5TechnologySalary not listed - ST

Engineering Manager, Core InfrastructureRecently posted Auto Apply
Stripe · Bengaluru · Hybrid · Posted 16 Nov 2026 🇮🇳
RubyAWSCapacity Planning+3Salary not listed - SU

Senior Manager, EngineeringRecently posted Auto Apply
Surveymonkey · Bengaluru · Hybrid · Posted 16 Nov 2026 🇮🇳
RecruitingSourcingBranding+6EngineeringSalary not listed
