Quick overview
Job description
Responsibilities
- Build, curate, and maintain agentic workflows along with datasets for training and evaluation.
- Run systematic LLM evaluations, track regressions, and ensure models meet quality bars.
- Define and implement LLM performance metrics(e.g., correctness, latency, hallucination control, safety).
- Work closely with Product and Customer-facing teams, experiment with prompting techniques, fine-tuning datasets, retrieval strategies, and model configurations and ensure agent behaviour aligns with user expectations.
- Develop internal tooling to speed up evaluation, annotation, and iteration cycles.
- Build automated pipelines for regression checks and model monitoring.
- Create mechanisms that turn real user interactions into actionable model improvements.
- Ensure agents behave consistently across large-scale production scenarios.
- Debug complex system behaviours spanning prompts, tools, APIs, and model responses.
- To understand real-world use cases and failure modes. Translate customer insights into model needs, data requirements, and product improvements.
- Tune latency, turn-taking, and conversational naturalness for voice AI systems.
- Write clean, reliable code in Python or JS for model pipelines, tools, and integrations.
- Think in systems: from data ingestion to model outputs to user-facing behaviour.
Success Looks Like
- Strong evaluation coverage with clear, actionable metrics.
- Faster iteration cycles due to improved internal tools and workflows.
- Measurable improvement in agent accuracy, consistency, and safety.
- Reliable agent performance in production — fewer escalations, fewer regressions.
- Clear alignment between customer needs and agent capabilities.
- Smooth collaboration across engineering, product, and customer teams.
What We’re Looking For
- 2–5 years experience in software engineering, with strong fundamentals in building reliable, scalable systems.1–2 years of experience working on AI-backed products(LLM engineering, evaluations, prompting, or agent development).
- Hands-on experience with LLMs, prompt engineering, evaluation frameworks, or dataset creation.
- Strong understanding of how LLMs work: prompting, fine-tuning, evaluation, guardrails, and system design.
- Solid software engineering skills and comfort with APIs and cloud environments.
- Experience building internal tools or automation pipelines.
- Ability to think from the customer backward: empathetic, detail-oriented, and outcome-driven.
- Excellent communicator who collaborates well across cross-functional teams.
- Fast learner who loves turning cutting-edge research into practical, dependable systems.
- Bonus: experience with voice AI, speech technologies, or real-time agent systems.
Why This Role Matters
- In healthcare, accuracy and trust are everything. Patients, providers, and payers rely on our AI agents to handle critical interactions. Deployment Engineers make this possible by turning AI from a raw model into something practical, reliable, and human-friendly.
Why 100ms.ai
- You'll be part of a small team at a fast-growing engineering-first startup.
- You'll work with engineers across the globe with experience in video at places like Facebook and Hotstar.
- You can grow as an individual contributor or as a team leader - freedom to set your own goals.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- CI
AI Specialist II Auto Apply
Cousera Internal Rehire Job Board · Mumbai · Hybrid · Posted 4 Nov 2026 🇮🇳
OnboardingSQLNLP+12Salary not listed - WI

AI Solutions Delivery Specialist Auto Apply
Within.ai · India · On-site · Posted 20 Oct 2026 🇮🇳
Stakeholder ManagementMental HealthServiceNow+2Salary not listed - CI
Enterprise Agentic AI Architect Auto Apply
Cousera Internal Rehire Job Board · India · Hybrid 🇮🇳
OnboardingContinuous ImprovementLLM+5TechnologySalary not listed - BG
AI ArchitectRecently posted Auto Apply
Bosch Group · bengaluru · On-site · Posted 15 Nov 2026 🇮🇳
DockerGCPMongoDB+23TechnologySalary not listed - NS
AI DeveloperRecently posted Auto Apply
Nemetschek SE2 · Bengaluru, Karnataka · Hybrid · Posted 15 Nov 2026 🇮🇳
Generative AIPrompt EngineeringTechnologySalary not listed - SI
Principal AI EngineerRecently posted Auto Apply
SAP IT Business Systeme · Bengaluru · Hybrid · Posted 13 Nov 2026 🇮🇳
Generative AITechnologySalary not listed - PA
AI EngineerRecently posted Auto Apply
Pavago · India · Remote · Posted 13 Nov 2026 🇮🇳
GCPAWSNLP+8TechnologySalary not listed - MO

AI EngineerRecently posted Auto Apply
Motive · Bengaluru · Hybrid · Posted 13 Nov 2026 🇮🇳
SnowflakeC++LLM+3TechnologySalary not listed - OI

Gen AI Lead/ArchitectRecently posted Auto Apply
Orion Innovation · India · Hybrid · Posted 13 Nov 2026 🇮🇳
MicroservicesAWSSnowflake+10Salary not listed - TR
R2R Domain Lead – AI Agents (Project-Based Contract | 3–6 Months)Recently posted Auto Apply
Trailhq · India · Remote · Posted 13 Nov 2026 🇮🇳
SQLMicrosoft ExcelCPA+7Salary not listed - NI
Senior Engineer with Python, Agentic AI, Multi-cloud, MCP along with IAM, Cyber experienceRecently posted Auto Apply
NielsenIQ · Chennai, TN · Hybrid · Posted 13 Nov 2026 🇮🇳
Continuous ImprovementTechnologySalary not listed - SI
Agentic AI BuilderRecently posted Auto Apply
Sia · Mumbai, Maharashtra · Hybrid · Posted 12 Nov 2026 🇮🇳
Generative AIAgilePrompt EngineeringEngineeringSalary not listed
