Quick overview
Job description
About Us
Meraki Labs (founded by Mukesh Bansal & Peeyush Ranjan) builds and rapidly scales AI-first, "moonshot" startups. We're looking for a high-velocity, production-grade engineer to build 0-to-1 products alongside founders.
About the Role
Proofline is live in production with institutional pilots underway. Building with AI is easy to prototype, but proving reliability in production is a major challenge. In AI-native codebases, verification is the key bottleneck for scaling capabilities. As an AI Evaluation Engineer, you will take ownership of the evaluation scaffolding and quality layer, including eval harnesses for AI-facing features, test infrastructure, and release gates. You’ll play a critical role in monitoring how our product behaves across AI assistants (e.g., Claude, ChatGPT), accounting for differences by host and continual changes.
AI Evaluation Engineer — Evals & Systems Verification
Location: HSR, Bengaluru (On-site)
Experience: 5+ years
Responsibilities
Build and maintain evaluation harnesses for AI-facing features to measure and tune system quality (e.g., capture quality, retrieval quality, guidance quality)
Own end-to-end and API test infrastructure (Playwright-class), supporting a continuous, daily-release cycle
Design and execute host-behavior probes using scripted user sessions across diverse AI assistants, ensuring product behavior aligns with expectations
Gate production releases through thorough user acceptance testing (UAT), regression analysis, and quality reporting
Requirements
Background in SDET/QA automation or ML evaluation, with proven ownership of test or evaluation infrastructure and not just executing tests
Strong programming skills in Python or TypeScript, with experience in API-level testing and tools like Playwright or Cypress
Familiarity with LLM applications or a demonstrated interest in evaluating non-deterministic systems
Highly autonomous and able to define your own workflows and processes for evaluation and verification
How We Work
Small team, high trust, written decisions. Designs get adversarial review before code; PRs get automated review driven to zero open findings; features aren't done until verified on a live system. AI agents do a large share of the implementation—your leverage is judgment: framing the problem, freezing the right design, and knowing when the machine is wrong.
You Should Apply If
• You enjoy turning messy problems into concrete solutions.
• You thrive in fast-moving, low-process environments.
• You're excited about production engineering (monitoring, reliability, cost, latency).
You Should Not Apply If
• You want remote/hybrid (this is onsite Bangalore only).
• You prefer narrow tickets and minimal ambiguity.
• You prefer highly structured, slow-moving product organizations.
The Recruiter for this role is Ritesh Kalvellu. Apply to the role directly, and he will get back to you if your profile is relevant to the hiring requirements.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- GR
AI AssociateRecently posted Auto Apply
group.one · Ahmedabad · Hybrid · Posted 17 Nov 2026 🇮🇳
Digital MarketingGenerative AISalary not listed - TI
Senior Azure AI Developer - Tieto Caretech (m/f/d)Recently posted Auto Apply
Tieto · Pune · Hybrid · Posted 17 Nov 2026 🇮🇳
GitC#.NET+7TechnologySalary not listed - SI
AI Architect (Product Architecture)Recently posted Auto Apply
Sia · Mumbai, Maharashtra · On-site · Posted 17 Nov 2026 🇮🇳
Generative AITechnologySalary not listed - BG
AI ArchitectRecently posted Auto Apply
Bosch Group · bengaluru · On-site · Posted 16 Nov 2026 🇮🇳
DockerGCPMongoDB+23TechnologySalary not listed - BA

AI EngineerRecently posted Auto Apply
Backbase · Hyderabad · Hybrid · Posted 17 Nov 2026 🇮🇳
LLMPythonGenerative AI+1TechnologySalary not listed - SH

Senior AI Product Builder (Pricing)Recently posted Auto Apply
ShipBob, Inc. · India · Remote · Posted 17 Nov 2026 🇮🇳
FigmaCustomer SuccessReconciliation+3Salary not listed - OR

AI Engineer– Microsoft FoundryRecently posted Auto Apply
Orbia · Pune · On-site · Posted 17 Nov 2026 🇮🇳
Microsoft OfficeGitApache Spark+20TechnologySalary not listed - OR

Lead Data & AI Engineer – Databricks Data Analytics & Agent BricksRecently posted Auto Apply
Orbia · Pune · Hybrid · Posted 17 Nov 2026 🇮🇳
SQLGitApache Spark+26TechnologySalary not listed - MA

AI Reliability Engineer (HPC)Recently posted Auto Apply
Microsoft AI · Bengaluru · Hybrid · Posted 17 Nov 2026 🇮🇳
DockerGCPMicrosoft Office+11TechnologySalary not listed - LR
Technical Lead - LLM & AIRecently posted Auto Apply
LRN Corporation · Mumbai, Maharashtra · On-site · Posted 16 Nov 2026 🇮🇳
GitNLPCompliance+6Salary not listed - CI
Senior AI SpecialistRecently posted Auto Apply
Cousera Internal Rehire Job Board · Mumbai · Hybrid · Posted 16 Nov 2026 🇮🇳
OnboardingDockerGCP+12Salary not listed - V1
Senior AI EngineerRecently posted Auto Apply
Version 1 · Bengaluru, KA · Hybrid · Posted 16 Nov 2026 🇮🇳
DockerOracleSQL+21TechnologySalary not listed
