Quick overview
Job description
About the Role
Lead the data quality team at an early-stage AI infrastructure company focused on reinforcement learning and AI evaluation. You will shape how training data is measured and improved, helping build reliable systems for evaluating agent tasks and producing useful post-training data.
What You'll Do
Lead systems that evaluate tasks across reinforcement learning environments, synthetic data, benchmarks, and domain-specific workflows.
Set data quality strategy by defining standards, building quality-control systems, and designing experiments to assess agent outputs.
Develop scalable synthetic data validation methods, including failure-mode analysis, task mutation checks, and trajectory audits.
Work with research engineers, subject-matter experts, and data vendors to identify quality issues and improve data generation workflows.
Turn research insights into production tools, dashboards, validation pipelines, and feedback loops.
Mentor research engineers and promote rigorous, clear, and efficient technical work.
What We're Looking For
At least 5 years of relevant research or engineering experience, including building AI or machine learning data evaluation systems.
Experience leading technical projects or teams from problem definition through implementation and iteration.
Advanced proficiency in Python, Docker, and Linux, with a technical academic or professional background.
Strong understanding of AI evaluations, post-training, and the qualities that make agent training data realistic, learnable, diverse, reliable, and useful.
Experience with synthetic data validation, evaluation infrastructure, benchmarks, metrics, experiments, and quality-control workflows.
Ability to translate research into production systems and collaborate with domain experts to scale their judgment into review or generation processes.
Clear written communication, attention to detail, and the ability to work independently in an early-stage startup environment.
Compensation & Benefits
Base salary: $130,000 to $225,000 annually. Visa sponsorship is available.
Location
On-site in the United States.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- IE
CAM EngineerRecently posted Auto Apply
Industrial Electric Manufacturing · Tualatin · Hybrid · Posted 20 Nov 2026 🇺🇸
AutoCADCADCAM+4EngineeringSalary not listed - CO

Senior Technical Program Manager, Assurance EngineeringRecently posted Auto Apply
CoreWeave · New York · Hybrid · Posted 22 Nov 2026 🇺🇸
SQLSpringGoogle Workspace+9Engineering$157k - $210k/yr - OP

Research Engineer / Research Scientist, HealthRecently posted Auto Apply
Openai · San Francisco · Hybrid · Posted 22 Nov 2026 🇺🇸
Clinical ResearchGenerative AIMachine LearningEngineering$380k - $500k/yr - OP

Research Engineer / Research Scientist, Human FlourishingRecently posted Auto Apply
Openai · San Francisco · Hybrid · Posted 22 Nov 2026 🇺🇸
LLMGenerative AIEngineering$380k - $500k/yr - OP

Research Engineer / Research Scientist - Post Training, MemoryRecently posted Auto Apply
Openai · San Francisco · Hybrid · Posted 22 Nov 2026 🇺🇸
Generative AIEngineering$380k - $500k/yr - OP

Research Engineer/Research Scientist- Post Training, Model ExperienceRecently posted Auto Apply
Openai · San Francisco · Hybrid · Posted 22 Nov 2026 🇺🇸
Generative AIMachine LearningEngineering$380k - $500k/yr - OP

Research Engineer/Research Scientist-Post Training, ProactivityRecently posted Auto Apply
Openai · San Francisco · Hybrid · Posted 22 Nov 2026 🇺🇸
LLMGenerative AIMachine LearningEngineering$380k - $500k/yr - GR

Forward Deployed EngineerRecently posted Auto Apply
Graymatter Robotics · United States · Remote · Posted 22 Nov 2026 🇺🇸
MATLABRoboticsROS+1Engineering$90k - $300k/yr - CL

Founding Engineering LeadRecently posted Auto Apply
Clera · New York · Remote · Posted 22 Nov 2026 🇺🇸
LLMMachine LearningEngineering$200k - $400k/yr - CL

Founding Harness EngineerRecently posted Auto Apply
Clera · San Francisco · On-site · Posted 22 Nov 2026 🇺🇸
LLMEngineering$200k/yr - RE

Solutions Engineering ManagerRecently posted Auto Apply
Resolveai · San Francisco · Hybrid · Posted 22 Nov 2026 🇺🇸
GitAWSLinear+6EngineeringSalary not listed - PE

Supplier Quality Engineer IRecently posted Auto Apply
Penumbrainc · Alameda · On-site · Posted 22 Nov 2026 🇺🇸
Microsoft ExcelMicrosoft PowerPointCompliance+2Engineering$76k - $100k/yr