Quick overview
Job description
About the Role
Build datasets, evaluation rubrics, and infrastructure that help improve advanced AI models. You will work with research teams to run experiments, identify model failure modes, and turn training objectives into practical data and evaluation systems.
What You'll Do
Design data slices that reveal meaningful model failure modes across finance, code, and enterprise workflows.
Build and refine evaluation rubrics and reward signals for RLHF and RLVR pipelines.
Model annotator behavior and run experiments to improve model capabilities.
Develop quantitative measures of dataset quality, diversity, and impact on model alignment and capabilities.
Create and manage real-world and synthetic data pipelines.
Partner with AI research teams to translate training goals into concrete data and evaluation specifications.
What We're Looking For
One to four years of software engineering experience building reliable data pipelines, experimentation infrastructure, or evaluation systems.
Strong Python skills and experience shipping software and infrastructure.
Hands-on experience with LLM evaluations, model post-training, reward modeling, RLHF, RLVR, or related ML infrastructure.
Experience creating or evaluating datasets, including assessing quality, diversity, coverage, and downstream model performance.
Strong experimental judgment: able to form hypotheses, define metrics, design controlled experiments, and interpret noisy results.
Experience designing data collection strategies and evaluation rubrics for machine learning models.
Compensation & Benefits
Base salary: $260,000 to $290,000 annually. Visa sponsorship is available.
Location
On-site in San Francisco, California.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- PT
Splunk Security EngineerRecently posted Auto Apply
Pt78 · Tampa · On-site · Posted 19 Nov 2026 🇺🇸
SplunkBash.NET+7Technology$145k - $155k/yr - AR

Senior Platform EngineerRecently posted Auto Apply
Artafinance · Bay Area · Hybrid · Posted 20 Nov 2026 🇺🇸
MicroservicesKafkaPython+2Technology$180k - $300k/yr - AL
Application Support Engineer (Remote) - USRecently posted Auto Apply
AlphaSights · Remote · Remote · Posted 19 Nov 2026 🇺🇸
Technology$80k/yr - IN

Service Desk AnalystRecently posted Auto Apply
Impact Networking · Lake Forest · On-site · Posted 19 Nov 2026 🇺🇸
Talent AcquisitionWorkforce PlanningMicrosoft Office+4Technology$20k/yr - TT
Senior Software Engineer, Autonomous Systems (C2/Backend/Service)Recently posted Auto Apply
True Talent Solutions Partners · Palo Alto · On-site · Posted 20 Nov 2026 🇺🇸
MicroservicesRustRobotics+4Technology$145k - $250k/yr - TT
Software Engineer, C++ SystemsRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 20 Nov 2026 🇺🇸
RustC++Kubernetes+1Technology$200k - $250k/yr - TC

Senior Platform EngineerRecently posted Auto Apply
Thyme Care · Remote · Remote · Posted 19 Nov 2026 🇺🇸
DockerRustGit+9Technology$170k - $200k/yr - TT
Lead Platform EngineerRecently posted Auto Apply
True Talent Solutions Partners · South San Francisco · On-site · Posted 19 Nov 2026 🇺🇸
DjangoFastAPIAWS+4Technology$205k - $235k/yr - TT
Technical Product ManagerRecently posted Auto Apply
True Talent Solutions Partners · Sunnyvale · On-site · Posted 19 Nov 2026 🇺🇸
SnowflakeAirflowDatabricks+4Technology$150k - $250k/yr - TT
Founding Full-Stack EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 19 Nov 2026 🇺🇸
DockerGCPAWS+7Technology$200k - $275k/yr - TT
Robotics Test EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 19 Nov 2026 🇺🇸
RoboticsC++Python+1Technology$182k - $230k/yr - TT
Machine Learning Infrastructure EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 19 Nov 2026 🇺🇸
DockerGCPAWS+8Technology$200k - $400k/yr