Quick overview
Job description
About the Role
Join a Series A AI training data company building datasets and evaluation systems for frontier models. You will work with research teams to design experiments, identify model failure modes, and create data and metrics that inform model training.
What You'll Do
Design data slices across finance, software development, and enterprise workflows to reveal meaningful model failure modes.
Build and refine evaluation rubrics and reward signals for RLHF and RLVR pipelines.
Model annotator behavior and run experiments to improve model capabilities.
Develop quantitative frameworks to measure dataset quality, diversity, and downstream effects on model alignment and capability.
Create and manage real-world and synthetic data pipelines.
Partner with research teams to translate training objectives into concrete data and evaluation specifications.
What We're Looking For
1 to 4 years of software engineering experience building reliable data pipelines, experimentation infrastructure, or evaluation systems.
Strong Python skills and experience shipping software and infrastructure.
Hands-on experience with LLM evaluations, model post-training, reward modeling, RLHF, RLVR, or related ML infrastructure.
Strong experimental judgment, including forming hypotheses, defining metrics, designing controlled experiments, and interpreting noisy results.
Experience creating or evaluating datasets, measuring quality, diversity, coverage, and downstream model performance.
Experience designing data collection strategies and evaluation rubrics for machine learning models.
Experience in AI safety, benchmarking, model evaluation, or research engineering is beneficial.
Compensation & Benefits
Base salary: $260,000 to $290,000 annually. Visa sponsorship is available.
Location
On-site in San Francisco, California.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- TT
Founding Senior EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 16 Nov 2026 🇺🇸
LLMPythonTypeScript+1Engineering$175k - $225k/yr - TT
Robotics Software EngineerRecently posted Auto Apply
True Talent Solutions Partners · Watertown · On-site · Posted 16 Nov 2026 🇺🇸
DockerEmbedded SystemsRobotics+6Technology$130k - $180k/yr - TT
Forward Deployed Infrastructure EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 16 Nov 2026 🇺🇸
DockerGCPAWS+6Technology$200k - $300k/yr - TT
Senior/Staff Software EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 16 Nov 2026 🇺🇸
RustAWSERP+5Technology$180k - $270k/yr - TT
Senior Forward Deployed Engineer, OperateRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 16 Nov 2026 🇺🇸
GCPNext.jsAWS+15Engineering$170k - $205k/yr - TT
Senior Software Engineer, Distributed Systems (Core)Recently posted Auto Apply
True Talent Solutions Partners · Remote · Remote · Posted 16 Nov 2026 🇺🇸
DockerSQLFlink+2Technology$180k - $320k/yr - TT
Software Engineer, C++ SystemsRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
RustC++Kubernetes+1Technology$250k - $325k/yr - TT
Robotics Infrastructure EngineerRecently posted Auto Apply
True Talent Solutions Partners · Pittsburgh · On-site · Posted 15 Nov 2026 🇺🇸
DockerGCPAWS+5Technology$100k - $300k/yr - TT
Software Engineer, iOSRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 15 Nov 2026 🇺🇸
SwiftiOSRESTTechnology$130k - $230k/yr - TT
Founding Product EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 15 Nov 2026 🇺🇸
Next.jsHTMLFigma+4$160k - $220k/yr - TT
Founding EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
OnboardingAWSLLM+4Engineering$180k - $220k/yr - TT
GTM EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
Generative AIn8nEngineering$140k - $180k/yr