Quick overview
Job description
About the Role
Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. Your work will help AI labs and enterprise teams assess model capabilities and make better-informed decisions about production use.
What You'll Do
Evaluate newly released AI models across a range of use cases.
Build benchmarks from the ground up, including dataset development and labeler coordination.
Develop scalable and accurate automated evaluation methods for generated text.
Work with engineering teams to implement and scale evaluation systems.
Collaborate with AI labs and enterprise teams to translate evaluation needs into research projects.
What We're Looking For
At least 3 years of full-time AI/ML research experience, particularly in benchmarking, evaluation, or language models.
A track record of published research, ideally in NLP or benchmarking, and hands-on experience creating benchmarks or evaluation methods.
Strong Python skills and experience building clean, maintainable software; Django, Flask, or similar Python HTTP server experience is useful.
Experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant deep learning techniques.
Strong computer science fundamentals and the ability to turn research findings into practical systems.
Experience collaborating through team development workflows, Git, code review, and feedback.
A degree in Computer Science, Machine Learning, or a related field; a Master's or PhD is preferred. Candidates with a Bachelor's degree and 1 to 3 years of applied experience may also be considered.
Compensation & Benefits
Base salary: $150,000 to $200,000 annually. Visa sponsorship is available.
Location
On-site in San Francisco, California.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- ST

Chief ScientistRecently posted Auto Apply
STR · Arlington · Hybrid · Posted 13 Nov 2026 🇺🇸
ComplianceGenerative AIMachine Learning+1$265k - $330k/yr - CL

Applied AI ResearcherRecently posted Auto Apply
Clera · New York · Remote · Posted 15 Nov 2026 🇺🇸
Computer VisionDeep LearningPyTorch+1Salary not listed - CA

Scientist, In Vivo PharmacologyRecently posted Auto Apply
Capable · San Francisco · On-site · Posted 15 Nov 2026 🇺🇸
HealthcareSalary not listed - HU
Bot Detection Researcher (EST)Recently posted Auto Apply
Human · United States · Hybrid · Posted 15 Nov 2026 🇺🇸
SQLKibanaPandasSalary not listed - TI

Research Scientist, Robotics Auto Apply
Tutor Intelligence · Watertown · Hybrid · Posted 7 Nov 2026 🇺🇸
RoboticsLeverPyTorch+2Salary not listed - TI

Research Scientist, Robotics Auto Apply
Tutor Intelligence · San Francisco · Hybrid · Posted 7 Nov 2026 🇺🇸
RoboticsLeverPyTorch+2Salary not listed - TL

Conceptual Reasoning Researcher (Contract) Auto Apply
Trajectory Labs · Berkeley · Remote · Posted 28 Oct 2026 🇺🇸
Generative AI$100 - $150/hr - PL
Member of Technical Staff, Research Scientist - AI Safety & Security Auto Apply
Plato.so · San Francisco · On-site · Posted 20 Oct 2026 🇺🇸
ConcreteSalary not listed - EN

Research Engineer Auto Apply
Engramme · San Francisco · On-site 🇺🇸
NLPA/B TestingDeep Learning+4EngineeringSalary not listed - EN

Research Scientist Auto Apply
Engramme · San Francisco · On-site 🇺🇸
NLPPythonMachine LearningSalary not listed - ET

Principal Scientist/ Associate Director, Toxicology Auto Apply
Edgewise Therapeutics · United States · Hybrid · Posted 4 Nov 2026 🇺🇸
ComplianceDartRegulatory Compliance+1$150k - $200k/yr - ET

Associate Director/Director Cardiovascular Pharmacology Auto Apply
Edgewise Therapeutics · Boulder · Hybrid 🇺🇸
Healthcare$185k - $235k/yr