Quick overview
Job description
About the Role
Develop benchmarks and evaluation methods for assessing advanced language models. This research and engineering role combines hands-on AI research with building practical evaluation systems, helping shape how models are tested before deployment.
What You'll Do
Evaluate newly released AI models across a range of capabilities and use cases.
Build benchmarks from the ground up, including dataset creation and coordination with labelers.
Develop research findings and contribute to publications on evaluation methods.
Improve automated methods for evaluating generated text.
Work with engineering teams to implement and scale evaluation approaches in production systems.
Collaborate with AI labs and enterprise customers to understand evaluation needs and define research initiatives.
What We're Looking For
At least 3 years of full-time AI or machine learning research experience, especially in benchmarking, model evaluation, NLP, or language models.
A track record of published research, ideally in NLP or benchmarking, alongside an interest in translating research into applied systems.
Strong Python skills and experience building clean, maintainable software, including HTTP services with Django, Flask, or similar frameworks.
Hands-on experience with PyTorch or TensorFlow, language modeling, and relevant AI/ML infrastructure and techniques.
Experience designing benchmarks and evaluation methodologies.
Strong computer science fundamentals and experience collaborating through Git workflows, code reviews, and cross-functional feedback.
A degree in computer science, machine learning, or a related field; a master's or PhD is preferred. Candidates with a bachelor's degree and 1 to 3 years of applied experience may also be considered.
Experience in an AI startup, research lab, or early-stage company is helpful.
Compensation & Benefits
Base salary: $150,000 to $200,000 annually. Visa sponsorship is available.
Location
On-site in San Francisco, California, United States.