Quick overview
Job description
Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pretraining, domain-specific RL, ultra-long context, and inference-time compute to achieve this goal.
About the role
The Evals team measures the performance and capabilities of internal and external models, helping teams across Magic make informed decisions. The team supports pretraining, RL, data, inference, and product, and sits on the critical path of many of the company's most important decisions.
As a Member of Technical Staff on Evals, you will develop new benchmarks and evaluation methodologies, run data ablations, assess dataset quality, and investigate whether our measurements accurately reflect model performance.
Sweating the details matters on this team. Many benchmarks, papers, and open-source evaluation frameworks contain subtle bugs or flawed assumptions that lead to misleading conclusions. We care deeply about correctness, reproducibility, and measurement quality.
What you'll work on
Design and validate new evaluations and benchmarks across model training and deployment
Investigate model behavior and identify gaps in existing evaluations
Measure dataset quality and identify opportunities to improve training data
Improve evaluation correctness, reproducibility, and reliability
Audit and improve upon public benchmarks, research papers, and open-source implementations
Define metrics that accurately reflect model quality
Build tools that make evaluation results accessible and useful across Magic
What we're looking for
Experience developing model evaluations, benchmarks, or ML experiments
Strong programming skills and experience working with machine learning systems
Track record of owning technical projects end-to-end
Strong understanding of post-training techniques and their impact on model performance
Skepticism toward results that cannot be reproduced, validated, or explained
Experience analyzing model performance and interpreting experimental results
Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure
Commitment to raising the standard of rigor in AI evaluations
Our culture
Integrity. Words and actions should be aligned
Hands-on. At Magic, everyone is building
Teamwork. We move as one team, not N individuals
Focus. Safely deploy AGI. Everything else is noise
Quality. Magic should feel like magic
Magic strives to be the place where high-potential individuals can do their best work. We value quick learning and grit just as much as skill and experience.
Compensation, benefits, and perks (US)
Annual salary range between $225K - $550K depending on experience
Equity is a significant part of total compensation, in addition to salary
401(k) plan with 6% salary matching
Generous health, dental, and vision insurance for you and your dependents
Unlimited paid time off
Visa sponsorship and relocation support for candidates moving to San Francisco
A small, fast-moving, highly collaborative team working on frontier AI systems
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- OK

Senior AI Analyst, People Digital SolutionsRecently posted Auto Apply
Okta · Bellevue · Hybrid · Posted 16 Nov 2026 🇺🇸
OnboardingSQLMicrosoft Excel+9$121k/yr - UD

Principal Consultant, AI EngineeringRecently posted Auto Apply
UDig · Nashville · Hybrid · Posted 16 Nov 2026 🇺🇸
ArticulateIntellectual Property$167.5k - $195k/yr - BL
Senior AI Creative Strategist (Legal Marketing)Recently posted Auto Apply
Block LLP · Burbank, California · On-site · Posted 16 Nov 2026 🇺🇸
Generative AILegalSalary not listed - TT
Founding AI Agent EngineerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 16 Nov 2026 🇺🇸
RecruitingClinical TrialsOutreach+2$150k - $220k/yr - TT
AI SchedulerRecently posted Auto Apply
True Talent Solutions Partners · San Francisco · On-site · Posted 16 Nov 2026 🇺🇸
BIMConstruction ManagementPrimavera P6+2$140k - $200k/yr - TT
Founding AI EngineerRecently posted Auto Apply
True Talent Solutions Partners · New York · On-site · Posted 15 Nov 2026 🇺🇸
Next.jsPlaywrightNode.js+1Technology$216k - $250k/yr - ER

Head of Applied AI Auto Apply
Eragon · San Francisco · On-site · Posted 30 Oct 2026 🇺🇸
$350k - $500k/yr - OR
Research Engineer, AIRecently posted Auto Apply
Orbis · Remote - US · Remote · Posted 13 Nov 2026 🇺🇸
LLMPythonMachine LearningEngineeringSalary not listed - OR
Research Engineer, AI (Entry-Level) Auto Apply
Orbis · Remote - US · Remote · Posted 3 Nov 2026 🇺🇸
DockerLLMPython+2Salary not listed - WI
AI Value Delivery Lead (Partner Motion) Auto Apply
Within.ai · San Francisco (HQ) · On-site 🇺🇸
Six SigmaGenerative AISlackOperations & Project ManagementSalary not listed - SI
AI Transformation ManagerRecently posted Auto Apply
Sia · Bellevue, WA · On-site · Posted 13 Nov 2026 🇺🇸
ScrumLean Six SigmaPMP+5Operations & Project Management$142.8k - $147.9k/yr - SI
AI Transformation ManagerRecently posted Auto Apply
Sia · Philadelphia, PA · On-site · Posted 13 Nov 2026 🇺🇸
ScrumLean Six SigmaPMP+5Operations & Project Management$129.5k - $134.6k/yr
