Quick overview
Job description
About the Role
Build evaluation systems for a B2B SaaS product that turns web content into structured data for AI applications. You will define how output quality is measured, build the systems that evaluate it, and connect results to model and product decisions.
What You'll Do
Design metrics for evaluating output quality across diverse websites, formats, and edge cases.
Build evaluation pipelines and harnesses that measure quality rigorously and at scale.
Generate and curate datasets that support trustworthy evaluation.
Connect evaluation results to model and product decisions, helping teams understand where systems succeed and where they need improvement.
What We're Looking For
Typically 4 to 10 or more years of relevant experience, including at least 2 years working on evaluation systems.
Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, evaluation frameworks, and statistical analysis.
Experience building evaluations for structured extraction from messy, real-world web pages or documents, including field-level accuracy measurement at scale.
Ability to design evaluation metrics and systems, build ground-truth datasets, and apply statistical rigor, including sampling, error bounds, and judge-to-human agreement.
Experience combining research methodology with hands-on engineering and using evaluation findings to inform model or product decisions.
Compensation & Benefits
Base salary is $250,000 to $290,000 USD annually. Visa sponsorship is available.
Location
Hybrid, with the primary location in San Francisco, California. The role is also available in Toronto, Canada, with five days per week on-site.