Research Engineer, Evals

About the Role

Build evaluation systems that measure the quality of web data extraction across diverse sites, formats, and edge cases. You will design the metrics, datasets, and pipelines that turn quality findings into model and product decisions.

What You'll Do

• Define rigorous metrics for evaluating structured extraction quality across varied web content.

• Build scalable evaluation pipelines and harnesses to measure system performance.

• Generate and curate ground-truth datasets for trustworthy evaluation.

• Connect evaluation results to model and product improvements, closing the feedback loop.

What We're Looking For

• Typically 4 to 10+ years of relevant experience, including at least 2 years working on evaluations.

• Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, and evaluation frameworks.

• Experience building evaluation systems for LLM extraction of structured fields from messy, real-world web pages or documents, at large scale.

• Ability to design evaluation methodology and systems from the ground up, rather than only running existing benchmarks.

• Statistical rigor in metric design and interpretation, including sampling, estimator bias and variance, error bounds, significance, and judge-to-human agreement.

• Experience creating gold-label datasets and using evaluation results to guide model or product decisions.

• A blend of research depth and engineering ability, with comfort tackling ambiguous problems and iterating quickly.

Compensation & Benefits

Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available.

Location

Hybrid, based in San Francisco, California. Toronto, Canada is also an available work location. The role requires working on-site five days per week in either city.