Back to Jobs
Recently postedFull-timeEngineering

Model Behavior Engineer

OvertoneNew York City🇺🇸United States

Quick overview

Salary$160k - $190k/yr
Employment typeFull-time
Work modeOn-site
LocationNew York City, United States
LLMPrompt Engineering

Job description

Your Role

Your job is to work with our relationship scientist and eval engineer to build and execute an eval strategy that keeps our LLM content high quality at scale.

  • Our relationship scientist defines what good looks like across our content surfaces and drafts golden examples. They also label for LLM judges that need subject matter expertise or calls on tone and voice.

  • Our eval engineer builds the infrastructure to power our content monitoring and iteration: human labeling tools, automated judge calibration tools, Braintrust experiments, and production monitoring dashboards.

  • You are the hands-on operator of this system: the eval engineer builds the tools, the relationship scientist defines goldens, and you use both every day to find and fix problems in our content.

The team is early, so you'll have real influence over how this workflow gets built.

What You’ll Do

  • Prompt iteration and engineering. Fix straightforward errors yourself: when our judges flag a problem in production content, diagnose the failure, change the prompt, and confirm the fix with evals before it ships. When a fix depends on subject matter expertise, partner with our relationship scientist. You’ll draft prompts for our content surfaces and our judge rubrics.

  • Eval + content engineering strategy. Design our infrastructure for evals and content iteration, including labeling, judge calibration and production quality monitoring and keep evolving it as the product grows.

  • Define what we measure. Work with our relationship scientist to decide which content features to measure, from proactive baseline quality checks to reactive checks driven by errors we find. Set up automated error analysis so new failure patterns surface on their own, not only when someone spots them.

  • Labeling strategy and tools. Own how we label and calibrate our judges, including the tools and workflows that scale our relationship scientist's judgment as efficiently as possible. Make sure they label only where their expertise is needed. Everywhere else, label production outputs and match assessments yourself or through outsourced labelers, so we can check that our judges agree with human judgment.

  • Synthetic testing. Design synthetic tests that tell us whether a new prompt is working before it reaches human testing, and review the results.

  • Production review and prioritization. Read real production outputs across our core surfaces and spot quality issues. Prioritize what to fix first across everything our judges, automated error analysis and manual reviews surface.

Who You Are

We're open to a very product-minded engineer, or a highly technical product person:

  • 3+ years in product management or engineering, including shipping LLM-powered features to real users

  • Hands-on experience with evals: writing rubrics, building or running LLM-as-judge setups, and aligning judges against human labels

  • Hands-on prompt engineering experience: you can diagnose why an output went wrong, rewrite the prompt, and show with evals that the fix worked

  • Comfortable writing code when the work calls for it, like scripts for error analysis or changes to our eval pipeline

  • A detail-oriented, editorial eye for tone, structure and voice in AI-written content

  • Eagerness to dive in and read lots of AI outputs by hand

Great to have:

  • Experience with Braintrust or similar eval tooling

  • Experience running human labeling programs

  • Experience with conversational or voice AI products

  • Interest in relationships, psychology or matchmaking

Compensation & Benefits

Salary of $160k to $190k. The offer within that range depends on experience and skills. This position will receive meaningful stock options with considerable upside potential.

We provide standard benefits such as:

  • Medical, dental and vision insurance

  • 401(k)

  • Commuter benefits, daily lunch and other perks

  • Paid time off, holidays and sick leave

Our Core Qualities

We look for these qualities in everyone we hire at Overtone.

Curiosity: You seek to understand before you decide. You ask thoughtful questions, test assumptions, and stay open to changing your mind as new information emerges.

Craft: You care deeply about quality and coherence. You simplify complexity, exercise good judgment, and take pride in work that feels intentional, thoughtful, and well-made.

Collaboration: You engage with others honestly and generatively. You challenge ideas with respect, listen deeply, and help synthesize perspectives into clearer, better outcomes.

Care: You take responsibility for people and outcomes. You act with integrity, follow through on commitments, and protect trust—with users, teammates, and the work itself.

About Overtone

Overtone is a modern matchmaking company, from the creator of Hinge. We’re building a voice- and audio-forward service, enabled by AI, grounded in relationship science. There’s no profiles or swiping. We get to know people in their own voice, then make thoughtful introductions, one at a time, to people we think are worth meeting. The service is in beta-testing and not yet open to the public. We’re building a small team of people passionate about relationships and human connection, to work together in person in our NYC office (5 days a week in Flatiron).

Equal Opportunity

Overtone is an equal opportunity employer. We consider all qualified applicants without regard to race, color, religion, creed, sex, pregnancy, sexual orientation, gender identity or expression, national origin, ancestry, age, disability, genetic information, marital or partnership status, caregiver status, veteran or military status, or any other status protected by law. We will consider qualified applicants with arrest and conviction records in a manner consistent with the New York City Fair Chance Act and other applicable law. If you need an accommodation at any point in the application or interview process, email careers@overto.ne and we'll work with you.

Haystack's AI agent can fill in and submit this application for you. How AI auto apply works

Will your resume get past this employer's screening? Check your resume against this job, free

Similar jobs