Recently postedTechnology

AI/ML Engineer

Avenue Code, LLCNew York, NY🇺🇸United States

Quick overview

SeniorityMid Senior

Work modeOn Site

LocationNew York, NY, United States

Stakeholder ManagementLLM

Job description

AI/ML Engineer

Location: New York or Boston (in-office), or remote with working hours in Eastern Time

AI Foundation develops the core ML and AI systems that power personalized experiences on Spotify. These systems rely on human judgment to ground their intelligence in the nuances of culture, personal taste, and real-world listening habits.

We’re looking for a Senior Product Quality Analyst to deliver high-quality human judgments that provide ground truth for evaluation, support model fine-tuning, and guide the alignment of LLM-based evaluators. You’ll lead annotation projects and drive improvements in annotation quality.

What you’ll do

Lead end-to-end execution of human annotation projects, from intake and scoping to delivery.
Maintain a roadmap of annotation needs, prioritizing requests and coordinating the appropriate internal or external annotation resources.
Manage vendors and coordinate contracts and related reviews with legal, security, privacy, finance, and accounting.
Develop and maintain reusable annotation frameworks and guidelines that can be adopted across teams, and consult on annotation task and rubric design.
Run feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
Evaluate annotation quality, identify sources of disagreement, and work with teams and vendors to improve guidelines and outputs.

Who you are

Background in product quality, evaluation, data operations, or related roles.
Hands-on experience designing, coordinating, or reviewing data annotation workflows for product development.
Familiarity with ML-driven products and human evaluation of AI/LLM-powered features.
Ability to translate product and modeling needs into clear annotation guidelines and evaluation rubrics.
Skilled at managing vendors and coordinating delivery across internal teams.
Excellent communication and stakeholder management skills, including setting expectations and resolving ambiguous requirements.