Senior Product Quality Analyst – AI Foundation

Senior Product Quality Analyst – AI Foundation
Location: Remote
Employment Type: W2 Contract

Job Description

We are seeking a Senior Product Quality Analyst to support AI and ML product development through high-quality human judgments, data annotation, and evaluation workflows. This role will help establish ground truth for model evaluation, support model fine-tuning, and improve the alignment of LLM-based evaluators.

The ideal candidate will have experience managing data annotation projects, evaluating AI/ML-powered products, developing annotation guidelines, and collaborating with cross-functional teams to improve product quality.

Key Responsibilities

• Lead end-to-end execution of human annotation projects, from intake and scoping through delivery.
• Maintain a roadmap of annotation requirements, prioritizing requests and coordinating internal and external annotation resources.
• Manage annotation vendors and coordinate contracts and reviews with Legal, Security, Privacy, Finance, and Accounting.
• Develop and maintain reusable annotation frameworks and guidelines across teams.
• Consult with product and modeling teams on annotation task design, evaluation rubrics, and quality standards.
• Conduct feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
• Evaluate annotation quality, identify sources of disagreement, and improve guidelines and annotation outputs.
• Translate product and modeling requirements into clear, actionable annotation workflows.
• Coordinate project delivery across internal stakeholders and external vendors.

Required Qualifications

• Experience in product quality, evaluation, data operations, or a related field.
• Hands-on experience designing, coordinating, or reviewing data annotation workflows for product development.
• Familiarity with machine learning-driven products and human evaluation of AI/LLM-powered features.
• Ability to translate product and modeling requirements into clear annotation guidelines and evaluation rubrics.
• Experience managing vendors and coordinating deliverables across internal teams.
• Strong communication and stakeholder management skills.
• Ability to manage ambiguous requirements and navigate complex enterprise processes.
• Strong project tracking and organizational skills.

Required Skills

Hard Skills

• End-to-end data annotation workflow management
• AI/ML product development and evaluation rubrics
• Vendor management and procurement coordination

Soft Skills

• Cross-functional communication
• Stakeholder management
• Enterprise process navigation
• Project coordination and delivery management

Preferred Qualifications

• Experience working with AI collaboration tools such as Claude or Codex.
• Basic coding abilities.
• Experience working within large enterprise organizations.
• Experience supporting AI evaluation, model fine-tuning, or LLM evaluator alignment.

Role Impact

This role bridges day-to-day data annotation and evaluation activities with Client’s broader AI strategy. The selected candidate will provide human judgments that help ground recommendation models in cultural nuance, personal taste, and real-world listening habits.

The role will have influence across AI Foundation by improving data quality, evaluation standards, and collaboration between product and modeling teams.

Key Challenges

• Navigating privacy, legal, and security approval processes when onboarding third-party annotation vendors.
• Coordinating delivery across multiple teams and stakeholders.
• Managing ambiguous requirements and ensuring consistent annotation quality.
• Balancing project timelines with enterprise review and approval processes.