Data Scientist - AI Evaluation Analytics (REMOTE)

Role Overview

Provide data science and analytical expertise to improve the measurement, reliability, and interpretation of AI system performance through data-driven evaluation approaches.

Key Responsibilities

• Support development and refinement of evaluation metrics, scoring approaches, and benchmark methodologies.
• Validate datasets, data quality criteria, and evaluation inputs to improve reliability of results.
• Analyze evaluation data, performance trends, and measurement signals to generate actionable insights.
• Apply statistical analysis and experimentation techniques to assess AI system effectiveness.
• Develop analytical approaches to identify trends, patterns, and opportunities for improvement.

Required Skills & Experience

• Strong background in data science, analytics, statistics, or related fields.
• Experience designing metrics, analytical frameworks, or measurement approaches.
• Proficiency in Python, SQL, and data analysis techniques.
• Experience working with large datasets and extracting meaningful insights.
• Strong communication skills with the ability to explain analytical results clearly.

Preferred Experience

• Experience evaluating AI/ML systems and agentic AI applications.
• Experience with experimentation, benchmarking, or performance analysis.
• Experience working with software engineering, productivity, or quality metrics.