AI Evaluation Specialist

Location: Atlanta, GA – Remote
Duration: Long-Term Contract

Job Overview

We are seeking an experienced AI Evaluation Specialist to assess enterprise AI tools and capabilities for effectiveness, safety, usability, and readiness for business adoption.

The ideal candidate will have hands-on experience evaluating Generative AI products, Large Language Models (LLMs), and AI-enabled applications. This role involves designing structured evaluation frameworks, testing real-world use cases, analyzing model behavior, and providing actionable recommendations.

The specialist will collaborate with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to evaluate AI solutions, including OpenAI products and life sciences models such as Rosalind.

Key Responsibilities

• Conduct hands-on evaluations of enterprise AI tools using real-world business scenarios, workflows, prompts, datasets, and user personas.
• Assess AI solutions for accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall effectiveness.
• Develop standardized evaluation frameworks, test cases, scorecards, rubrics, benchmarks, and acceptance criteria.
• Evaluate AI systems for safety, privacy, security, data handling, governance, and responsible AI considerations.
• Perform comparative evaluations and proof-of-concept assessments across AI products, models, configurations, and vendors.
• Analyze functionality, integration requirements, performance, and cost.
• Evaluate AI agents and tool-enabled workflows, including RAG, enterprise search, APIs, connectors, file and data analysis, and MCP-enabled capabilities.
• Identify hallucinations, biases, failure modes, and other quality issues.
• Collaborate with technical and business stakeholders to communicate evaluation findings and recommendations.
• Stay current with Generative AI, AI safety, responsible AI, and emerging enterprise AI products.

Required Qualifications

• 5+ years of experience in model evaluation, AI quality assurance, or hands-on evaluation of AI-enabled products or services.
• Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
• Hands-on experience with Generative AI tools and LLMs, including prompt design, systematic testing, output analysis, and hallucination identification.
• Experience evaluating AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost.
• Familiarity with:
  • Large Language Models (LLMs)
  • Generative AI
  • Retrieval-Augmented Generation (RAG)
  • AI Agents
  • APIs and Enterprise Search
  • Tool and Connector Integrations
• Bachelor's degree required.

Preferred Qualifications

• Experience evaluating enterprise AI platforms, including OpenAI products and life sciences models such as Rosalind.
• Experience with automated LLM evaluation frameworks, model-as-a-judge, human-in-the-loop evaluation, red teaming, or adversarial testing.
• Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows.
• Experience developing dashboards and reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results.
• Knowledge of Python, SQL, spreadsheets, notebooks, or analytics platforms.
• Experience analyzing evaluation metrics and identifying quality trends.
• Industry-recognized certification in AI, Machine Learning, Data Science, Cybersecurity, Cloud Technologies, or Responsible AI.

Work Arrangement

• Remote Position
• Atlanta, GA
• Long-Term Contract

Candidates with strong experience in Generative AI evaluation, LLM testing, AI quality assurance, and enterprise AI assessment are encouraged to apply.