Validation Engineer - AI & Data

Validation Engineer - AI & Data

Summary: Validate the accuracy, reliability, completeness, and consistency of business context and AI generated responses against approved business requirements and ground truth answers.

Responsibilities:

• Design and implement the testing strategy for the Domain Context Model, Business Context Repository, ontology mappings, and AI response evaluation.
• Develop representative ground truth question and answer test sets with defined expected results and tolerances, validated by data owners.
• Build and maintain automated evaluation tests for AI responses generated through the client’s Context Injection toolkit.
• Evaluate AI responses with and without domain context to establish baseline accuracy and measure improvements.
• Validate numeric answers, SQL and query results, business logic, aggregations, join relationships, and narrative responses.
• Identify and classify failures, including hallucinations, stale information, incorrect schemas, faulty joins, missing business context, and persona mismatches.
• Develop repeatable evaluation scripts, scoring rubrics, regression tests, and accuracy reporting dashboards or scorecards.
• Verify context repository data completeness, classification consistency, access control behavior, and changes introduced through automated context updates.
• Document test evidence, defects, remediation results, AI readiness assessments, and validation outcomes.
• Support release readiness, weekly demonstrations, and handover of reusable evaluation frameworks and documentation.

Required Technical Expertise: Data quality engineering, SQL, Python, automated testing, PyTest or equivalent frameworks, ETL and data validation, AI and LLM evaluation, ground truth testing, regression testing, and test automation.