AI ML Test Engineer

Job Title: Senior AI Test Engineer / AI Quality Engineer
Location: PA, USA (Hybrid / Remote)
Duration: Long-Term Contract (6–12+ months)
Experience: 8+ years in QA Automation, including hands-on AI/GenAI testing experience

Job Summary

We are seeking an experienced AI Test Engineer with strong expertise in AI/ML application testing, Generative AI validation, LLM evaluation, and test automation.

The ideal candidate will have hands-on experience testing AI-powered applications, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) pipelines, conversational AI, and agentic AI workflows.

The role requires strong Python-based automation skills, experience with Selenium or Playwright, API testing, and the ability to build scalable AI quality validation frameworks.

Experience in the Insurance or Financial Services domain is highly preferred.

Required Skills

• Strong hands-on experience with AI/GenAI application testing and LLM evaluation.

• Experience with hallucination detection, response accuracy, relevance, faithfulness, and prompt validation.

• Hands-on experience testing RAG pipelines, including retrieval accuracy and groundedness.

• Proficiency in Python for AI test automation and validation.

• Experience with DeepEval, RAGAS, Promptfoo, or similar LLM evaluation frameworks.

• Strong experience with Selenium or Playwright for UI automation.

• Experience with REST API testing using Postman, REST Assured, or Python-based frameworks.

• Experience validating AI applications integrated with enterprise systems.

• Knowledge of CI/CD pipelines using Jenkins, GitHub Actions, or Azure DevOps.

• Experience with Agile/Scrum methodologies and defect management.