Haystack
← Back to Jobs
Technology
VS

Senior AI Test Engineer

Vaisesika Systems Consulting LLCUnited States🇺🇸United StatesPosted 18 Aug 2026

Why This Role Stands Out

This hybrid Senior AI Test Engineer role at Vaisesika Systems Consulting LLC offers exciting opportunities for professional development and impactful work within a reputable company. You'll thrive here if you are passionate about AI, eager to expand your technical expertise, and enjoy collaborating in a supportive team environment. Apply today to advance your career in cutting-edge technology!

Quick Overview

Work Type
Hybrid
Level
Mid Senior

Job Description

*]:pointer-events-auto R6Vx5W_threadScrollVars scroll-mb-[calc(var(--scroll-root-safe-area-inset-bottom,0px)+var(--thread-response-height))] scroll-mt-[calc(var(--header-height)+min(200px,max(70px,20svh)))]" dir="auto" data-turn-id="request-WEB:f78be4ea-f785-4b61-af55-5aa7c4960d89-2" data-turn-id-container="request-WEB:f78be4ea-f785-4b61-af55-5aa7c4960d89-2" data-testid="conversation-turn-6" data-turn="assistant">
 
 
 
 
 

AI/GenAI Test Engineer

Job Summary

We are looking for an AI/GenAI Test Engineer with experience in testing Generative AI, LLM, and AI-powered applications. The candidate should have strong Python and test automation skills.

Required Skills

  • 6+ years of experience in QA / Test Automation.
  • Hands-on experience testing AI/ML or Generative AI applications.
  • Strong Python scripting/programming skills.
  • Experience with LLM / GenAI testing and evaluation.
  • Experience with RAG, prompt testing, or chatbot testing.
  • Strong experience in API testing using REST APIs/Postman.
  • Experience with PyTest, Selenium, or Playwright.
  • Knowledge of SQL and data validation.
  • Experience with Git and CI/CD.
  • Good understanding of functional, regression, integration, and automation testing.

Preferred

  • Experience with OpenAI, Azure OpenAI, AWS Bedrock, or similar LLM platforms.
  • Knowledge of LLM evaluation, hallucination testing, accuracy, relevance, and response validation.
  • Experience with tools such as Ragas, DeepEval, Promptfoo, or LangSmith.

Key Responsibilities

  • Develop and execute test cases for GenAI/LLM applications.
  • Validate LLM responses for accuracy, relevance, consistency, and hallucinations.
  • Automate AI testing using Python and PyTest.
  • Perform API, functional, regression, and integration testing.
  • Test RAG pipelines, prompts, chatbots, and AI workflows.
  • Identify, document, and track defects.
  • Collaborate with Developers, Data Scientists, and AI/ML Engineers.
 
 
 
 
 
 

Skills

SAFe

Similar jobs

Browse related jobs

Apply now