Haystack
← Back to Jobs
Technology

AI Automation QA Engineer - 100% Onsite in Baltimore,MD

Unisoft Technology IncBaltimore, MD🇺🇸United StatesPosted 14 Aug 2026

Quick Overview

Work Type
On Site
Level
Mid Senior

Job Description

We are looking for a AI Automation QA Engineer who are local to DMV area. This position requried 100% Onsite and In-Person Interview in Baltimore,MD.

Responsibilities:

       AI/LLM Testing: Validating AI outputs, including GenAI hallucination detection, prompt-response accuracy, and evaluating models using metrics like BLEU or perplexity.

       Test Automation: Developing and maintaining automated test suites for AI systems, including integrating tools like PyTest, Selenium, or specialized AI frameworks such as RAGAS and promptfoo into CI/CD pipelines.

       Collaboration: Working closely with data scientists, developers, and product managers to define, test, and implement AI features within Agile environments.

       Performance Monitoring: Monitoring AI system metrics such as latency, cost, and model drift

       Develop and document test cases, testing plans and procedures in an agile environment.

       Develop, execute and coordinate IT software tests and evaluate results to ensure compliance with applicable regulations.

       Design and prepare all test data needed. Reviews test results and evaluates for conformance to design.

       Develop and document test cases, testing plans and procedures in an agile environment.

       Develop and maintain automated regression and integration test plans for validation.

 

         General Experience: The proposed candidate must have at least eight (8) years of information systems quality assurance experience. 2–5+ years of experience in software quality assurance, with at least 1–2 years specifically in AI/ML or automation testing.

       Technical Skills: Proficiency in Python (essential for GenAI libraries like LangChain and LlamaIndex) and API testing tools (Postman, REST).

       Domain Knowledge: Solid understanding of machine learning concepts, data pipelines, and NLP (natural language processing).

       Drive software quality assurance lifecycle within an Agile process

       Establish and coordinate test strategies with development/product teams

       Design and implement test plans and test cases

       Develop and execute automated UI and functional tests

       Enhance and maintain automated CI flows

Skills

Machine Learning
NLP
Selenium
Agile
LLM
Postman
Python
REST
pytest

Similar jobs

Browse related jobs

Apply now