Haystack
← Back to Jobs
Technology
AI

Automation QA Engineer with AI/LLM_Alpharetta, GA- Onsite- Only Locals

American IT SystemsAlpharetta, GA🇺🇸United StatesPosted Sep 5, 2026

Why This Role Stands Out

This role offers a unique opportunity to pioneer AI and LLM testing, building cutting-edge evaluation frameworks and automating complex conversational agents, perfect for experienced QA engineers eager to shape the future of intelligent systems. If you excel in Python/JavaScript, possess strong automation skills, and are excited by the prospect of working with advanced AI technologies, you'll thrive in this impactful position. Apply now to join a forward-thinking team and grow your expertise in a critical emerging field.

Quick Overview

Seniority
Mid Senior
Work mode
Hybrid
Location
Alpharetta, GA, United States
Posted
3 weeks ago
SeleniumJavaScriptJiraLLMPythonpytest

Job Description

Automation QA Engineer with AI/LLM

Alpharetta, GA- Only Locals

6+ Months

Experience: 10+ years


Tech Stack: Python/JavaScript, Test Automation, Pytest/Selenium, LLM Evaluation, Jira/Test Management

Job Description:

  • Design and execute test cases for multi-turn agent conversations.
  • Build evaluation frameworks to assess LLM quality.
  • Automate API and UI testing.
  • Create regression suites for agentic outputs.
  • Drive test coverage and quality metrics.
  • Develop automated testing frameworks using Python/JavaScript, Pytest, Selenium, or similar technologies.
  • Validate AI/LLM responses and agent behavior against defined quality criteria.
  • Track defects and test results using Jira or other test-management tools.

Similar jobs

Browse related jobs

Apply now