Senior Quality Engineer – AI/LLM Test Automation

Position : Senior Quality Engineer – AI/LLM Test Automation

Location: Birmingham, AL
Type: Full-Time

Must Have

• 5+ years of QA / SDET / Test Automation experience.
• 1–2+ years of hands-on experience testing AI/LLM applications.
• Hands-on experience with LLM evaluation / AI evaluation.
• Experience creating or working with evaluation datasets and ground truth.
• Understanding of LLM judges / AI-as-a-judge, evaluators, scoring, and quality thresholds.
• Experience testing nondeterministic AI-generated outputs.
• Hands-on LangSmith experience.
• Hands-on LangGraph or agent workflow experience.
• Strong Playwright experience, preferably with TypeScript.
• Strong Python and/or TypeScript skills.
• API, integration, regression, and end-to-end testing experience.
• Experience integrating automated tests into CI/CD.
• Strong understanding of Git and Agile development.

Ideal Background

Someone who has worked as an SDET / Senior QA Automation Engineer / AI Test Engineer / AI Quality Engineer and can explain, with real examples:

Production/User Behavior ? Evaluation Dataset ? Ground Truth ? Evaluator/LLM Judge ? Experiment ? Score/Threshold ? Regression ? CI/CD

The candidate should be able to explain how they tested AI outputs, not simply how they used AI tools to write traditional automation tests.