Quick overview
Job description
Position : AI Quality Engineer
Location: Austin, TX (Hybrid Onsite)
Duration: 12+ Months Contract
Job Description:
AI Quality Engineering (AI QE) Engineer
We are seeking an AI Quality Engineering (AI QE) Engineer to establish and scale quality and evaluation frameworks for our next-generation conversational AI and agentic systems. In this role, you will design automated evaluation pipelines, lead Guardrail teaming and safety assessments, and develop test suites that handle non-deterministic outputs across multi-turn dialogues, tool-use workflows, and backend service integrations.
Key Responsibilities
- Design & Build AI Eval Pipelines: Architect automated evaluation frameworks measuring faithfulness, answer completeness, hallucination rates, and task completion across complex conversational and agentic workflows.
- LLM-as-a-Judge & Heuristic Scoring: Develop, calibrate, and maintain rubric-based LLM-as-a-Judge evaluators.
- Adversarial & Safety Testing (Guardrail Validations): Design systematic stress tests for prompt injection, jailbreaks, PII leakage, guardrail boundaries, and adversarial edge cases.
- Partner with ML/Platform Engineers: Inspect distributed traces, logs, and telemetry to pinpoint whether failures stem from prompts, models, RAG retrieval, or downstream connectors.
- Synthetic Test Data Generation: Build programmatic pipelines to generate diverse persona simulations, multi-turn edge cases, and high-quality golden evaluation datasets.
- CI/CD Integration: Integrate automated regression suites into build/release pipelines to gate prompt changes, model bumps, and backend API updates.
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- AT

QA EngineerRecently posted
Acadia Technologies, Inc. · Edison, NJ · Hybrid · Posted 18 Nov 2026 🇺🇸
DockerSeleniumJava+2TechnologySalary not listed - PI
Mainframe TesterRecently posted
Purview Infotech · Dallas, TX · On-site · Posted 18 Nov 2026 🇺🇸
JavaTechnologySalary not listed - NG
NetSuite Tester / QA ConsultantRecently posted
Nexylum Global LLC · United States · Hybrid · Posted 18 Nov 2026 🇺🇸
ERPNetSuiteProcurement+4TechnologySalary not listed - IT
Quality Assurance Analyst Mid 102529 with Security ClearanceRecently posted
Information Technology Engineering Corporation · Washington, DC, DC · Hybrid · Posted 18 Nov 2026 🇺🇸
ScrumKanbanAgile+1TechnologySalary not listed - LM

Test Engineer - Andover, MA with Security ClearanceRecently posted
Lockheed Martin · Andover, MA · Hybrid · Posted 18 Nov 2026 🇺🇸
Microsoft OfficeAssemblyTechnology$73.4k - $129.3k/yr - WS
Software Development Engineer in Test (SDET) with Security ClearanceRecently posted
Wallach Search Group · Reston, VA · Hybrid · Posted 14 Nov 2026 🇺🇸
RustGitJava+3TechnologySalary not listed - PI

Senior Java SDET EngineerRecently posted
Paradigm Infotech · Baltimore, MD · On-site · Posted 18 Nov 2026 🇺🇸
DockerOracleSQL+22TechnologySalary not listed - SJ
Senior QA Engineer-W2 ONLYRecently posted
Select Jarvis.com · Charlotte, NC · Hybrid · Posted 18 Nov 2026 🇺🇸
OracleSQLShell+8TechnologySalary not listed - AI
Quality Assurance AnalystRecently posted
Aasritha infotech Inc · Washington, DC · Hybrid · Posted 18 Nov 2026 🇺🇸
ETLScrumSelenium+7TechnologySalary not listed - AT
Calypso QA EngineerRecently posted
Avinya Technologies LLC · United States · Hybrid · Posted 18 Nov 2026 🇺🇸
SAFeAgileTechnology$65/hr - TI
Senior Healthcare QA AnalystRecently posted
Tek Inspirations LLC · United States · Hybrid · Posted 18 Nov 2026 🇺🇸
SQLGitAWS+8TechnologySalary not listed - RI
QA AnalystRecently posted
Raas Infotek LLC · Austin, TX · Hybrid · Posted 18 Nov 2026 🇺🇸
SQLScrumSelenium+4TechnologySalary not listed
