Back to Jobs
Recently postedOther

AI Quality Engineer

ClifyXAustin, TX🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationAustin, TX, United States
LLMCI/CDRAG

Job description

Position : AI Quality Engineer

Location: Austin, TX (Hybrid Onsite)

Duration: 12+ Months Contract

Job Description:

AI Quality Engineering (AI QE) Engineer

We are seeking an AI Quality Engineering (AI QE) Engineer to establish and scale quality and evaluation frameworks for our next-generation conversational AI and agentic systems. In this role, you will design automated evaluation pipelines, lead Guardrail teaming and safety assessments, and develop test suites that handle non-deterministic outputs across multi-turn dialogues, tool-use workflows, and backend service integrations.

Key Responsibilities

- Design & Build AI Eval Pipelines: Architect automated evaluation frameworks measuring faithfulness, answer completeness, hallucination rates, and task completion across complex conversational and agentic workflows.

- LLM-as-a-Judge & Heuristic Scoring: Develop, calibrate, and maintain rubric-based LLM-as-a-Judge evaluators.

- Adversarial & Safety Testing (Guardrail Validations): Design systematic stress tests for prompt injection, jailbreaks, PII leakage, guardrail boundaries, and adversarial edge cases.

- Partner with ML/Platform Engineers: Inspect distributed traces, logs, and telemetry to pinpoint whether failures stem from prompts, models, RAG retrieval, or downstream connectors.

- Synthetic Test Data Generation: Build programmatic pipelines to generate diverse persona simulations, multi-turn edge cases, and high-quality golden evaluation datasets.

- CI/CD Integration: Integrate automated regression suites into build/release pipelines to gate prompt changes, model bumps, and backend API updates.

Will your resume get past this employer's screening? Check your resume against this job, free

Similar jobs