Application Engineer

Summary

We are seeking an AI QA / Test Engineer to test, evaluate, and validate AI-powered agent experiences across customer and internal business workflows. This role focuses on prompt/agent quality assurance, evaluation framework execution, defect management, and AI observability.

Responsibilities

• Test and evaluate existing prompts, system instructions, and agent workflows for accuracy, consistency, groundedness, and user satisfaction.
• Conduct structured prompt experiments and A/B test result analysis (execution and analysis, not prompt authorship).
• Document evaluation findings, failure patterns, and recommended refinements for the engineering team to implement.

AI Quality Assurance & Testing

• Design and execute comprehensive test plans covering AI agents, web applications, APIs, and backend services.
• Develop evaluation frameworks and scoring methodologies for AI output quality.
• Create test cases covering:

• Accuracy
• Hallucination detection
• Safety
• Tone and style adherence
• Instruction following
• Context retention
• Response consistency

• Perform regression testing for prompt and model changes.
• Track, prioritise, validate, and retest defects using Azure DevOps.

AI Evaluation & Observability

• Utilise Arize or similar observability platforms to monitor model and agent performance.
• Analyse user interactions and evaluation results to identify trends and improvement opportunities.
• Develop quality metrics and dashboards for:

• Accuracy
• Relevance
• Latency
• User satisfaction
• Defect rates
• Prompt performance

• Investigate production issues and recommend prompt, workflow, or architectural improvements.

Documentation & Collaboration

• Maintain documentation for evaluation methodologies, testing procedures, and defect logs
• Participate in sprint planning, backlog refinement, and Agile ceremonies.
• Communicate findings and recommendations to technical and non-technical stakeholders.
• Share best practices and lessons learned across the team.