AI QA and Test Engineer

Software Guidance & Assistance, Inc., (SGA), is searching for an AI QA and Test Engineer for a contract assignment with one of our premier Infrastructure services clients in GA . This role will be remote.
Seeking an AI QA / Test Engineer to test, evaluate, and validate AI-powered agent experiences across customer and internal business workflows. This role focuses on prompt/agent quality assurance, evaluation framework execution, defect management, and AI observability.
The ideal candidate is curious, analytical, and detail-oriented, with hands-on experience testing LLM-based systems, designing evaluation criteria, and using AI observability tooling. You will partner with product managers, software engineers, and business stakeholders to identify quality gaps and drive agent reliability, accuracy, and user experience improvements.
Responsibilities :

• Prompt & Agent Evaluation
  • Test and evaluate existing prompts, system instructions, and agent workflows for accuracy, consistency, groundedness, and user satisfaction.
  • Conduct structured prompt experiments and A/B test result analysis (execution and analysis, not prompt authorship).
  • Document evaluation findings, failure patterns, and recommended refinements for the engineering team to implement.
• AI Quality Assurance & Testing
  • Design and execute comprehensive test plans covering AI agents, web applications, APIs, and backend services.
  • Develop evaluation frameworks and scoring methodologies for AI output quality.
  • Create test cases covering:
    • Accuracy
    • Hallucination detection
    • Safety
    • Tone and style adherence
    • Instruction following
    • Context retention
    • Response consistency
  • Perform regression testing for prompt and model changes.
  • Track, prioritize, validate, and retest defects using Azure DevOps.
• AI Evaluation & Observability
  • Utilize Arize or similar observability platforms to monitor model and agent performance.
  • Analyze user interactions and evaluation results to identify trends and improvement opportunities.
  • Develop quality metrics and dashboards for:
    • Accuracy
    • Relevance
    • Latency
    • User satisfaction
    • Defect rates
    • Prompt performance
  • Investigate production issues and recommend prompt, workflow, or architectural improvements.
• Documentation & Collaboration
  • Maintain documentation for evaluation methodologies, testing procedures, and defect logs
  • Participate in sprint planning, backlog refinement, and Agile ceremonies.
  • Communicate findings and recommendations to technical and non-technical stakeholders.
  • Share best practices and lessons learned across the team.

By applying for a job with SGA, you agree to allow SGA to process your application for this and future opportunities in accordance with our Privacy Policy. Also, to ensure timely processing, you agree to be contacted by our AI recruiter via email, text, or phone. Message frequency varies and data rates may apply, but you can reply STOP to any SMS message to opt-out of texts and may contact SGA at to opt-out of AI communications. The choice not to engage with AI will not adversely impact your consideration for placement. AI is not used to make any hiring determinations.
SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at .
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company to request an accommodation or assistance regarding our policy.