Start searching
Back to Jobs
RemoteTechnology

AI Test Engineer

Tandym TechVienna, VA🇺🇸United States

Why this role stands out

Step into the forefront of innovation as an AI Test Engineer, where you will build cutting-edge evaluation frameworks for generative AI and LLMs within a forward-thinking enterprise environment. This role is perfect for a detail-oriented quality engineer who thrives on solving complex technical challenges and wants to shape the future of AI reliability. Apply today to advance your career and make a significant impact through flexible, remote-friendly work.

Quick overview

SeniorityMid Senior
Work modeRemote
LocationVienna, VA, United States
PythonMicroservicesAzureGitMachine LearningLLMGenerative AIHugging Face

Job description



An enterprise financial services organization Virginia is seeking an AI Test Engineer to join their team and focus on building automated AI testing frameworks, evaluation pipelines, and quality engineering practices for generative AI, LLM, RAG, and AI-enabled developer experience solutions.


About the Opportunity:




  • Schedule: Monday to Friday




  • Hours: Standard business




  • Setting: Hybrid (at least 3 days a week onsite; Remote work is an option for the right candidate)




Responsibilities:




  • Design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, LLM, RAG, and AI-enabled developer experience solutions




  • Create and execute automated test suites to validate AI model behavior, output quality, accuracy, reliability, consistency, and performance




  • Develop automated evaluation pipelines integrated with CI/CD workflows to support continuous quality validation




  • Perform functional, regression, integration, performance, scalability, and scenario-based testing of AI-enabled applications and services




  • Analyze AI-generated outputs, investigate defects, and document testing findings, metrics, and recommendations for stakeholders




Qualifications:




  • 3+ years of experience in software testing, quality engineering, test automation, AI/ML testing, or a related discipline




  • Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field




  • Strong programming and scripting skills in Python




  • Experience designing and implementing automated test strategies for complex software applications




  • Hands-on experience building automated test frameworks and testing pipelines




  • Experience testing APIs, integrations, microservices, or distributed applications




  • Experience working with CI/CD environments and modern software delivery practices




  • Working knowledge of machine learning, generative AI, large language models, or AI-powered applications




  • Experience with data validation, test-result analysis, defect investigation, and root-cause analysis




  • Knowledge of modern software testing methodologies, quality engineering practices, and software development lifecycles




  • Knowledge of statistical analysis and evaluation techniques used to validate system performance




  • Experience with Git or comparable source control systems and collaborative development practices




Desired Skills:




  • Experience testing generative AI applications, LLM-based solutions, AI agents, or Retrieval-Augmented Generation systems




  • Experience with AI evaluation frameworks such as DeepEval, Ragas, LangSmith, or comparable tools




  • Experience with LLM observability, monitoring, tracing, and evaluation practices




  • Experience with Azure AI, GitHub Copilot, OpenAI, Hugging Face, LangChain, or comparable AI platforms and frameworks




  • Familiarity with prompt engineering, benchmark dataset creation, prompt libraries, and AI output validation techniques




  • Understanding of Responsible AI principles, AI governance, model risk management, and enterprise AI controls




  • Experience with performance, scalability, reliability, and resilience testing




  • Experience supporting Developer Experience, developer productivity, software engineering enablement, or internal developer platforms




  • Experience working within enterprise, financial services, regulated, or highly controlled technology environments




  • Experience developing quality dashboards, automated reporting, or AI evaluation analytics




  • Familiarity with model comparison, A/B testing, experimentation, and statistical evaluation methodologies




  • Strong curiosity about emerging AI technologies and a passion for building scalable quality engineering capabilities




Similar jobs

Apply