AI Testing Engineer - Remote

Title: AI Testing Engineer

Work Location: 100% Remote

Start Date: ASAP, subject to candidate availability

Technology Environment: AWS technology stack, working closely with the AWS team

Interview Process: 2 rounds

Engagement: Enterprise, revenue-critical real-time streaming/AI program

Role Overview: AI Testing Engineers will validate and assure the quality of AI/ML systems powering real-time contextual advertising on live video streams. The role involves testing multimodal AI outputs across video, text, audio and transcripts, ensuring the accuracy of detected contextual moments, and automating validation workflows.

Key Responsibilities

• Validate AI-generated outputs against labelled datasets and expected outcomes
• Perform content validation for contextual/moment detection across video streams
• Design and implement Python-based automated testing frameworks
• Build validation harnesses for:
• JSON parsing and comparison
• Structured output validation using defined schemas
• Data validation
• Conduct multimodal validation across video, audio and transcript outputs
• Execute system integration and end-to-end testing
• Track accuracy metrics, defect logging and root-cause analysis
• Automate regression and validation pipelines across datasets
• Collaborate with Applied Scientists to validate LLM and GenAI outputs
• Work within an AWS environment using services such as:
• Amazon S3
• AWS Lambda
• Amazon CloudWatch

Required Skills & Experience

• Strong Python scripting experience for automation and testing
• Experience with AI/ML model validation and QA workflows
• Hands-on experience with JSON parsing, schema comparison and data validation pipelines
• Experience testing multimodal AI systems, including:
• Vision
• NLP
• Audio
• LLMs
• Generative AI pipelines
• Good understanding of structured output validation and AI accuracy evaluation techniques
• Familiarity with AWS services, particularly S3, Lambda, logging and monitoring
• Experience with media/video content analysis workflows is preferred
• Ability to work across both manual and automated testing approaches