AI Automation Engineer

Job Title: AI Automation Engineer

Location: Bellevue, WA Onsite

Job Type: Full-Time

Compensation: $150,000/Year

Experience: 15+ Years of Overall IT Experience, including 3+ Years of AI/ML Experience

Job Summary

We are seeking a highly experienced AI Automation Engineer to design, develop, and implement automated testing, evaluation, observability, and quality frameworks for AI/ML platforms and applications.

The ideal candidate will have strong expertise in Python, SQL, ETL pipelines, AI/ML, AIOps, MLOps, LLMOps, AgentOps, system design, and MCP client-server architecture. The candidate will be responsible for building scalable AI testing and evaluation frameworks, ensuring the quality, reliability, performance, and observability of AI-powered systems.

Key Responsibilities

Design and build end-to-end AI testing frameworks covering unit, functional, integration, and regression testing.

Develop automated test case generation frameworks and validate AI/ML model outputs against defined quality benchmarks.

Build and execute load and performance testing frameworks to simulate high-concurrency scenarios and validate scalability, latency SLAs, and platform stability.

Develop LLM evaluation pipelines to measure relevance, groundedness, factual accuracy, hallucination, and other quality metrics.

Integrate AI/LLM evaluation frameworks into CI/CD pipelines and establish automated quality gates before model, agent, or application deployments.

Own and maintain the AI regression testing strategy, ensuring automated test suites validate platform functionality after every release.

Develop automated testing for AI agents, RAG pipelines, LLM applications, and API contracts.

Design and implement AIOps, MLOps, LLMOps, and AgentOps workflows for the complete AI lifecycle.

Build observability solutions for AI systems covering model performance, agent behavior, latency, errors, data quality, and system health.

Design and implement scalable ETL/data pipelines supporting AI/ML workflows and evaluation frameworks.

Develop AI automation solutions using Python and SQL.

Design robust system architectures for AI-powered applications, testing platforms, and automation frameworks.

Design and implement MCP (Model Context Protocol) client-server architectures and integrations.

Collaborate with data scientists, ML engineers, software engineers, DevOps teams, architects, and business stakeholders.

Identify quality, performance, reliability, and scalability risks across AI systems and implement automated solutions to address them.

Establish engineering best practices for AI quality assurance, observability, automation, and continuous delivery.

Required Technical Skills

15+ years of overall IT experience.

3+ years of hands-on AI/ML experience.

Strong experience with:

Python

SQL

ETL/Data Pipelines

Machine Learning / Artificial Intelligence

AIOps

MLOps

LLMOps

AgentOps

AI Observability

Strong understanding of AI/ML testing and automation frameworks.

Experience with LLM evaluation methodologies, including relevance, groundedness, hallucination detection, and model output validation.

Experience building RAG pipeline testing and evaluation frameworks.

Strong knowledge of CI/CD automation and quality gates.

Experience with load, performance, scalability, and high-concurrency testing.

Strong system design and architecture experience.

Hands-on understanding of MCP client-server architecture and design.

Experience designing automated workflows across the AI/ML lifecycle.

Preferred Qualifications

Experience with enterprise-scale Generative AI and LLM applications.

Experience testing and monitoring AI agents and agentic workflows.

Experience with RAG architectures, vector databases, embeddings, and retrieval evaluation.

Experience implementing AI/ML observability and monitoring solutions.

Knowledge of cloud-based AI/ML platforms and modern DevOps practices.

Experience working in large-scale enterprise environments.

Strong communication, analytical, troubleshooting, and problem-solving skills.

Education

Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Science, Artificial Intelligence, or a related field preferred.

Work Location

Bellevue, WA Onsite

Employment Type & Compensation

Full-Time | $150,000 per annum