AI Quality Engineer Lead

Competency Area

Technical Expectations

Agentic AI Architecture

Understanding of AI agents, multi-agent workflows, MCP (Model Context Protocol), orchestration frameworks, tool calling, memory management, RAG patterns, and agent lifecycle management.

AI/LLM Validation & Assurance

Ability to define and implement testing strategies for AI systems including hallucination detection, response quality evaluation, guardrail testing, prompt validation, grounding verification, and reliability testing.

QE Automation Engineering

Strong hands-on experience with Playwright, Selenium, API automation, test frameworks, CI/CD integration, test data management, and automation architecture.

Agent Development & Customization

Ability to configure, extend, and customize agents for project-specific workflows, enterprise tools, APIs, business rules, and testing use cases.

AI Observability & Monitoring

Knowledge of agent telemetry, trace analysis, execution monitoring, prompt/response tracking, drift detection, performance analytics, and operational dashboards.

Cloud & Platform Engineering

Working knowledge of Azure AI, AWS Bedrock, OpenAI, Databricks, Kubernetes, containers, APIs, security integrations, and enterprise deployment patterns.

Data & API Engineering

Strong understanding of API contracts, service orchestration, structured/unstructured data, vector databases, embeddings, and data validation techniques.

Responsible AI & Governance

Experience validating security, privacy, compliance, explainability, bias detection, human-in-the-loop controls, and enterprise guardrails for AI agents.

Performance & Scalability Testing

Ability to validate agent response latency, concurrency, token consumption, workflow scalability, resiliency, and failover behavior.

Solution Architecture & Consulting

Capability to translate business use cases into agentic solutions, define reusable marketplace assets, establish standards, and mentor engineering teams.