Senior Quality Assurance Engineer

Roles & Responsibilities :

• Design and implement test automation strategies for backend APIs, agent orchestration logic, and Angular-based agent UIs.
• Build and maintain Pytest-based test suites for Python backend services and agent workflow validation.
• Implement Playwright automation for Angular-based UI, covering interaction flows, approval queues, and streaming response handling.
• Validate multi-step agent workflows by asserting on intermediate states, tool call outputs, and handoff results, not just final API responses.
• Use Langfuse traces to validate agent reasoning chains, confidence scores, and execution paths.
• Build and maintain LLM evaluation harnesses covering prompt regression testing, RAG output validation, and model version change detection.
• Integrate test suites into Azure DevOps CI/CD pipelines as automated quality gates.
• Verify audit log integrity, including immutable log structure, correlation ID chains, and SOX-compliant event emission.

Test Automation - Primary Gate

• 5+ years of QA automation experience for API and web-based applications.
• Pytest as the primary test framework for Python backend and agent logic validation.
• Playwright for UI automation targeting Angular-based interfaces.
• REST API testing using Postman, HTTPX, or equivalent tools for contract and schema validation.
• Python for test scripting, strongly preferred given the stack.
• SQL/NoSQL for test data setup, teardown, and database state validation.
• Azure DevOps CI/CD pipeline integration for automated quality gates.

LLM / Agent Testing Awareness - Required

• Understands that LLM outputs are non-deterministic and knows why standard assertion patterns must be adapted, even without prior hands-on production experience.
• Familiarity with multi-step agent graph testing, including asserting on intermediate states, tool calls, and handoff outputs rather than just end results.
• Understanding of what Langfuse traces contain and how they can be used for agent validation. Hands-on production experience is a plus but not required.
• Awareness of model version regression risks and how to detect quality degradation when model versions change.