Quick overview
Job description
Job Summary:
The Lead Software Engineer in Test – Agentic AI is a hands-on technical leader on Nextech’s horizontal Software Engineer in Test (SET) team, which serves as a platform layer supporting engineering teams across product development. This role advances the company’s Agentic Development Lifecycle (ADLC) by building AI-driven testing agents, shared test infrastructure, and test tooling, and by putting the governance and guardrails in place so agentic testing can scale.
All activities must be in compliance with Equal Employment Opportunity laws, HIPAA, ERISA and other regulations, as appropriate.
Essential Functions
-
In addition to adhering to Nextech’s conduct and behavioral standards, responsibilities include:
-
Design, develop, and maintain scalable automation frameworks and regression test suites across UI, API, backend, and AI-driven applications.
-
Lead quality engineering initiatives for Agentic AI systems, including LLM workflows, RAG pipelines, autonomous agents, and distributed orchestration platforms.
-
Validate complex AI agent behaviors such as multi-agent coordination, tool calling, prompt chaining, memory retention, conversational workflows, and fallback/retry mechanisms.
-
Develop and execute functional, integration, regression, end-to-end, and performance testing strategies across cloud-native and distributed environments.
-
Collaborate with Software Engineers, Product Managers, and DevOps teams to integrate testing throughout the software development lifecycle.
-
Build reusable automation solutions to validate APIs, orchestration layers, asynchronous workflows, enterprise integrations, and AI-driven business processes.
-
Design and maintain robust test environments, CI/CD integrations, cloud execution platforms, and automation pipelines to improve efficiency and reliability.
-
Establish and drive best practices related to code quality, maintainability, scalability, dependency injection, observability, and testability.
-
Lead validation efforts for AI workflow resiliency, distributed system reliability, data consistency, and orchestration performance under failure and edge-case conditions.
-
Mentor QA engineers and contribute to the development of scalable QA standards, automation strategies, reporting practices, and continuous improvement initiatives.
Success Measures:
- AI-driven test automation adoption and effectiveness.
- Agentic testing reliability, orchestration performance, and efficiency.
- Automated test coverage, execution efficiency, and defect detection improvements.
- Scalability, maintainability, and adoption of shared testing frameworks and infrastructure.
- Technical mentorship, engineering capability development, and adoption of best practices.
- Cross-functional collaboration, innovation, and measurable quality engineering improvements.
Minimum Requirements
- Bachelor’s degree in Computer Science, MIS, Engineering, or a related technical field.
- 8+ years of experience in Software Development or Software Engineer in Test (SDET/SET) roles.5+ years of hands-on experience developing automation frameworks using JavaScript/TypeScript or Java.
- Strong experience with automation tools such as Playwright, Cypress, Postman, K6, or equivalent frameworks.2+ years of experience testing AI-powered applications, LLM systems, Agentic AI platforms, orchestration frameworks, or autonomous workflows.
- Hands-on experience validating agent orchestration workflows, conversational AI applications, multi-step reasoning systems, and AI-driven business processes.
- Experience testing AI orchestration frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar technologies.
- Strong expertise in API testing, backend integration testing, UI automation, and end-to-end workflow validation.
- Solid understanding of automation architecture, design patterns, anti-patterns, dependency injection, and inversion of control principles.
- Experience working in cloud-native and containerized environments (Azure, AWS, GCP, Docker, Kubernetes). Proven experience integrating automated test suites into CI/CD pipelines and distributed cloud environments.
- Experience validating asynchronous, event-driven, and distributed systems with a focus on resiliency and fault tolerance.
- Strong analytical, debugging, troubleshooting, and problem-solving skills.
- Excellent organizational, communication, and stakeholder collaboration skills in Agile environments.
- Experience with AI observability, telemetry, RAG pipeline validation, vector databases, or prompt evaluation frameworks (preferred).
Preferred Qualifications
- Experience with Azure DevOps, GitHub Actions, or similar CI/CD platforms.
- Experience with AI observability tools, telemetry platforms, and prompt evaluation frameworks.Experience validating RAG pipelines, vector databases, and AI response accuracy.
- Knowledge of healthcare systems, HIPAA compliance, and regulated enterprise environments.
- Experience with event-driven architectures, messaging systems, and distributed data platforms (including SQL Server).
- Prior experience leading QA teams, mentoring engineers, and implementing enterprise test reporting and quality metrics practices.
Work Environment/Physical Demands
- Remote work
- Activities require a significant amount of work in front of a computer monitor.
Generous annual bonus opportunity
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- TO
Senior Software Engineer, Care AutomationRecently posted Auto Apply
Toast · Remote · Remote · Posted 18 Nov 2026 🇺🇸
MicroservicesGraphQLJava+2Technology$125k - $254k/yr - RL

Senior Flight Software Engineer IRecently posted Auto Apply
Rocket Lab Corporation · Littleton · Hybrid · Posted 19 Nov 2026 🇺🇸
RustEmbedded SystemsC+++2Technology$150k - $220k/yr - KP

Firmware Development EngineerRecently posted Auto Apply
Kairos Power · Alameda · Hybrid · Posted 19 Nov 2026 🇺🇸
Embedded SystemsFPGAVHDL+1Engineering$92.5k - $108.8k/yr - BP

Embedded Software EngineerRecently posted Auto Apply
Base Power · Austin · On-site · Posted 19 Nov 2026 🇺🇸
RustEmbedded SystemsC+++2TechnologySalary not listed - AI

Software Engineer, AgentsRecently posted Auto Apply
Adaptive Innovations · New York · Hybrid · Posted 19 Nov 2026 🇺🇸
DjangoDockerGCP+14TechnologySalary not listed - PI

Senior/Staff Software Engineer, Search & Retrieval InfrastructureRecently posted Auto Apply
Pinecone · New York City · Hybrid · Posted 19 Nov 2026 🇺🇸
RustC++Java+6TechnologySalary not listed - PI

Senior/Staff Software Engineer, Search & Retrieval InfrastructureRecently posted Auto Apply
Pinecone · US Remote · Remote · Posted 19 Nov 2026 🇺🇸
RustC++Java+6TechnologySalary not listed - AO
Safety & Systems Engineering InternRecently posted Auto Apply
Aurora Operations Inc · Mountain View · On-site · Posted 19 Nov 2026 🇺🇸
Engineering$58/hr - UP

Senior Software Engineer - Machine Learning PlatformRecently posted Auto Apply
Upstart · United States | Remote · Remote · Posted 18 Nov 2026 🇺🇸
Apache SparkAWSMLOps+7Technology$166.9k/yr - TT

Senior Software Engineer - Cloud ServicesRecently posted Auto Apply
The Trade Desk · Bellevue; Irvine; Seattle; Ventura · Hybrid · Posted 18 Nov 2026 🇺🇸
DockerAWSAnsible+11Technology$124.9k/yr - ST

Software Engineer II - Seller ExperienceRecently posted Auto Apply
StubHub · Aliso Viejo · Hybrid · Posted 18 Nov 2026 🇺🇸
SwiftC#Java+2Technology$165k/yr - PE

Staff Software Engineer (AI)Recently posted Auto Apply
Pendo · New York · Hybrid · Posted 17 Nov 2026 🇺🇸
LLMLangChainRAGTechnology$300k - $325k/yr
