Quick Overview
Seniority
Mid Senior
Work mode
Hybrid
Location
New York, NY, United States
Posted
Yesterday
SpringAWSA/B TestingCOBOLComplianceJavaLLMPython
Job Description
Role: AI Evaluations Engineer (Senior)
Location: US (remote)
Duration: 12 Months +
Key Responsibilities
- Design and operate end-to-end evaluation frameworks for agentic AI systems (offline regression, online monitoring, A/B testing)
- Define quality rubrics, scoring architectures, and evaluation standards across multi-agent workflows
- Build LLM-as-judge and Agent-as-judge pipelines assessing trajectory, tool-call accuracy, groundedness, and safety
- Own and deliver evaluation infrastructure end-to-end; mentor teams and uplevel customers on evals best practices
Must Have
- 5-7+ years building production-grade ML or GenAI systems; advanced Python
- Hands-on experience with agent evaluation: offline/on-demand regression testing and online/continuous live-traffic scoring
- LLM-as-judge proficiency: rubric design, scoring at session, response, and tool-call granularity
- Core eval metrics: task completion, tool-call accuracy, trajectory quality, groundedness, safety
- Agentic framework experience (Strands, LangGraph, CrewAI or similar)
- AWS Bedrock (Models, Guardrails) + CloudWatch; Lambda, S3, IAM fundamentals
Should Have
- Bedrock AgentCore Evaluations (on-demand SDK runner, online monitoring, A/B testing)
- Agent-as-a-Judge framework; HITL evaluation pipelines
- Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction
- RAG quality evaluation (faithfulness, context precision, recall)
- CI/CD integration for eval regression; observability tooling (tracing, structured logging)
- Legacy technology familiarity (Java/Spring, Sybase, DB2, or COBOL) as source context for agentic modernization or integration pattern design
- Financial services or regulated-industry experience (compliance-aware delivery: trade surveillance, data controls, audit requirements)
- Large-scale portfolio delivery experience - reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement
Similar jobs
- VN
Hardware Test & Integration Engineer
NewVistance Networks
Wallingford, CT🇺🇸$87.4k/yrHybrid2 days agoFiberLESSPythonTechnology - PR
Software Engineer I/II
Prosync
Annapolis Junction, Maryland🇺🇸On-site4 days agoRustContinuous ImprovementC+++2Technology - SO
PlainID IAM Engineer
NewSystem One
Birmingham, AL🇺🇸HybridYesterdayAWSTechnology - SS
Manual QA Engineer
NewSouthern Scripts
Dallas, Texas🇺🇸Remote16 minutes agoSQLScrumAgile+3Technology - SS
Controls and QA Analyst
NewSouthern Scripts
Saint Louis, Missouri🇺🇸Hybrid16 minutes agoComplianceContinuous ImprovementHIPAA+1Technology - TE
Principle Network Engineer
NewTekSynap
Arlington, Virginia🇺🇸$48k - $125k/yrHybrid16 minutes agoLESSTechnology
