Observability and Evaluation Engineer

Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
Create runbooks and support documentation for priority agent releases.
Analyze production behavior and recommend improvements to reliability, performance, and quality.
Required Qualifications
7+ years of engineering experience with observability, monitoring, test automation, platform operations, or AI/ML systems.
5+ years of Experience with dashboards, metrics, alerts, traces, logs, SLOs, and production monitoring.

5+ years of strong hands-on Python experience.

5+ years of Experience working in Agile engineering teams and production support environments.
5+ years of Understanding of LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment approaches.

Required Skills / Knowledge
Python, telemetry, tracing, monitoring, dashboards, alerting, SLOs, evaluation frameworks, test automation, and production operations.

Experience with observability tools and open telemetry concepts.
Understanding of LLMs, agents, RAG, prompt performance, retrieval quality, latency, and reliability metrics.