Other

AI Agentic Engineer

Quality IT Source, LLCVienna, VA🇺🇸United StatesPosted Sep 24, 2026

Why This Role Stands Out

This hybrid AI Agentic Engineer role offers a unique opportunity to shape the future of intelligent systems with a recognized IT company. You'll thrive here if you have extensive experience building production-level LLM-powered agentic systems, leveraging your expertise in tool use, multi-agent workflows, and robust guardrails. Apply now to join a forward-thinking team and advance your career in cutting-edge AI development.

Quick Overview

Seniority

Mid Senior

Work mode

Hybrid

Location

Vienna, VA, United States

Posted

2 weeks ago

MLflowA/B TestingLLM

Job Description

· 10+ years of AI/ML engineering experience, with at least 3 years building LLM-powered systems that have run in production with real users.

· Hands-on experience building agentic systems: tool use and function calling, multi-step and multi-agent workflows, orchestration frameworks, state and session management, and recovery from partial or failed tool calls.

· Demonstrated experience designing guardrails and policy enforcement for agents, scoped permissions, action gating, escalation to humans.

· Strong RAG fundamentals: chunking, embeddings, vector databases, indexing, and re-ranking, with a track record of diagnosing and measurably improving retrieval quality.

· Demonstrated experience evaluating GenAI systems quantitatively: golden/labeled datasets, offline and online eval pipelines, LLM-as-judge scoring paired with human review, regression and A/B testing.

· Hands-on experience with GenAI evaluation and observability tooling (e.g., Ragas, DeepEval, LangSmith/Langfuse, Amazon Bedrock model evaluation, MLflow), including running evals in CI and tracking metrics over time.