Job Title: AI Architect – AI Observability

Location: Schaumburg, IL
Work Mode: Hybrid – 1 day per week onsite
Duration: Short-term engagement (3–4 weeks or less)
Engagement Type: Contract

Job Description

We are looking for an experienced AI Architect with expertise in AI Observability to support a short-term engagement focused on evaluating, designing, and improving observability for AI/ML applications and systems. The ideal candidate will have a strong understanding of AI architecture, monitoring, tracing, performance evaluation, and troubleshooting of AI-powered applications, including LLM-based solutions.

Key Responsibilities

• Assess existing AI/ML application architecture and observability capabilities.
• Design and recommend observability strategies for AI/ML and Generative AI applications.
• Implement or enhance monitoring, logging, tracing, and performance measurement for AI workloads.
• Establish visibility into LLM interactions, model behavior, latency, token usage, errors, and resource consumption.
• Define metrics and dashboards to monitor AI application reliability, performance, and operational health.
• Evaluate AI observability tools and recommend suitable solutions based on project requirements.
• Identify bottlenecks, troubleshoot issues, and recommend improvements to AI application performance and reliability.
• Collaborate with engineering, DevOps, and platform teams to establish observability best practices.

Required Qualifications

• Proven experience as an AI Architect, AI/ML Engineer, or similar technical leadership role.
• Strong knowledge of AI/ML architecture and AI observability practices.
• Hands-on experience monitoring and troubleshooting AI/ML or Generative AI applications.
• Understanding of LLM observability, distributed tracing, application logging, metrics, and performance monitoring.
• Experience with observability tools and frameworks such as LangSmith, Langfuse, Arize Phoenix, OpenTelemetry, Datadog, or similar platforms.
• Familiarity with LLM evaluation, prompt tracing, model performance, latency, token consumption, and error analysis.
• Strong analytical, problem-solving, and communication skills.

Engagement Details

• Short-term assignment lasting approximately 3–4 weeks or less.
• Hybrid work arrangement with one day per week onsite in Schaumburg, IL.
• Candidates should be available to assess the current environment and provide practical recommendations or implementation support within a limited timeframe.