AI Service Experience Analyst

Title: AI Service Experience Analyst

Location: Remote in USA

Duration: Long Term

Position Summary

The AI Service Experience Designer will shape, refine, and continuously improve AI-enabled service experiences that are intuitive, accurate, compliant, and aligned to business outcomes. This role owns the business design of AI journeys, prompt frameworks, agent instructions, knowledge readiness, testing methods, and performance feedback loops—partnering closely with AI Engineers, product, technology, operations, compliance, governance, and vendor teams to bring scalable AI solutions to life.

Responsibilities

·         Own the business setup and ongoing refinement of AI-enabled service experiences, including experience objectives, user journeys, conversational flows, agent instructions, escalation paths, and desired outcomes.

·         Design, test, and optimize prompts, agent instructions, and interaction patterns for generative AI tools, AI agents, knowledge assistants, and other AI-enabled service solutions.

·         Assess, structure, modify, and maintain knowledge base content, procedures, job aids, scripts, and source materials to improve AI readiness, retrieval quality, response accuracy, and consistency.

·         Provide business oversight of AI agents by monitoring performance, reviewing outputs, identifying gaps or risks, validating behaviour against business expectations, and recommending corrective actions.

·         Define business requirements, test scenarios, acceptance criteria, quality standards, and success measures for AI-enabled service experiences and agent performance.

·         Conduct structured testing and quality reviews of prompts, knowledge retrieval, agent responses, conversational experiences, and end-to-end service flows using experimentation, user feedback, and performance data.

·         Analyze AI outputs for accuracy, relevance, clarity, consistency, usability, bias, risk, compliance considerations, and alignment with business intent; iterate and improve as needed.

·         Collaborate with technology, product, data, compliance, governance, operations, and vendor partners to ensure AI-enabled experiences are technically feasible, secure, scalable, governed, and aligned with approved AI guardrails.

·         Partner with AI Engineers to translate business-owned experience designs, prompt logic, agent behaviour expectations, knowledge requirements, test findings, and performance gaps into engineering-ready requirements, build considerations, troubleshooting needs, and enhancement recommendations.

·         Create and maintain reusable prompt frameworks, agent playbooks, knowledge readiness standards, testing methods, documentation, and business guidance to support consistent AI adoption and continuous improvement.

·         Measure and communicate AI experience effectiveness, agent performance, adoption, efficiency gains, service quality improvements, knowledge quality improvements, and contribution to business value or ROI targets.

·         Apply healthcare service journey and experience design principles to identify pain points, map current and future-state experiences, and shape AI-enabled solutions that support operational needs and improve employee, member, provider, or stakeholder experiences.

·         Stay current on emerging AI capabilities, prompt and agent design methods, knowledge management practices, model behaviour, evaluation techniques, healthcare payer trends, and industry best practices.

·         Perform any other job-related duties as requested.

Qualifications

·         Experience designing or improving AI-enabled service experiences, conversational flows, user journeys, knowledge assistants, AI agents, or generative AI interactions.

·         Strong understanding of prompt design, agent instructions, interaction patterns, knowledge retrieval, and AI response evaluation.

·         Experience working with knowledge base content, procedures, scripts, job aids, or source materials to improve accuracy, consistency, usability, and AI readiness.

·         Ability to define business requirements, acceptance criteria, quality standards, test scenarios, and success measures for digital or AI-enabled solutions.

·         Strong analytical skills with the ability to review AI outputs for accuracy, relevance, clarity, consistency, usability, bias, risk, and business alignment.

·         Experience collaborating with cross-functional teams, including technology, product, data, operations, compliance, governance, vendors, and engineering partners.

·         Ability to translate business expectations, experience designs, knowledge needs, prompt logic, and test findings into clear requirements and actionable recommendations.

·         Strong documentation skills, including the ability to create reusable playbooks, frameworks, standards, testing methods, and business guidance.

·         Comfortable working in an evolving AI environment with a continuous improvement mindset and a focus on measurable business value.

·         Healthcare, healthcare payer, member/provider service, or service journey design experience is a plus.

·         Familiarity with AI governance, approved AI guardrails, compliance considerations, and risk-aware design practices is a plus.