QA Engineer AI chatbot and GenAI testing and QA

Location: Charlotte, NC (100% Hybrid preference)
Openings: 3 Positions
Experience Level: Entry-Level (1 3 Years Experience; Internships Count)

Position Overview

We are seeking 3 Artificial Intelligence Quality Assurance Specialists to join our team as the first line of defense in evaluating cutting-edge generative AI experiences for personal finance products. In this data-heavy role, you will evaluate AI chatbot interactions and structured AI experiences to identify errors, trends, member frustration, and compliance gaps.

Because automated evaluations can only go so far, this role serves as the critical human feedback loop to ensure that AI output delivered to millions of members is accurate, empathetic, and aligns with partner guidelines and security policies.

Key Responsibilities

• AI Evaluation & Auditing: Daily review of 1 to 2 AI-driven member experiences (including conversational chatbots, debt payoff calculators, credit card/personal loan guidance, and tax refund allocations) for accuracy and user experience.
• Data Verification: Audit AI responses against raw customer data files such as full credit reports, bank statements, and transaction histories to confirm that generated metrics and financial figures are accurate and traceable.
• Stakeholder Collaboration & Feedback: Work closely with cross-functional product and engineering teams to highlight gaps, propose response improvements, and update evaluation rubrics and audit criteria.
• Quality Assurance Operations: Navigate customer data platforms and QA tools (e.g., MaestroQA) to log tickets, upload datasets, maintain sample distributions, and participate in calibration and "audit the auditor" programs.
• Data Analysis & Reporting: Utilize Microsoft Excel or Google Sheets to apply filters, analyze qualitative and quantitative trends, build charts, and compile pivot tables for insight readouts.

Qualifications & Background

• Experience: 1 3 years of experience in customer service quality assurance, banking/fintech operations, call center support, or Tier 1 technical support. Relevant internship experience is fully recognized.
• Domain Knowledge: A strong interest or background in personal finance, banking, credit reporting, or consumer lending.
• Analytical & Detail-Oriented Mindset: Exceptionally detail-oriented with a natural instinct to dissect complex contextual data and verify underlying sources.
• Technical Skills: Proficiency with spreadsheet tools like Excel or Google Sheets (pivot tables, data filtering). Familiarity with CRM systems, customer data platforms, or QA/ticketing software (e.g., MaestroQA/Rippit) is a plus.
• AI Familiarity: Basic understanding of AI/GPT response generation is preferred, though specialized prior AI experience is not required (comprehensive on-the-job training provided).
• Communication: Strong empathetic mindset with clear written and verbal communication skills.

Tech Skills

Excel Pivot Tables Data Filtering Data Auditing QA Software MaestroQA Rippit CRM Customer Data Platform (CDP) AI Quality Assurance Generative AI Evaluation GPT Models Machine Learning QA CSV Handling Ticketing Systems Data Analysis

QA Analyst AI QA Specialist Quality Assurance Specialist Customer Experience QA Analyst Fintech QA Analyst Banking Operations Specialist Tier 1 Tech Support Analyst Customer Service QA Specialist Data Quality Analyst QA Analyst Level 1