Quick overview
Job description
This is a unique opportunity to define the future of AI quality in a domain where it matters most, mentoring a talented team and establishing the gold standard for testing intelligent systems that impact patient care. This is a full-time, remote position open to candidates in the U.S. or Canada.
What your impact will be:
- Build the core AI system end-to-end, from the service layer and data pipelines to the evaluation infrastructure, and own its architecture.
- Design and build the evaluation infrastructure, including dataset management, automated test harnesses, and replay of AI systems against historical workflows.
- Build and maintain evaluation datasets with clinician-reviewed correct answers, ensuring they evolve as our systems change.
- Create error metrics weighted by severity, so mistakes with clinical or financial impact are prioritized accordingly.
- Build production monitoring for accuracy, overrides, and safety issues, with actionable alerts for the engineering team.
- Write adversarial tests for AI systems, including hidden or malicious instructions in documents and invented content.
- Work with product and clinical leads to define accuracy thresholds and the criteria for expanding or pulling back an AI capability.
- The candidate must possess a curiosity and willingness to actively adopt and leverage emerging AI tools to improve workflows, solve problems, and drive efficiency along with being comfortable using a range of AI-enabled tools (such as copilots, chat-based AI, and automation solutions) as part of everyday work.
What we're looking for:
- 7 or more years in software or machine learning engineering, with at least 3 years focused on evaluation, testing, or quality for AI systems.
- A track record of designing and building complete systems from scratch, taking them from prototype to production.
- A track record of building and running evaluation systems that teams use in production, not just for one-off analyses.
- At least 2 years of experience evaluating large language model (LLM) systems in a production environment.
- Strong statistical skills for measuring accuracy, including sampling, confidence intervals, and error analysis.
- 1 year + of experience designing evaluation sets for document extraction, classification, or structured output.
- Deep knowledge of explainable AI (XAI) techniques and AI-powered tools (e.g., SHAP, LIME, Claude).
- Clear written communication, including design documents, and the ability to explain complex trade-offs to clinicians, engineers, and executives.
What would make you stand out:
- Experience working within the healthcare or another highly regulated industry.
- Familiarity with AI-specific testing techniques like metamorphic testing, adversarial testing, or data validation with tools like Great Expectations.
- A history of mentoring engineers and influencing technical direction across multiple teams.
What we offer:
- Plenty of opportunities to grow your career
- Comprehensive medical, dental, and vision benefits
- 3 weeks of vacation plus 5 personal days to recharge
- Employee stock ownership, RRSP program, 401k matching
- A chance to give back through community involvement
- Flexible work arrangements to suit your lifestyle
Salary range: 120,000-160,000 depending on experience and qualifications.
About Harris Computer:
Harris provides mission critical software solutions for the Public Sector, Healthcare, Utilities and Private Sector verticals throughout North America, Europe, Asia and Australia.
Working for Harris is the perfect opportunity to fulfill your professional goals as well as achieve your personal dreams!
Our employees enjoy a casual work environment that offers comfort while providing superior service to our customers. We offer a comprehensive benefit package as well as other additional "Perks"!
- We empower our employees to make a difference
- We have an award-winning culture
- We offer opportunity to learn
- We are financially strong and we are owned by the largest software company in Canada (CSI)
- We have fun!
#LI-remote
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- TC

Software test engineer with Security ClearanceRecently posted
TEKsystems c/o Allegis Group · Huntsville, AL · On-site · Posted 19 Nov 2026 🇺🇸
C++JiraAgileTechnologySalary not listed - KT

Remote Lead / Principal QA Automation ArchitectRecently posted
Kforce Technology Staffing · San Diego, CA · Remote · Posted 19 Nov 2026 🇺🇸
GitSeleniumC#+15TechnologySalary not listed - KT

Remote Senior Manager - QA AutomationRecently posted
Kforce Technology Staffing · San Diego, CA · Remote · Posted 19 Nov 2026 🇺🇸
Stakeholder ManagementSQLPerformance Management+2TechnologySalary not listed - IG

System Tester with Security ClearanceRecently posted
Insight Global, Inc. · Colorado Spgs, CO · On-site · Posted 19 Nov 2026 🇺🇸
AgileAdministrativeSalary not listed - GI
On-Site Senior Business/QA Analyst (Financial Systems - ECBA/CCBA/CBAP/PBA/Data Analytics Certification)Recently posted
Global Information Services · Tallahassee, FL · On-site · Posted 19 Nov 2026 🇺🇸
TreasuryJiraData Analysis+1TechnologySalary not listed - AP

QA Manual Tester LeadRecently posted
ApTask · San Francisco, CA · On-site · Posted 19 Nov 2026 🇺🇸
SQLERPDatabricks+4Salary not listed - RD

Senior Quality AssuranceRecently posted
Randstad Digital · TN · Hybrid · Posted 19 Nov 2026 🇺🇸
SQLGitScrum+10$85k - $95k/yr - DV
Sr. QA Engineer with Medical Devices exp. (Hybrid) :: Waltham, MARecently posted
DMS Vision Inc. · Waltham, MA · On-site · Posted 19 Nov 2026 🇺🇸
HealthcareSalary not listed - GF
QA Automation Engineer - W2 (No C2C or 1099) - Hybrid in Woodbridge, NJ (Posted SAM)Recently posted
Global Force USA · Woodbridge Township, NJ · Hybrid · Posted 19 Nov 2026 🇺🇸
SQLGitGenerative AITechnologySalary not listed - SC

SDET/ / / FullTime Role/TX/NC/MARecently posted
Source Code Technologies LLC · Dallas, TX · Hybrid · Posted 19 Nov 2026 🇺🇸
PlaywrightTechnologySalary not listed - AS

Automation QA TesterRecently posted
Acro Service Corporation · Carson City, Nevada · On-site · Posted 19 Nov 2026 🇺🇸
TechnologySalary not listed - AF
Test EngineerRecently posted Auto Apply
Accenture Federal Services · Lorton · Hybrid · Posted 19 Nov 2026 🇺🇸
GitSeleniumActive Directory+8Technology$86.4k - $176.2k/yr