Why this role stands out
This remote role offers a unique opportunity to shape the future of AI by leveraging your deep engineering expertise to evaluate cutting-edge coding agents, with a highly competitive hourly rate of CAD 100-200. You'll thrive here if you possess exceptional judgment and a keen eye for sophisticated problem-solving, allowing you to contribute to a groundbreaking field from anywhere in the world. Apply now to make a significant impact on AI development.
Quick overview
Job description
Contract | Remote, worldwide | $100-$200/hour depending on experience and location | 10-20 hrs/week
Please note that eligibility may vary depending on your location, as some regions may not be supported by the client.
Check out this Loom video for more details: https://www.loom.com/share/b0d1b0bf24c44ae8b95dca84b9db60e5
We're looking for highly experienced software engineer (SR+) to help evaluate the quality of interactions with modern coding agents such as OpenAI Codex and Claude Code.
This is not a traditional engineering role. You won't be writing production code. You'll be evaluating something harder: whether the model thinks like a great engineer.
What this role actually is
You will assess how AI coding agents behave in real-world scenarios, focusing on:
• Whether the response makes sense
• Whether the preamble and reasoning are useful
• Whether the output reflects strong engineering judgment
• Whether the interaction feels right to an experienced developer
This role is about engineering taste. Syntax correctness is the easy part.
What you'll be doing
• Evaluate AI-generated coding interactions end to end
• Judge whether outputs are useful, correct at a high level, and aligned with how a strong engineer would think
• Assess the quality of explanations and reasoning, not just the code
• Distinguish between levels of response quality (what makes something a 2 vs a 4)
• Give clear, opinionated feedback in writing on what worked, what didn't, and what felt off or misleading
• Help define what great looks like when working with tools like Cursor, Codex and Claude Code
What we mean by "taste"
We're looking for engineers who can answer questions like:
• Does this feel like something a strong engineer would actually say?
• Is this explanation helpful, or just technically correct?
• Is the model guiding the user well, or just dumping output?
• Would this interaction build or erode trust?
You should be comfortable making subjective but rigorous judgments, and explaining them clearly.
Who you are
• Senior, Staff or Principal-level engineer (or equivalent experience)
• Strong background in TypeScript/JavaScript or Python
• Hands-on experience with at least one of OpenAI Codex, Claude Code or Cursor
• Deep familiarity with modern AI-assisted dev workflows
• Able to evaluate code without executing it or reviewing every line
• Strong written and spoken English (B2 or above). You'll be writing detailed feedback and recording short video explanations, so this is a hard requirement
• Comfortable giving direct, opinionated feedback
• High bar for what good engineering looks like
Nice to have
• Prior exposure to prompt design or evaluation workflows
• Experience mentoring senior engineers or defining engineering standards
Engagement details
• Rate: $100-$200/hour depending on experience and location
• Hours: 10-20 hours/week, flexible
• Duration: ongoing. Projects run from about two weeks to a few months each, and we offer new ones to evaluators who do well
• Start: as soon as you clear the take-home and a project has an open seat
• Process: one take-home evaluation exercise with a recorded Loom walkthrough. No interview
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Similar jobs
- AF

Senior Software Engineer, Backend (Workload Safety)Recently posted Auto Apply
Affirm · Remote Canada · Remote · Posted 17 Nov 2026 🇨🇦
PythonKubernetesMavenTechnology$16k - $24k/yr - SI

Associate Application Engineer | 1768Recently posted Auto Apply
SiTime Corporation · Ottawa · Hybrid · Posted 17 Nov 2026 🇨🇦
CADIoTPython+1TechnologyC$78.7k - C$108.2k/yr - CP
Senior Gameplay ProgrammerRecently posted Auto Apply
CD PROJEKT RED · Remote · Remote · Posted 16 Nov 2026 🇨🇦
CADUnreal EngineEngineeringC$127.2k - C$209.9k/yr - UB
Pipeline Technical Lead - HelixRecently posted Auto Apply
Ubisoft · Montreal, QC · On-site · Posted 16 Nov 2026 🇨🇦
ManufacturingSalary not listed - RY
Firmware EngineerRecently posted Auto Apply
RYSE Inc · Toronto, Ontario · Hybrid · Posted 16 Nov 2026 🇨🇦
GitEmbedded SystemsIoT+2TechnologySalary not listed - TA

Senior Software Engineer - RDB/SPORecently posted Auto Apply
Tanium · Remote · Remote · Posted 16 Nov 2026 🇨🇦
RustGitAWS+10TechnologyC$154k - C$236k/yr - ST

Software Engineer, IdentityRecently posted Auto Apply
StackAdapt · Canada · Hybrid · Posted 16 Nov 2026 🇨🇦
OnboardingMicroservicesApache Spark+8TechnologyC$140k/yr - ST

Senior Software Engineer, InventoryRecently posted Auto Apply
StackAdapt · Toronto · Hybrid · Posted 16 Nov 2026 🇨🇦
OnboardingMicroservicesRust+8TechnologyC$140k/yr - ST

Integration Reliability Engineer, Technical OperationsRecently posted Auto Apply
Stripe · Toronto · Hybrid · Posted 16 Nov 2026 🇨🇦
SQLSplunkLLM+2TechnologySalary not listed - ST

Product Integration Reliability Engineer, Money ManagementRecently posted Auto Apply
Stripe · Toronto · Hybrid · Posted 16 Nov 2026 🇨🇦
RubySQLGit+3TechnologySalary not listed - RO

Senior Software Developer, Kubernetes ComputeRecently posted Auto Apply
Robinhood · Toronto · Hybrid · Posted 16 Nov 2026 🇨🇦
RecruitingAWSLoad Balancing+4TechnologyC$166k - C$195k/yr - PI

Sr. Software Engineer, iOSRecently posted Auto Apply
Pinterest · Toronto · Hybrid · Posted 16 Nov 2026 🇨🇦
SwiftXcodeiOSTechnologySalary not listed
