Quick overview
Job description
What We’re Building
We create high-quality coding problems for training AI models — drawn from real production repositories, not synthetic puzzles. Your role is to ensure these problems are fair, clear, and technically sound.
The Role
Part-Time Code Quality Reviewer | Remote | $40/hr | 0-25 hrs/week
We’re looking for engineers who can trace the intent behind a code change, understand what changes were made and why, and express that clearly in natural language.
This is judgment work, not mechanical testing. You'll make philosophical decisions about fairness and clarity in technical contexts.
What You'll Actually Do
Review code changes across unfamiliar repositories and tech stacks
Assess problem fairness: Could a skilled engineer solve this with only the given context?
Evaluate test quality: Do tests properly validate the intended solution?
Write clear justifications for your assessments (2-3 sentences per decision)
Flag coherence issues: Inconsistent artifacts, unclear requirements, missing context
Time commitment: 1-4 hours per problem, fully async — work whenever fits your schedule.
Who Should Apply
Ideal candidates:
PhD students in CS/computational fields seeking long-term flexible income
Open source contributors who care deeply about code quality
Undergrad with at least 1 year in CS, ECE, Computational Biology, or related technical programs
Professional engineers seeking intellectually interesting part-time work
You'll thrive if you:
Have strong software engineering fundamentals
Can work independently with minimal supervision
Write clearly and precisely
Enjoy making judgment calls in ambiguous situations
Stay focused during 1-3 hour review sessions
Requirements
1+ years of software engineering experience (internships, research projects, or professional work)
Fluent English; strong writing is a must
Responsive, independent, and attentive to detail
0–25 hours per week, flexible
Compensation & Growth
$40/hour for all review work
Performance-based advancement: Top contributors can receive promotions and bonuses
Long-term engagement: We're building a scalable team, not a temporary workforce
Application Process
15-minute async video interview — answer 4 questions about your technical background and work style
Sample problem review — demonstrate your assessment skills on a real coding problem
Onboarding — join our active problem design engineering community and start reviewing
FAQ
Q: Is this suitable for PhD students?
Yes — many of our top contributors are PhD students. The work is fully async and fits around research schedules.
Q: Do I need to be available during specific hours?
No. Work whenever you want. Some problem design engineers work weekends, others weekday evenings.
Q: What if I'm not familiar with the programming language in a problem?
That's expected. You're assessing problem fairness and clarity, not implementing solutions. Familiarity with general software engineering principles is what matters.
Q: How quickly can I start?
If you pass the video interview and sample review, you can start within 1-2 weeks.
Q: Is this a path to full-time employment?
This is a long-term contract role, not a pipeline to full-time positions. We value contributors who excel at this specific type of work.
About idlexperts
idlexperts is the quality assurance division of Idler (YC S25), where we ensure coding problems meet research-grade standards for AI training. Our mission is to build the most reliable QA system for AI coding benchmarks.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- UF

Application Lead Auto Apply
Uft · Irving · On-site 🇺🇸
SQLETLERP+10Salary not listed - PA

Member of Technical Staff, Core InfrastructureRecently posted Auto Apply
Parallel · San Francisco · On-site · Posted 20 Nov 2026 🇺🇸
GCPRustEnvoy+13$150k - $300k/yr - EX

Senior Infrastructure Software Engineer (Austin, TX)Recently posted Auto Apply
Exowatt · Austin · On-site · Posted 20 Nov 2026 🇺🇸
OnboardingSOC 2Compliance+6TechnologySalary not listed - RA

Sr. Deployment EngineerRecently posted Auto Apply
Robust Ai · San Carlos · Remote · Posted 20 Nov 2026 🇺🇸
SQLGitRobotics+9TechnologySalary not listed - QU

Software Engineer Auto Apply
Questanalytics · United States · Remote · Posted 9 Nov 2026 🇺🇸
SQLGitAWS+10TechnologySalary not listed - EA

Software Engineer, Cybersecurity Benchmarks Auto Apply
Epoch Ai · United States · Remote · Posted 10 Nov 2026 🇺🇸
Machine LearningTechnology$150k - $350k/yr - SA

Staff Software EngineerRecently posted Auto Apply
Samsara · United States · Remote · Posted 19 Nov 2026 🇺🇸
GraphQLIoTLLM+2Technology$154.7k/yr - NR

Associate Software Engineer - Java (Distributed Systems and Databases)Recently posted Auto Apply
New Relic · Portland · Hybrid · Posted 19 Nov 2026 🇺🇸
DockerDynamoDBGCP+9Technology$98k - $123k/yr - KO

Software Engineer, Controls (C++)Recently posted Auto Apply
Kodiak · Mountain View · Hybrid · Posted 20 Nov 2026 🇺🇸
MATLABSpringAWS+4Technology$145k/yr - IN

Senior Software EngineerRecently posted Auto Apply
Instawork · San Francisco · Hybrid · Posted 19 Nov 2026 🇺🇸
DjangoRoboticsAngular+8Technology$170k - $200k/yr - ME

Product Engineer, Talent ExperienceRecently posted Auto Apply
Mercor · San Francisco · On-site · Posted 20 Nov 2026 🇺🇸
GitLLMMental Health+3Salary not listed - PA

Product EngineerRecently posted Auto Apply
Paradigmai · San Francisco · On-site · Posted 19 Nov 2026 🇺🇸
ZustandReactTypeScriptSalary not listed
