Quick overview
Job description
Job Title: Software Engineer — AI Evaluation & Automation
Location: Remote
Duration: 6 Months with possible extension
Local to CA Only
Job Description:
Role Overview
Help build and scale the tooling we use to measure how well AI-powered software development tools actually perform. You'll develop evaluation harnesses, automate benchmark runs and help make sure the results we produce are reproducible and hold up to scrutiny. This is an engineering role, but a lot of the work is about getting the measurement right, not just automating it.
Key Responsibilities
- Build and integrate evaluation harnesses and automation for software development use cases, including turning real engineering artifacts like merged pull requests into repeatable benchmark tasks.
- Build versioned, repeatable processes to evaluate AI tools, models, and harnesses, with reproducible run environments (pinned dependencies, containerized runs, isolated worktrees) so results stay comparable over time.
- Validate and calibrate evaluation approaches against human judgment, so scores are consistent and correct rather than just repeatable.
- Support execution-based benchmarking across quality, productivity, and eJiciency measures, including cost and latency.
- Analyze results across repeated runs, looking at variance, failure patterns, and cost per outcome, and find ways to make the workflows more reliable and more automated.
- Work with engineering and data teams to improve the tooling, and document how the evaluations work and what they found for both technical and leadership audiences.
Required Skills & Experience
- Strong software engineering background, with real experience building automation, developer tooling, or test and validation systems.
- Proficient in at least one general-purpose language such as Python, Java, or JavaScript the specific language background is flexible.
- Solid working knowledge of Git, including how branches, history, and working trees behave, and of containerization with Docker.
- Experience with APIs, development environments, CI/CD pipelines, and typical engineering workflows.
- Understanding of how AI, LLM, or agent evaluation works and where it goes wrong, such as why a judge can be consistent but still wrong, why a single run can mislead, and how benchmark contamination happens.
- Able to troubleshoot technical problems, think clearly about whether a measurement is valid, and analyze results carefully.
Preferred Experience
- Hands-on experience using AI coding tools and agentic harnesses such as Claude Code, Devin, or Cursor, and command of the best practices for working with them effectively.
- Experience designing benchmarks or evaluations for software systems, especially execution-based grading that verifies against tests.
- Familiarity with LLM-as-judge or agent-as-judge approaches, and how to check them against human raters.
- Experience with build-system-aware test selection, such as Bazel or mapping changed files to the tests that cover them.
- Experience building reproducible test environments and managing versioned evaluation datasets.
- Comfortable writing up methodology and results for engineering leadership.
Manager Notes:
- Based on the job description, what are the must have non-negotiable items that a candidate must have to be successful in this role? They must have experience using existing AI tools with harnesses such as Claude, Devin, Cursor etc and the best practices using them. Evaluation and benchmarking experience (getting the measurement right, not just automating it). Solid working knowledge of Git, including how branches, history, and working trees behave, and of containerization with Docker.
- Does this position have the opportunity to extend beyond the initial contract or convert to FTE? Yes, the intention is to renew every 6 months, and if FTE positions become available contractors will be considered first
- Is there a specific laptop that needs to be used for this role (i.e. - Mac or Windows)? Client will provide the MAC
- Is there a required time zone? Must work Pacific Time zone. Not accepting candidates with more than a two-hour difference.
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- AS
Sr. Data Center EngineerRecently posted
Apex Systems · New Haven, CT · Hybrid · Posted 19 Nov 2026 🇺🇸
TechnologySalary not listed - AS
AI software engineerRecently posted
Algo Soft Solutions LLC · United States · Hybrid · Posted 19 Nov 2026 🇺🇸
MicroservicesAWSGoogle Cloud+3TechnologySalary not listed - BM

Mid-Level Software Engineer - Embedded SystemsRecently posted
Battelle Memorial Institute · Columbus, OH · Hybrid · Posted 19 Nov 2026 🇺🇸
MATLABRustEmbedded Systems+8Technology$100k - $110k/yr - BC
Associate Software Engineer - DeveloperRecently posted
Boeing Company · Tukwila, WA · On-site · Posted 19 Nov 2026 🇺🇸
MATLABFPGAC#+7Technology$106.3k - $143.8k/yr - CI

Systems EngineerRecently posted
CACI International, Inc. · Sterling, VA · Hybrid · Posted 19 Nov 2026 🇺🇸
MicroservicesConfluenceJira+3Technology$82.1k - $172.4k/yr - BA

Software EngineerRecently posted
Booz Allen Hamilton · Oklahoma City, OK · On-site · Posted 19 Nov 2026 🇺🇸
SQLScalaSpring+6Technology$86.8k - $198k/yr - HA

Senior Associate, Systems EngineeringRecently posted
Harris Corporation · Cincinnati, OH · On-site · Posted 19 Nov 2026 🇺🇸
EngineeringSalary not listed - BA

Systems Engineer, Senior
BOOZ, ALLEN & HAMILTON, INC. · Rome, NY · Hybrid 🇺🇸
Data AnalysisTechnology$86.8k - $198k/yr - MI

SYSTEMS ENGINEER I/II - COMBAT SYSTEMS INTEGRATION
MIKEL, Inc. · Middletown, RI · Remote 🇺🇸
LinuxNetwork AdministrationTechnologySalary not listed - IN

Software Engineer Level 4 with Security ClearanceRecently posted
IntelliGenesis · Annapolis Junction, MD · Hybrid · Posted 19 Nov 2026 🇺🇸
ConfluenceHadoopJava+3Technology$198.1k - $223.1k/yr - MS

HCM Functional LeadRecently posted
M9 Solutions · Trust, NC · Remote · Posted 20 Nov 2026 🇺🇸
OracleProject ManagementPeopleSoft$225k - $260k/yr - SA
Principal Systems EngineerRecently posted
SAIC · Suffolk, VA · Remote · Posted 20 Nov 2026 🇺🇸
TCP/IPActive DirectoryAnsible+12Technology$120.0k - $160k/yr
