Software Engineer AI Evaluation & Automation

Tittle – Software Engineer — AI Evaluation & Automation

Location- San Diego, CA / Santa Clara, CA (Locals only)

Contract – 6 - 12 months

Department is currently involved in creating automation testing tools to test large datasets.

Need candidates local to San Diego, CA or Santa Clara, CA as the team is based in those locations.

They are looking for candidates with AI evaluation automation experience.

Must-Have experience with Python or Javascript.

They must have experience using existing AI tools with harnesses such as Claude, Devin (Used to be Windsor), Cursor, etc., and the best practices for using them. - Non-Negotiable and must-be highlighted on the resume.

Must demonstrate a solid working knowledge of Git, including how branches, history, and working trees behave, and of containerization with Docker.

Candidates with experience with experimental design will be a BIG PLUS

Required Skills & Experience

• Strong software engineering background, with real experience building automation, developer tooling, or test and validation systems.
• Proficient in at least one general-purpose language such as Python, Java, or JavaScript — the specific language background is flexible.
• Solid working knowledge of Git, including how branches, history, and working trees behave, and of containerization with Docker.
• Experience with APIs, development environments, CI/CD pipelines, and typical engineering workflows.
• Understanding of how AI, LLM, or agent evaluation works and where it goes wrong, such
• as why a judge can be consistent but still wrong, why a single run can mislead, and how benchmark contamination happens.
• Able to troubleshoot technical problems, think clearly about whether a measurement is valid, and analyze results carefully.
• Hands-on experience using AI coding tools and agentic harnesses such as Claude Code, Devin, or Cursor, and command of the best practices for working with them effectively.

Preferred Experience

• Experience designing benchmarks or evaluations for software systems, especially execution-based grading that verifies against tests.
• Familiarity with LLM-as-judge or agent-as-judge approaches, and how to check them against human raters.
• Experience with build-system-aware test selection, such as Bazel or mapping changed files to the tests that cover them.
• Experience building reproducible test environments and managing versioned evaluation datasets.
• Comfortable writing up methodology and results for engineering leadership.