Applied AI Safety & Evaluation Researcher

• Develop product-specific threat models and harm taxonomies for conversational, recommender, and tool-using AI systems.
• Design and run single- and multi-turn adversarial evaluations, combining expert red teaming, automated attack generation, synthetic data, and sampled production data.
• Build reusable Python evaluation pipelines, LLM-as-a-judge workflows, regression tests, dashboards, and curated golden datasets.
• Validate evaluators against human labels and quantify coverage, judge reliability, false positives, false negatives, and safety–utility trade-offs.
• Turn findings into practical mitigations, including policy and prompt changes, context engineering, classifiers, data improvements, preference tuning, and system-level controls.
• Work directly with Engineering and Trust & Safety to embed evaluations into product-development and monitoring loops.
• Communicate results clearly to technical and non-technical stakeholders.
• Full-time availability is preferred, although part-time arrangements may be considered.

• You have personally delivered safety evaluations or mitigations for a real AI or machine-learning product, using automation to scale processes.
• You have strong coding agent and practical data-analysis skills, with sufficient programming and SQL to independently judge code and queries.
• You have experience designing adversarial tests, evaluation datasets, taxonomies, rubrics, and metrics.
• You can work autonomously when the risk, success criteria, and methodology are initially unclear.
• You have experience working across research, engineering, product, policy, or Trust & Safety.
• You communicate clearly in writing and have a record of turning research findings into action.

• Experience evaluating multi-turn or tool-using agents.
• Experience calibrating LLM judges or building human-in-the-loop evaluations.
• Experience with multilingual or multimodal evaluation.
• Experience with preference tuning or other model-alignment techniques.
• An MSc or PhD in an AI/ML-related field.

Notes