Back to Jobs
Technology

Data Scientist Generative AI & RAG Evaluation

Tymon GlobalMountain View, CA🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationMountain View, CA, United States
SQLMicrosoft ExcelAWSMLflowDatabricksLLMPandasPythonGenerative AIData AnalysisLangChainLangGraph

Job description

We are hiring a Data Scientist with strong Applied AI experience to develop, test, and improve intelligent applications powered by Large Language Models. The role focuses on building reliable AI solutions, optimizing retrieval-based architectures, and establishing systematic evaluation approaches for LLM-generated outputs. Candidates should have strong Python and analytical capabilities, along with practical experience in model benchmarking, automated quality assessment, and human-assisted evaluation. Familiarity with modern AI orchestration frameworks and cloud-based ML platforms is expected.

Roles & Responsibilities

  • Develop and enhance intelligent applications using Large Language Models, advanced prompting strategies, Python-based data processing, and enterprise AI APIs.
  • Design retrieval-based AI solutions by implementing embedding generation, vector search, document indexing, context retrieval, and response optimization techniques.
  • Establish model assessment methodologies to measure the consistency, factual accuracy, relevance, and overall performance of GenAI applications.
  • Implement automated evaluation pipelines using LLM-as-a-Judge, human feedback mechanisms, and frameworks such as RAGAS, DeepEval, TruLens, and LangSmith.
  • Analyze model performance using quantitative measures, including precision, recall, F1-score, retrieval quality, groundedness, and response relevance.
  • Develop multi-step AI workflows and intelligent agent solutions using LangGraph, LangChain, and other orchestration technologies.
  • Perform exploratory data analysis, benchmarking, and evaluation reporting using Python, Pandas, SQL, Excel, and relevant ML experimentation tools.
  • Integrate AI monitoring, experiment tracking, evaluation logging, and model performance analysis using MLflow, Databricks, Arize AI, and AWS services.

Similar jobs