Haystack
← Back to Jobs
Other

Content Evaluator/Reviewer

Amiseq Inc.United States🇺🇸United StatesPosted 21 Jul 2026

Why This Role Stands Out

This hybrid Content Evaluator role at Amiseq Inc. offers a fantastic opportunity to sharpen your analytical skills and contribute to improving search relevance by assessing document quality and user intent. You'll thrive if you have a keen eye for detail, enjoy research, and are eager to develop your understanding of information retrieval within a flexible work environment. Apply today to join a dynamic team and make a real impact on how users find information.

Quick Overview

Work Type
Hybrid
Level
Mid Senior

Job Description

About the role:

  • As a relevance evaluator you will be responsible for evaluating how relevance a given document (or image, video) is for a particular user query.
  • You will have to rate the document relevance 1-5 (from not relevant at all to exactly what the user was looking for).
  • Rating relevance requires first trying to understand the user intent, for example, where they researching a topic, or they were looking for their own document, and forming an opinion what would be a good document (or image, etc.) for this particular query for this particular user.
  • Other similar tasks involve evaluating quality of assistant answers, including various aspects of the answer: is it correct (the information answers the user intent correctly), is it descriptive, or is the system apologizing for not knowing the answer. Understanding correctness also requires some research on the terminology used.

 

Responsibilities:

  • Responsibility includes reviewing the query and document pairs, including any context provided by the system (such as when the query was issued and who did it), researching the problem trying to understand the user intent by using the product (search, including formulating queries for elastic search index in KQL), and then rating the document or other required artifact (e.g. assistant answer) according to the provided relevance rubric.

 

Requirements:

  • Understanding of English
  • Ability to formulate search or chatgpt-like queries

 

Similar jobs

Apply now