AI Benchmark Quality Reviewer

About the role

We are looking for a detail‑oriented reviewer to assess the quality and fairness of challenging AI benchmark tasks. You will examine task instructions, model execution traces and grading behavior to determine whether results reflect true model performance or issues with the task, grader or environment.

Key responsibilities

• Check task instructions, source materials, reference solutions and evaluation criteria for consistency and completeness.
• Review model execution traces, tool calls and deliverables to assess whether successes and failures are justified.
• Identify brittle grading checks, unsupported criteria and valid alternative solutions that may have been marked incorrect.
• Investigate discrepancies and distinguish model limitations from task, grader, tool or environment issues.
• Write concise, evidence‑backed findings and verify that revisions address the issues found.

Required profile

• At least five years of relevant technical or analytical experience.
• Strong written English, analytical judgment and attention to detail.
• Ability to give specific, reproducible feedback and explain uncertainty clearly.

Required skills

• Ability to read Python code.
• Ability to read SQL queries.
• Ability to read shell scripts and execution logs.
• Familiarity with structured data analysis.

What we offer

• Remote contractor assignment for eight weeks, 40 hours per week.
• Compensation of $23 per hour.
• Eight hours of daily overlap with Pacific Time.