SWE-Bench Task Auditor
About this role
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity, and provide clear, rubric-based written feedback.
Basic Qualifications
- 3+ years professional software engineering
- Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
- Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
- Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications
- Familiarity with SWE-Bench (Verified) or similar repository benchmarks
- Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
- Prior code-review or task-grading experience
Details
- Pay: $70 to $90/hr
- Company: Mercor
- Location: Remote
- Eligible locations: United States
- Platform: Mercor
Frequently asked questions
How much does the SWE-Bench Task Auditor role pay?
Mercor lists this SWE-Bench Task Auditor position at $70 to $90/hr. Pay is set by Mercor and your exact offer depends on your experience and the scope of the work.
Is the SWE-Bench Task Auditor role remote, and can I work from home?
Yes. Mercor lists this SWE-Bench Task Auditor position as fully remote, so you can work from home or anywhere with a reliable computer and internet connection.
Is SWE-Bench Task Auditor full time or part time?
Mercor lists this as a contractor remote role. Full details are in the role description above.
Which countries can apply for SWE-Bench Task Auditor?
Mercor lists eligibility for United States. Check the Mercor listing for current requirements before applying.
Do I need experience to apply for SWE-Bench Task Auditor?
The requirements are listed in the role description above. Mercor screens applicants through its own process, and relevant skills or credentials strengthen your application.
How do I apply for the SWE-Bench Task Auditor role, and is it legit?
Use the apply button on this page to go straight to the official Mercor listing. Mercor is a real hiring marketplace and applying is free. No legitimate employer charges you to apply.
Stay Updated on Roles Like This
Subscribe to receive fresh openings aligned with AI Training & Evaluation and AI training roles across Mercor and JobHub by NeonLabs