Model Evaluator, DevOps
About this role
About Turing
Turing is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.
About the Role:
We are seeking experienced Software Engineers to support a frontier AI training and evaluation initiative focused on improving the coding capabilities of AI agents.
In this role, you will review and compare AI-generated coding solutions across real-world software engineering tasks. You will analyze complete agent execution workflows, evaluate code correctness, efficiency, safety, and overall quality, and provide structured ratings and evidence-based feedback to help improve AI-assisted software development.
What does day-to-day look like:
- You'll review three AI coding agents that were each given the same real developer request.
- You read each agent's full run: the commands it ran, the files it changed, what it checked and its final message.
- Then you judge which run did the job best.
For every task you
- Rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication.
- Give each run an overall score from 1 to 5 and rank the three.
- Write a short rationale explaining your ranking, with evidence from the runs.
Required Skills: - Minimum 3+ years of overall experience
- Strong experience with at least one of the following languages: Python, JavaScript/TypeScript, Java, Go, PHP or similar.
- Proficiency with Shell commands, Git, Docker, and basic software pipeline setup.
- Comfortable running, modifying, and testing real-world projects locally.
Assessment & Selection Process
- Candidates must complete three practice evaluation tasks as part of the selection process.
- Each assessment requires quality ratings, overall scores, comparative rankings, and written rationales.
- A reviewer will assess each task for technical accuracy, consistency, and quality of reasoning.
- Candidates will be onboarded only after all three practice tasks have been approved.
IMPORTANT, STRICT NO-AI POLICY
- DO NOT USE CHATGPT OR ANY OTHER AI TOOL TO WRITE YOUR EVALUATION RATIONALES, EXPLANATIONS, OR REASONS.
- ALL RATIONALES AND REASONING MUST BE WRITTEN INDEPENDENTLY BY THE CANDIDATE.
- COMPLIANCE WILL BE CHECKED. ANY USE OF AI-GENERATED RATIONALES OR REASONS WILL RESULT IN AUTOMATIC REJECTION.
Perks of Freelancing With Turing
- Work in a fully remote environment.
- Opportunity to work on cutting-edge AI projects with leading LLM companies.
Offer Details
- Commitments Required: At least 8 hours per day and 40 hours per week with overlap of 4 hours with PST.
- Employment type : Contractor assignment (no medical/paid leave)
- Duration of contrac : 1 month; [expected start date is next week]
- Payment model: $25 per approved task
- Location: India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico, LA
Evaluation Process (approximately 75 mins) :
- Assessment based review
Key skills
Details
- Pay: Shared during the application process
- Company: Turing
- Location: Remote
- Eligible locations: India, Pakistan, Nigeria, Kenya, Egypt, Ghana and 3 more
- Platform: Turing
Frequently asked questions
How much does the Model Evaluator, DevOps role pay?
Turing does not publish a pay rate for this Model Evaluator, DevOps position. Pay is shared during the application process and depends on your experience and the scope of the work.
Is the Model Evaluator, DevOps role remote, and can I work from home?
Yes. Turing lists this Model Evaluator, DevOps position as fully remote, so you can work from home or anywhere with a reliable computer and internet connection.
Is Model Evaluator, DevOps full time or part time?
Turing lists this as a contractor remote role. Full details are in the role description above.
Which countries can apply for Model Evaluator, DevOps?
Turing lists eligibility for India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, and 1 more countries. Check the Turing listing for current requirements before applying.
Do I need experience to apply for Model Evaluator, DevOps?
The requirements are listed in the role description above. Turing screens applicants through its own process, and relevant skills or credentials strengthen your application.
How do I apply for the Model Evaluator, DevOps role, and is it legit?
Use the apply button on this page to go straight to the official Turing listing. Turing is a real hiring marketplace and applying is free. No legitimate employer charges you to apply.
Stay Updated on Roles Like This
Subscribe to receive fresh openings aligned with Finance & Accounting and AI training roles across Turing, Mercor and JobHub by NeonLabs