Engineering Manager, LLM Evaluation
About this role
About Turing
Turing is a leading AI company accelerating the advancement and deployment of frontier AI systems. We work with the world’s top AI labs and enterprises to improve capabilities in reasoning, coding, agentic behavior, multimodality, and other advanced AI domains.
Role Overview
We are looking for an Engineering Manager to lead teams creating high-quality conversational data for training and evaluating advanced Large Language Models (LLMs). The role involves managing the design of realistic, multi-turn interactions between users and AI assistants, including tool-based workflows involving calendars, email, maps, cloud storage, and other applications.
The manager will oversee project execution, maintain quality standards, guide technical decision-making, and collaborate with engineering, operations, and review teams to deliver reliable training and evaluation datasets.
What You’ll Do
- Lead and manage teams responsible for creating and evaluating multi-turn LLM conversations.
- Define workflows, priorities, quality standards, and delivery goals for conversation data projects.
- Guide the design of realistic assistant interactions involving APIs, JSON, function calls, and external tools.
- Review complex scenarios involving decision-making, clarification, error handling, constraints, and task completion.
- Ensure conversations accurately reflect realistic user expectations and assistant capabilities.
- Establish quality-control processes and monitor team performance, consistency, and deliverable quality.
- Coach team members on technical reasoning, conversation design, tool usage, and adherence to project guidelines.
- Collaborate with engineering, product, operations, and quality teams to resolve issues and improve workflows.
- Analyze reviewer feedback and implement process improvements to increase realism, accuracy, and training value.
- Manage project risks, timelines, escalations, and stakeholder communication.
What We’re Looking For
- 5+ years of professional experience in software engineering, technical operations, data, AI, or a related technical field.
- Prior experience managing, mentoring, or leading technical teams.
- Strong understanding of APIs, JSON, data formats, logical reasoning, and structured workflows.
- Ability to guide teams in breaking down complex tasks into realistic conversational and tool-use flows.
- Strong understanding of assistant behavior, including tool selection, clarification, limitations, and error handling.
- Excellent written and verbal English communication skills.
- Strong attention to detail and ability to establish and enforce quality standards.
- Proven ability to manage multiple priorities, provide actionable feedback, and deliver projects within deadlines.
- Ability to collaborate effectively with cross-functional teams and external stakeholders.
Perks of Freelancing With Turing
- Work in a fully remote environment.
- Opportunity to work on cutting-edge AI projects with leading LLM companies.
Offer Details
- Commitments Required: At least 4 hours per day and minimum 40 hours per week with overlap of 4 hours with PST.
- Duration of Contract: 3 months (adjustable based on engagement)
Key skills
Details
- Pay: Shared during the application process
- Company: Turing
- Location: Remote
- Eligible locations: Worldwide (no restriction)
- Platform: Turing
Frequently asked questions
How much does the Engineering Manager, LLM Evaluation role pay?
Turing does not publish a pay rate for this Engineering Manager, LLM Evaluation position. Pay is shared during the application process and depends on your experience and the scope of the work.
Is the Engineering Manager, LLM Evaluation role remote, and can I work from home?
Yes. Turing lists this Engineering Manager, LLM Evaluation position as fully remote, so you can work from home or anywhere with a reliable computer and internet connection.
Is Engineering Manager, LLM Evaluation full time or part time?
Turing lists this as a contractor remote role. Full details are in the role description above.
Which countries can apply for Engineering Manager, LLM Evaluation?
Turing lists this role with no country restriction. Check the Turing listing for current requirements before applying.
Do I need experience to apply for Engineering Manager, LLM Evaluation?
The requirements are listed in the role description above. Turing screens applicants through its own process, and relevant skills or credentials strengthen your application.
How do I apply for the Engineering Manager, LLM Evaluation role, and is it legit?
Use the apply button on this page to go straight to the official Turing listing. Turing is a real hiring marketplace and applying is free. No legitimate employer charges you to apply.
Stay Updated on Roles Like This
Subscribe to receive fresh openings aligned with Finance & Accounting and AI training roles across Turing, Mercor and JobHub by NeonLabs