Role Description We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks. Essentially, we're building an olympiad for AI models: the better the tasks, the more precisely we see where models are weak, and the faster they improve. What you'll actually do: - Create tasks and the criteria to evaluate them. Each task consists of: - An isolated environment - an emulation of a developer's workstation: a Linux machine with dev tools (terminal, CLI), MCP servers (repository, task tracker, messenger, documentation, etc.), and a real web application codebase; - A task - a prompt, just like in real work (a ticket link, a messenger conversation, anything). The agent figures it out on its own - reads the ticket, checks discussions, searches docs, explores the code. All of that context is also part of the task and needs to be created and thought through; - Tests and evaluation criteria - how to verify the agent solved the task correctly. - The work breaks down into two phases: - Building the virtual company - following a high-level plan, you create a company: codebase, infrastructure, context (conversations, documentation, tickets). The result is a realistic environment with development history. - Assembling and calibrating tasks - from intermediate states of the company, task drafts are assembled (based on a taxonomy). You turn a draft into a full task: write the prompt, define evaluation criteria, ensure the task is solvable and the evaluation is fair. - Then an AI agent receives the task and tries to solve it the way a developer would - and you evaluate the result. What the work looks like in practice: - Iterating with an AI agent on tests - they should catch real problems, not miss bad solutions, not break on good ones; - Reviewing code written by agents; - Analyzing why an agent failed or succeeded; - Designing edge cases and adversarial scenarios. A significant part of the work is done together with AI - it's very hard to create tasks that are hard for frontier models without using frontier models. Why this is hard: - Frontier models are already good at coding. - Creating a task that genuinely challenges the best models is non-trivial - you need to understand where models fail and what reveals the difference between a good and a bad solution. - Tasks have many valid solutions. - Tests must be neither too strict (breaking valid solutions) nor too lenient (passing bad ones). Core tech stack: - Backend: Python, FastAPI - Frontend: JavaScript/TypeScript, React - Infrastructure: Docker, Postgres, Kafka, Redis You don't need to be an expert in every item, but you should be comfortable reading and reasoning about code across the stack - that's what lets you design realistic tasks, write meaningful tests, and evaluate results. Growth: - You start with task and dataset work, and if it goes well, we expect you to move into a project team - with a broader scope and deeper involvement. Qualifications - Strong software engineering experience: 8+ years in any modern programming language, with a willingness and ability to work primarily in Python in our projects; - Hands-on experience using coding agents such as Claude Code and Codex as part of practical engineering workflows, while maintaining ownership of architecture, correctness, code quality, and delivery; - Problem-solving and clarity of thinking; - Ability to communicate technical decisions and reasoning; - Practical understanding of Docker and development workflows; - Overall readiness to contribute in a technical, execution-focused role. Benefits - Freelance collaboration, with a path into a project team if things go well; - Flexible, fully remote schedule; - Work at the leading edge of AI development, alongside a dedicated and dynamic team of experts; - Projects with customers that are AI industry leaders and well-known household names; - Friendly community. Important Notice Scam Alert Regarding Fake Job Postings: It has come to our attention that an individual or group is fraudulently impersonating Toloka to post fake jobs and solicit personal information from applicants. Please be aware: - Our recruiting team will only contact you from an official "toloka.ai" email address. We will NEVER use Gmail, Yahoo, Tolokainc, toloka.inc, or other personal or seemingly business email accounts. - We will never ask for your bank account details, credit card number, or any fees as part of the application or interview process. - All legitimate job openings are posted on our official careers page: https://toloka.ai/careers#job-list If you see a suspicious job posting or have been contacted by someone you suspect is a scammer, please do not provide any personal information. Instead, report the incident to us directly at security@toloka.ai and report the profile/post to LinkedIn. We are taking this matter very seriously and are working with the appropriate parties to resolve it. Thank you for your vigilance!