Role Description This role sits at the intersection of cutting-edge AI and scalable production systems, focusing on building the infrastructure that powers high-performing, reliable AI applications. You will play a foundational role in shaping how AI systems are evaluated, monitored, and continuously improved in real-world environments. Working in a highly autonomous and distributed team, you’ll contribute to developing intelligent features such as AI agents and copilots while ensuring their quality, speed, and accuracy at scale. The position offers a unique opportunity to influence a growing AI platform from the ground up, combining experimentation with robust engineering practices. If you thrive in fast-paced, async environments and enjoy turning data into actionable insights, this role offers both ownership and impact. - Design and build evaluation frameworks to measure AI system performance, including both offline development metrics and real-time production monitoring - Develop observability tools and dashboards to track system quality, latency, and accuracy over time - Investigate and diagnose performance issues across AI pipelines, identifying root causes such as retrieval, ranking, or prompt design flaws - Experiment with different models, architectures, and agent configurations to optimize system outcomes - Prototype and implement improvements to retrieval-augmented generation (RAG) pipelines, including chunking, retrieval, and re-ranking strategies - Analyze user interactions with AI features to uncover opportunities for enhancement and innovation - Collaborate closely with engineering teams to validate improvements and ensure system reliability - Stay up to date with emerging AI tools, frameworks, and best practices to continuously evolve platform capabilities Qualifications - 6+ years of experience in software engineering, data science, or machine learning, with exposure to domains such as NLP, search, or recommendation systems - Proven experience building and evaluating production-grade AI systems, including LLM-based applications and RAG pipelines - Strong proficiency in Python for experimentation, prototyping, and data analysis - Solid understanding of AI system architecture, including embeddings, retrieval mechanisms, and context management - Experience designing evaluation pipelines, A/B testing frameworks, and performance measurement systems - Ability to build internal tools and infrastructure such as dashboards and data processing pipelines - Willingness to work with and learn additional technologies (e.g., Ruby on Rails) for system integration - Strong analytical mindset with the ability to interpret complex data and communicate insights clearly - Comfortable working in fast-paced, ambiguous environments with high autonomy and accountability - Excellent collaboration and communication skills in a distributed, async team - Full professional fluency in English Requirements - Competitive salary package ranging from $130,000 to $140,000 USD annually, with regular compensation reviews - Equity participation and performance-based bonus opportunities - Fully remote work environment with flexibility to work from anywhere - Generous paid time off (35 days annually) plus sabbatical opportunities after long-term tenure - Comprehensive healthcare coverage for employees and their families (or equivalent reimbursement options) - Parental leave to support growing families - Home office setup allowance - Learning and development stipend for continuous skill growth - Biannual company retreats in international locations - High-trust, outcome-driven culture with strong emphasis on autonomy and impact

• Build agentic applications and workflows using the LLM frameworks • Design and implement tool integrations, function-calling patterns, and orchestration logic • Translate agent specifications and prompt strategies into robust, deployable services • Implement RAG pipelines, including vector store integration, chunking strategies, and retrieval optimization • Own the full lifecycle of agent systems from prototype through production • Build evaluation and observability infrastructure • Collaborate with Data Science on prompt engineering, model behavior tuning, and guardrail enforcement • Work with platform and infrastructure teams to deploy and maintain agent services • Contribute to internal tooling, SDKs, and shared libraries

Please submit your resume in English and indicate your level of English. At Toloka, we connect smart, curious people from around the world with freelance online tasks that train and improve artificial intelligence. What we do The Toloka Annotators connects individuals with Generative AI projects from leading tech innovators. Our mission is to unlock the full potential of AI by involving real people from around the world in the development process. About the Role Annotation is what helps AI make sense of the world. As an annotator, you may be invited to take part in online projects such as rating AI-generated content, evaluating factual accuracy, or comparing responses - when projects are available. Responsibilities: - Carefully review provided data (text, images, or videos) - Label or classify content based on project guidelines - Identify and flag factually incorrect, sensitive, inappropriate, or unclear material Important note: This is project-based work. Tasks are available only when projects are active. You may be invited to one or more projects depending on your profile and current opportunities. Each project has its own compensation level based on scope and expertise required. On this project, AI trainers earn up to $23 per hour equivalent.

Please submit your resume in English and indicate your level of English. At Toloka, we connect smart, curious people from around the world with freelance online tasks that train and improve artificial intelligence. What we do The Toloka Annotators connects individuals with Generative AI projects from leading tech innovators. Our mission is to unlock the full potential of AI by involving real people from around the world in the development process. About the Role Annotation is what helps AI make sense of the world. As an annotator, you may be invited to take part in online projects such as rating AI-generated content, evaluating factual accuracy, or comparing responses - when projects are available. Responsibilities: - Carefully review provided data (text, images, or videos) - Label or classify content based on project guidelines - Identify and flag factually incorrect, sensitive, inappropriate, or unclear material Important note: This is project-based work. Tasks are available only when projects are active. You may be invited to one or more projects depending on your profile and current opportunities. Each project has its own compensation level based on scope and expertise required. On this project, AI trainers earn up to $23 per hour equivalent.