Role Description You will design and execute rigorous benchmarks and define dataset standards. Collaborating closely with our R&D team, you will build the evaluation infrastructure that guides the evolution of Pathway’s post‑transformer models. - Proactively identify, prioritize, and curate relevant public and client-driven benchmarks across our target use cases and markets. - Evaluate candidate benchmarks for clarity, data quality, evaluation methodology, and fit with our model roadmap. - Run benchmarks with baseline models to validate setup, uncover edge cases, and de‑risk R&D runs. - Hand off “benchmark-ready” packages to R&D (specs, data, evaluation scripts, expected metrics, constraints). - Maintain a shared vocabulary and documentation around benchmarks, datasets, and evaluation formats that GTM and R&D can both use. - Track and organize benchmark results, model leaderboards, and “what good looks like” for different customers and scenarios. - Contribute to demos and public‑facing proof points based on benchmark outcomes. You will play a key role in defining and driving the benchmarking process for AI model evaluation. Your work will directly influence what we build, how we talk about it, and how customers and the market experience BDH. Qualifications - You have published at least one paper at NeurIPS, ICLR, or ICML - where you were the lead author or made significant conceptual & code contributions. - You have significantly contributed to an LLM training effort which became newsworthy (topped a Hugging Face benchmark, best in class model, etc.), preferably using multiple GPUs. - You have spent at least 6 months working in a leading Machine Learning research center (e.g. at: Google Brain / Deepmind, Apple, Meta, Anthropic, Nvidia, MILA). - You were an ICPC World Finalist, or an IOI, IMO, or IPhO medalist in High School. Requirements - Have experience with ML/LLM evaluation, data science, or technical product roles, ideally around benchmarks or experimentation. - Are comfortable reading papers, leaderboards, and Github repos, and turning them into clear, repeatable benchmark specs. - Can talk comfortably with both engineers and customers, and translate between technical detail and business value. - Care about high‑quality data, reproducible experiments, and crisp documentation. - Are respectful of others. - Are fluent in English. Bonus Points - Published or open‑sourced work on LLM evaluation, benchmarking or data quality. - Experience designing custom benchmarks or evaluation protocols for novel model capabilities. Benefits - Join an intellectually stimulating work environment. - Be a pioneer: you get to work with a new type of "Live AI" challenges around long sequences and changing data. - Be part of one of an early-stage AI startup that believes in impactful research and foundational changes. Type of Contract Full-time, permanent Preferable Joining Date Immediate. The positions are open until filled – please apply immediately. Compensation Based on profile and location. Location Remote work. Possibility to work or meet with other team members in one of our offices: Palo Alto, CA; Paris, France or Wroclaw, Poland. Candidates based anywhere in the EU, UK, United States, and Canada will be considered. Additional Information If you meet our broad requirements but are missing some experience, don’t hesitate to reach out to us.

Role Description Templar is looking for a Research Scientist to perform novel research in decentralized LLM pre-training. You will develop and evaluate ideas for scaling large-scale pre-training in bandwidth-constrained, heterogeneous, and error-prone environments — the exact conditions that define real-world decentralized infrastructure. This is a research role with direct production impact. Your work will be ported to Templar's pre-training platform and published at major venues. If you want your research to advance the field and ship to a real system, this is that role. What You'll Do - Perform novel research in decentralized training of LLMs with a focus on pre-training - Develop, implement, and evaluate ideas for scaling large-scale pre-training in bandwidth-constrained, heterogeneous, and error-prone environments - Contribute to porting methodology to Templar's pre-training platform - Publish and present work at major conferences Qualifications - PhD in Machine Learning or equivalent research experience - Publications at top-tier venues such as NeurIPS, ICML, ICLR, CVPR, COLM, or EMNLP - Strong programming skills in PyTorch or JAX, with experience training models across multiple devices Requirements - Experience with distributed training or federated learning - Familiarity with efficient LLM techniques including quantization, optimization, inference, or architecture design - Background in optimization theory

Please submit your resume in English and indicate your level of English. At Toloka, we connect smart, curious people from around the world with freelance online tasks that train and improve artificial intelligence. What we do The Toloka Annotators connects individuals with Generative AI projects from leading tech innovators. Our mission is to unlock the full potential of AI by involving real people from around the world in the development process. About the Role Annotation is what helps AI make sense of the world. As an annotator, you may be invited to take part in online projects such as rating AI-generated content, evaluating factual accuracy, or comparing responses - when projects are available. Responsibilities: - Carefully review provided data (text, images, or videos) - Label or classify content based on project guidelines - Identify and flag factually incorrect, sensitive, inappropriate, or unclear material Important note: This is project-based work. Tasks are available only when projects are active. You may be invited to one or more projects depending on your profile and current opportunities. Each project has its own compensation level based on scope and expertise required. On this project, AI trainers earn up to $23 per hour equivalent.

Role Description - Develop high-quality data sets by creating challenging problems in General Relativity, Astrophysics, and Cosmology. - Evaluate and improve AI model performance using rigorous physics expertise. - Collaborate with AI research teams to enhance the quality of training data. - Work independently and asynchronously to meet deadlines and contribute effectively. - Follow complex instructions to ensure the accuracy and depth of AI-generated content. Qualifications - PhD in Physics with specialization in General Relativity, Astrophysics, or Cosmology. - Graduate degree from US/UK/Canada/Western Europe. - High attention to detail. - Exceptional written and verbal communication skills. - Excellent proficiency in English. Requirements - Contract position. - Compensation: $70–$90/hour. - Location: Remote. - Commitment: 4–6 tasks/week. Company Description