Machine Learning Engineer, Evals
Location
United States
Posted
4 days ago
Salary
0
Seniority
Mid Level
No structured requirement data.
Job Description
Machine Learning Engineer, Evals
Nous Research
Role Description You'll work across the lab on agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together. This is a high-growth, high-ownership role on a small team, and you'll ship evaluation infrastructure that researchers depend on from day one. Responsibilities - Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task. - Build a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it. - Extend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA. - Run failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes. - Own a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use. Qualifications - 3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job. - Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short. - Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents. - Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it. - Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job). - Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric. - At least 3 of the following: - You can explain why LLM-as-judge needs calibration. - You've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues. - You know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each. - You've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples. - You've thought about non-determinism in eval, how you sample, how many runs, how you report variance. - You communicate clearly to both researchers and engineers, in the right language for each. - You're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help. Preferred - RLVR / RLHF pipeline experience. - Training data curation experience. - Distributed eval orchestration experience. - Benchmark design from scratch. - Red teaming and adversarial eval experience. - Familiarity with psychometrics or measurement theory.
Related Guides
Related Job Pages
More Machine Learning Engineer Jobs
Machine Learning Engineer
VerveVerve Napa Valley offers curated wine country events and tours and aims to introduce guests to Napa and Sonoma wine country "like only a local can." A one-stop
Role Description We are looking for a Machine Learning Engineer to join our engineering team to help us manage our diverse and growing set of initiatives. This position is full-time and 100% remote. - Experiment with emerging technologies and contribute to building new models and systems - Implement prototypes of the algorithms and models you design in Python - Focus on delivering solutions to production (this is not a research-only role) - Design training and evaluation protocols - Set up monitoring for performance metrics and overall system behaviour including alerts for any anomaly detected - Partner closely with the platform engineering team to streamline and optimize MLOps workflows - Use Kanban to manage multiple releases per week - Maintain high code quality through code reviews and automated tests Here are a few indicators that you're the right person: - You enjoy a fun, creative, and engaging working atmosphere free of brilliant jerks - You want to be part of a small team inside a large company with massive opportunity for growth - You enjoy collaboration with other teams including product, biz dev, and our in-house QA team - You eagerly dig into complex engineering problems Qualifications - You have hands-on experience implementing production machine learning systems at scale - Expertise with Python ML libraries like TensorFlow, PyTorch, Scikit-Learn etc. - Familiarity with ML tools like MLFlow, Ray Serving - Familiarity with building data pipelines including SQL and manipulating large structured or unstructured datasets for analysis - Familiarity with AWS and Google Cloud big data products Requirements - We welcome diversity and non-traditional paths into all of our roles. - We believe in hiring the right person as opposed to the right combination of keywords. Benefits - Verve provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.
Senior Machine Learning Engineer - Reinforcement Learning
Path RoboticsEnabling Robots To Build So That Humans Can Create.
Role Description As a Sr. ML Engineer focused on Reinforcement Learning, you will design, implement, and optimize RL algorithms that enable intelligent agents to operate in dynamic, unstructured environments. This role involves working closely with cross-functional teams to design, test, and deploy innovative solutions that improve the performance and capabilities of our robotic systems. This role can be located in our Columbus, Ohio Headquarters or Remote. - Design, implement, and evaluate RL algorithms for robotic control, motion planning, and adaptive behaviors in dynamic, unstructured environments. - Develop and integrate RL policies with robot control systems, ensuring compatibility with hardware constraints and real-time requirements. - Collaborate with perception teams to fuse RL with vision, depth, and sensor data for robust decision-making. - Build and maintain sim-to-real pipelines, including domain randomization and transfer learning techniques. - Conduct experiments on physical robots, including designing safety protocols and monitoring for unexpected behaviors. - Leverage simulation environments (Isaac Gym, Gazebo, MuJoCo, PyBullet) for large-scale training before real-world validation. - Continuously improve model efficiency to operate within compute and latency constraints on embedded robotic systems. Qualifications - Master’s or PhD in Computer Science, Robotics, Machine Learning, or related field, or equivalent practical experience. - Experience developing and deploying reinforcement learning algorithms on real-world systems. - Proficiency in Python and deep learning frameworks such as PyTorch or TensorFlow. - Experience with simulation environments (e.g., MuJoCo, Isaac Gym). - Solid understanding of probability, statistics, and optimization. - Experience with training and deploying ML models in production systems. Benefits - Daily free lunch to keep you fueled and connected with the team. - Flexible PTO so you can take the time you need, when you need it. - Comprehensive medical, dental, and vision coverage. - 6 weeks fully paid parental leave, plus an additional 6–8 weeks for birthing parents (12–14 weeks total). - 401(k) retirement plan through Empower. - Generous employee referral bonuses—help us grow our team! Company Description At Path Robotics we love coming to work to solve interesting and tough challenges but also because our ideas are welcomed and valued. We encourage unique thinking and are dedicated to creating a diverse and inclusive environment. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status. If you require a reasonable accommodation to participate in the application process or any part of the hiring process, please contact HR@path-robotics.com. We are committed to providing equal access and will work with qualified individuals to ensure a fair and accessible hiring experience. We will respond to your request within 48 hours.
Role Description Own recommendation, search, ranking, retrieval, and discovery for Sekai’s content ecosystem. What you will own: - Build recommendation and search across feed, discovery, search, and content continuation. - Own retrieval/ranking: candidate generation, embeddings, two-tower models, features, and serving quality. - Design, launch, and analyze recommendation/search experiments. Qualifications - 5+ years industry experience building production ML systems with senior ownership. - Hands-on recommendation, search, ranking, ads ranking, feed ranking, or content discovery systems. - Experience with consumer apps, entertainment, social, gaming, creator, or engagement-driven products. - Knowledge of two-tower models, embedding retrieval, candidate generation, ranking, and online/offline evaluation. Requirements - 5+ years industry experience building production ML systems with senior ownership. - Hands-on recommendation, search, ranking, ads ranking, feed ranking, or content discovery systems. - Experience with consumer apps, entertainment, social, gaming, creator, or engagement-driven products. - Knowledge of two-tower models, embedding retrieval, candidate generation, ranking, and online/offline evaluation. Benefits - Bonus signal: 5+ years production ML - Bonus signal: recommendation systems - Bonus signal: search ranking - Bonus signal: embedding retrieval Anti-signals - Cannot show core Senior Machine Learning Engineer, Recommendation experience. - Not comfortable with the listed work mode. - Low ownership, coordination-only, or no shipped examples. This is a remote position.
Senior Machine Learning Engineer
Encora DigitalEncora, a leader in digital engineering, drives innovation by crafting cutting-edge, cloud-first, data-first, and AI-first solutions that redefine industries. S
Role Description We at Coforge are hiring a Senior Machine Learning Engineer (#22151) with the following skill set. - Deploy, productionize, and optimize machine learning models, collaborating with Data Scientists to build scalable batch and real-time inference solutions. - Design and develop secure, scalable REST APIs and AI-driven data services that expose machine learning models and curated datasets for low-latency consumption. - Build and optimize high-performance inference services, improving model response times, throughput, caching strategies, and resource utilization. - Develop and maintain reliable data pipelines within Azure and Databricks, ensuring scalable, production-ready data and AI workflows. Qualifications - 5+ years of experience in Machine Learning Engineering, MLOps, or AI Platform Engineering. - Strong experience deploying machine learning models into production environments. - Hands-on experience with Azure services and Databricks for building scalable data and AI solutions. - Strong Python programming skills for machine learning and data engineering. - Experience designing and developing REST APIs and microservices for AI applications. - Experience building batch and real-time inference pipelines. - Knowledge of scalable model serving, performance optimization, and low-latency architectures. - Experience developing and maintaining production data pipelines and workflow orchestration. - Familiarity with MLOps best practices, version control, monitoring, and CI/CD pipelines. - Bachelor's degree in Computer Science, Engineering, Data Science, or a related field (or equivalent professional experience). Requirements - Experience with Azure Machine Learning, MLflow, or similar model lifecycle management tools. - Familiarity with dbt, Spark, and distributed data processing. - Experience with containerization technologies such as Docker and Kubernetes. - Knowledge of API security, authentication, and cloud-native architectures. - Experience working in Agile environments with cross-functional Data Science and Engineering teams. Company Description At Coforge, we hire professionals based solely on their skills and qualifications and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.
