Job Closed
This listing is no longer active.
Machine Learning Engineer, Core Evaluations
Location
Europe
Posted
83 days ago
Salary
0
Seniority
Senior
Job Description
Machine Learning Engineer, Core Evaluations
Cantina Labs
• Designing model evaluation pipelines for models in development and production • Designing user studies for subjective model evaluations. • Converting requirements into measurable metrics. • Designing and developing automated evaluation dashboard to see model performances and compare results. • Training new models to capture new and different evaluation metrics. • Communicating with the model team to help design better models based on the evaluation results. • Communicating with the data team to help decide the type of data necessary to improve model performance. • Communication with the product-manager to make sure product requirements are correctly measured. • Help grow the evaluation team as the founding member. • Lead the evaluation team in the future.
Job Requirements
- Strong experience and intuition for designing metrics that capture model performance.
- Strong experience with designing user studies on Mechanical Turk or similar platforms. .
- Strong experience with model training and fine-tuning for model evaluation.
- Strong statistical knowledge and experience to statistically compare evaluation results and take decisions.
- Very strong engineering and programming skills.
- Experience with training ASR, TTS models.
- Experience at ML teams working on large-scale machine learning problems. (>3B models with >1m hours of data)
Benefits
- Please Note: to ensure that candidates select the most relevant jobs for their skills, we have set up limits to the number of times candidates can apply. The following limits apply to all roles at Cantina:
- Candidates may not apply more than 3 times in any 60 days for any job
- Candidates may not re-apply to any role within 30 days if not presented with an offer (for re-applying to the same job, the limit is 90 days)
Related Guides
Related Job Pages
More Machine Learning Engineer Jobs
Senior Software Engineer – Machine Learning, Marketplace Dynamics
YelpYelp helps people find great local businesses through crowdsourced reviews and paid advertising services. As an employer, Yelp aims to build a world-class team
• Identify and own challenging problems, form testable hypotheses, and drive significant business impact • Lead the design and analysis of experiments or development of causal and predictive models to test your ideas • Collaborate with product and engineering to affect changes in production systems and provide intelligence to other teams and communicate your conclusions to technical and non-technical audiences alike • Keep the team and our projects current on new developments in ML and statistics by reading papers and attending conferences and local events • Productionize and automate model pipelines within Python services
Senior Software Engineer – Machine Learning, Marketplace Dynamics
YelpYelp helps people find great local businesses through crowdsourced reviews and paid advertising services. As an employer, Yelp aims to build a world-class team
• Identify and own challenging problems, form testable hypotheses, and drive significant business impact • Lead the design and analysis of experiments or development of causal and predictive models to test your ideas • Collaborate with product and engineering to affect changes in production systems and provide intelligence to other teams and communicate your conclusions to technical and non-technical audiences alike • Keep the team and our projects current on new developments in ML and statistics by reading papers and attending conferences and local events • Productionize and automate model pipelines within Python services
• Ship production LLM agents with tool use: orchestration (state, pause/resume, journey/mode routing, failure handling) • MCP or gateway-style tool integration (schemas, auth, idempotency, audit) • Customized evals • Build alongside internal team
• Design, build, and deploy AI agent workflows that enable measurable efficiencies across business functions • Implement orchestration patterns including single-agent and multi-agent systems with appropriate state management • Integrate AI capabilities with existing tools and platforms via APIs, plugins, and function calling • Define and enforce guardrails for agent behavior, capabilities, and permissions in partnership with product and security • Build reusable automation patterns and templates that can be adopted across teams • Work alongside our data platform team to enable AI-driven analytics and reporting tools • Implement monitoring, observability, and feedback loops for deployed AI workflows • Ensure responsible AI practices, including fairness, transparency, and risk mitigation • Evaluate and recommend AI tooling, frameworks, and model providers based on business needs and cost • Stay current on industry trends, emerging AI capabilities, and best practices in automation engineering • Mentor engineers on integrating AI into their development workflows
