Job Closed
This listing is no longer active.
The world’s work marketplace.
Senior Lead Machine Learning Engineer
Location
California
Posted
150 days ago
Salary
$195K - $308K / year
Seniority
Senior
Job Description
Senior Lead Machine Learning Engineer
Upwork
• Design and implement comprehensive evaluation frameworks that reflect real-world task success for agentic systems, with a focus on human+AI collaboration outcomes • Build benchmarking pipelines that capture nuanced success indicators including trust calibration, intervention frequency, and agent handoff quality • Lead development of observability tools and instrumentation for analyzing agent behavior in production • Translate complex qualitative and quantitative signals into actionable insights that inform model iteration and product prioritization • Collaborate with researchers, engineers, and product teams to align evaluation methodologies with business and user goals • Own benchmarking infrastructure that enables reproducible, scalable evaluation across AI initiatives • Champion rigorous experimental design and statistical analysis across teams to ensure consistent and meaningful measurement standards
Job Requirements
- Proven experience designing evaluation systems for agentic or LLM-based AI, ideally in complex, interactive or open-ended environments
- Deep expertise in statistical experimentation, benchmark creation, and human-AI interaction assessment
- Fluency in building data pipelines and tooling using Python, SQL, and distributed data processing frameworks
- Demonstrated ability to influence product and model roadmaps through evaluation insights and performance measurement
- Adaptive-level proficiency in integrating AI tools into technical workflows for analysis, experimentation, and observability refinement
Benefits
- Comprehensive medical coverage for you and your family
- Unlimited PTO
- 401(k) plan with matching
- 12 weeks of paid parental leave
- Employee Stock Purchase Plan
Related Guides
Related Job Pages
More Machine Learning Engineer Jobs
Senior Staff Machine Learning Engineer, Ads Ranking
RedditReddit is an online platform utilized by thousands of communities to connect and converse about a wide variety of topics, including TV and movie fan theories, s
• Design and train advanced ML models (e.g., DNNs, transformers) to power Reddit Ads Ranking models. • Develop and optimize features including embeddings, contextual signals, and cross-session behavior. • Own experimentation strategy: offline metrics, A/B design, readouts, and “next iteration” planning. • Work closely with product, infra, and data teams to drive end-to-end model deployment and performance analysis. • Mentor other MLEs and contribute to modeling best practices across the org. • Shape the long-term modeling vision across one or more domains (conversion, app ads, shopping, brand, etc.).
Senior Staff ML Infra Engineer, Ads Ranking
RedditReddit is an online platform utilized by thousands of communities to connect and converse about a wide variety of topics, including TV and movie fan theories, s
• Architect and significantly influence ML training/serving systems and tooling that unlock faster iteration and larger models. • Drive adoption and reliability of ML systems and own a portfolio of initiatives across multiple teams. • Improve efficiency: GPU utilization, training runtime, data loading, feature performance, and serving latency. • Write high-quality design docs; run design reviews; set standards for correctness, reliability, and velocity. • Partner cross-org on shared infra sequencing, requirements, and adoption. • Mentor other engineers and contribute to engineering best practices across the org.
Senior AI/ML Engineer
LeidosLeidos is an innovation company rapidly addressing the world’s most vexing challenges in national security and health.
• Design, train, and deploy a wide range of AI/ML models for mission-critical applications, ensuring they meet performance and scalability requirements. • Implement and manage automated MLOps pipelines for model monitoring, retraining, and lifecycle management to ensure continuous delivery and reliability. • Optimize model performance, scalability, and resource consumption in production cloud and on-premise environments. • Collaborate with data scientists, software engineers, and systems architects to translate model prototypes into hardened, production-grade solutions. • Uphold software engineering best practices, including robust version control, comprehensive testing, and CI/CD processes. • Stay current with industry trends and best practices in MLOps and operational AI to continuously improve the team's capabilities.
Senior Staff Machine Learning Engineer
SpotifyPassionate music fans. Innovative tech pros. Perfect harmony. Join our band.
• Define and drive the machine learning technical strategy for Spotify Home, spanning retrieval, ranking, page composition, layout optimization, and real-time personalization • Work at the intersection of recommender systems, generative models, and user intent understanding to deliver highly adaptive and engaging Home experiences • Lead hands-on development of ML models and systems, from prototyping new ideas to productionizing solutions at global scale • Provide senior-level technical leadership across multiple teams, influencing architecture, modeling choices, and long-term investments • Design, build, evaluate, ship, and iterate on large-scale ML systems that directly power Home for hundreds of millions of users • Partner closely with product, design, and user research to translate UX goals and user needs into robust ML systems • Drive best practices in experimentation, offline/online evaluation, model lifecycle management, and reliability • Help evolve Home toward more contextual, generative, and intent-aware experiences, leveraging transformers, sequence models, and emerging techniques • Collaborate with platform and foundation teams to effectively leverage shared models and infrastructure while tailoring solutions to Home’s unique needs • Mentor senior engineers and influence technical standards through thoughtful reviews, documentation, and architectural guidance • Act as a technical ambassador for Home within Spotify’s broader ML community, staying current with research and industry trends



