We empower the restaurant community to delight guests, do what they love, and thrive.
Principal Machine Learning Engineer
Location
California
Posted
144 days ago
Salary
$180K - $368K / year
Seniority
Lead
Job Description
Principal Machine Learning Engineer
Toast
• Spend the majority of your time "hands-on-keyboard," architecting and coding high-performance backend services and ML pipelines • Build and prototype new internal products from scratch that leverage LLMs and Agentic AI • Design and implement ML models that provide real-time recommendations • Develop the backend between custom quoting engine, Salesforce, and internal data lakes • Act as a domain expert to solve complex synchronization and architectural challenges • Drive engineering excellence through code reviews and automated testing • Deliver significant core capabilities that have broad impact • Anticipate shifts in product needs and build flexible backend systems • Ensure the reliability of GTM tools
Job Requirements
- Extensive experience in backend languages (Java, Go, Python)
- Practical, hands-on experience deploying Machine Learning models and building with LLMs
- Experience building or deeply integrating with complex enterprise software
- Ability to design event-driven architectures and manage data flow
- Bachelor’s or Master’s degree in Computer Science, or a related technical field
Benefits
- Health insurance
- 401(k) matching
- Flexible work hours
- Paid time off
- Remote work options
Related Guides
Related Job Pages
More Machine Learning Engineer Jobs
Senior Machine Learning Engineer
TrueMLTrueML is a fintech company building software to create positive experiences for consumers seeking financial health.
• Architect the ML Ecosystem: You will own the end-to-end lifecycle of our ML infrastructure, designing a scalable, modern environment that enables models to thrive in production. • Productionize Innovation: Partner closely with our Data Science team to take complex algorithms from the 'lab' to the 'real-world', building the high-performance pipelines required to scale them. • Engineer Feature Intelligence: Design and maintain both offline and online feature stores, ensuring our models have the high-quality data they need for instant decision-making. • Escale the Data Platform: Collaborate with Data Engineers to evolve our data lake and ETL architectures, ensuring our data platform remains robust and future-proof. • Ensure System Health: Lead the monitoring and observability strategy for all production models, ensuring reliability and performance through proactive maintenance. • Shape Technical Strategy: Act as a key stakeholder in architectural decisions, helping the broader team define the strategy for our data products and event-driven architectures.
• Migrate box and barcode detection pipelines to cloud infrastructure following MLOps best practices • Build and maintain CI/CD pipelines for deployment across production and non-production environments • Implement automated rollback, canary, and blue-green deployment strategies for ML microservices • Build out a multi-tenant MLOps platform using tools like Prefect, ZenML, or similar orchestration frameworks • Establish a centralized model registry and versioning system for all production assets • Instrument observability across the ML stack — logging, metrics, and distributed tracing — to ensure reliability at scale
• Design, build, and deploy production-grade machine learning models and systems at scale • Own the full ML lifecycle: from problem definition and feature engineering to training, evaluation, deployment, and monitoring • Build scalable data and model pipelines with strong reliability, observability, and automated retraining • Work with large-scale datasets to improve ranking, recommendations, search relevance, prediction, content/user understanding, and optimization systems. • Partner cross-functionally with Product, Data Science, Infrastructure, and Engineering teams to translate complex problems into ML solutions • Improve system performance across latency, throughput, and model quality metrics • Research and apply state-of-the-art machine learning and AI techniques, including deep learning, graph & transformers based, and LLM evaluation/alignment • Contribute to technical strategy, architecture, and long-term ML roadmap
Senior Machine Learning Engineer
DatatonicGoogle Cloud AI+ML Partner of the Year. We drive business impact through innovative cloud engineering, analytics and AI.
• Interpret vague requirements and develop models to solve real-world problems. • Conduct ML experiments using programming languages with machine learning libraries. • Leverage generative AI to develop innovative solutions. • Optimise machine learning solutions for performance and scalability. • Implement tailored machine learning code to meet specific needs. • Ensure efficient data flow between databases and backend systems. • Automate ML workflows, focusing on testing, reproducibility, and feature/metadata storage. • Create machine learning architectures using Google Cloud tools and services. • Build and deploy production-grade software for machine learning and data-driven solutions.




