• Operate across the full ML lifecycle (problem framing, data, modeling, evaluation, deployment, monitoring, and oncall), designing scalable ML pipelines and championing responsible AI (bias, safety, explainability) for ACU’s models and signals in production. • Provide technical leadership and mentorship to MLEs and SWEs doing ML work in ACU, design reviews, setting technical standards, and uplifting the team’s modeling and systems craft. • Develop evaluation systems and quality monitoring systems for content understanding signals, using SOTA LM-judge practices. • Drive operational excellence for ACU’s ML systems by defining SLOs, alerting, and dashboards for key signals (coverage, latency, precision/recall, cost) • Build and evolve content understanding capabilities for commercial conversations (e.g., reviews vs. recommendations vs. comparisons vs. Q&A; sentiment and stance; product entities and categories) and operationalize them as robust signals that power contextual and shopping ads, auto-targeting, new formats, and insights products. • Drive LLM and modern ML best practices within ACU: define when to prompt, finetune, or distill; design evaluation and safety harnesses; and lead at least one major distillation effort to replace external APIs with in-house models.

Role Description You'll work across the lab on agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together. This is a high-growth, high-ownership role on a small team, and you'll ship evaluation infrastructure that researchers depend on from day one. Responsibilities - Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task. - Build a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it. - Extend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA. - Run failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes. - Own a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use. Qualifications - 3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job. - Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short. - Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents. - Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it. - Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job). - Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric. - At least 3 of the following: - You can explain why LLM-as-judge needs calibration. - You've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues. - You know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each. - You've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples. - You've thought about non-determinism in eval, how you sample, how many runs, how you report variance. - You communicate clearly to both researchers and engineers, in the right language for each. - You're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help. Preferred - RLVR / RLHF pipeline experience. - Training data curation experience. - Distributed eval orchestration experience. - Benchmark design from scratch. - Red teaming and adversarial eval experience. - Familiarity with psychometrics or measurement theory.

• Develops, tests, deploys, and maintains software for recommendation engines • Collaborates in agile processes to create valuable user stories • Mentors junior engineers on software development practices • Leads system design initiatives and drives best practices in scalable architecture • Integrates AI and machine learning capabilities into the product recommendation journey

Role Description We are looking for a Machine Learning Engineer to join our engineering team to help us manage our diverse and growing set of initiatives. This position is full-time and 100% remote. - Experiment with emerging technologies and contribute to building new models and systems - Implement prototypes of the algorithms and models you design in Python - Focus on delivering solutions to production (this is not a research-only role) - Design training and evaluation protocols - Set up monitoring for performance metrics and overall system behaviour including alerts for any anomaly detected - Partner closely with the platform engineering team to streamline and optimize MLOps workflows - Use Kanban to manage multiple releases per week - Maintain high code quality through code reviews and automated tests Here are a few indicators that you're the right person: - You enjoy a fun, creative, and engaging working atmosphere free of brilliant jerks - You want to be part of a small team inside a large company with massive opportunity for growth - You enjoy collaboration with other teams including product, biz dev, and our in-house QA team - You eagerly dig into complex engineering problems Qualifications - You have hands-on experience implementing production machine learning systems at scale - Expertise with Python ML libraries like TensorFlow, PyTorch, Scikit-Learn etc. - Familiarity with ML tools like MLFlow, Ray Serving - Familiarity with building data pipelines including SQL and manipulating large structured or unstructured datasets for analysis - Familiarity with AWS and Google Cloud big data products Requirements - We welcome diversity and non-traditional paths into all of our roles. - We believe in hiring the right person as opposed to the right combination of keywords. Benefits - Verve provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.