• Own end-to-end product experimentation: hypothesis generation, metric definition, experimental design (A/B, multivariate, sequential testing), analysis, and executive-level interpretation. • Design and maintain ML-powered evaluation frameworks for product changes, automation quality, and system reliability (e.g., order accuracy, routing, error rates, conversion). • Build and deploy predictive models, classifiers, and ranking systems that power experimentation, personalization, and product optimization. • Partner with product and engineering to test new features, workflows, and ML models through controlled experiments and incremental rollouts. • Lead offline and online model evaluation, comparing baselines, candidate models, and product variants using rigorous statistical methods. • Use causal inference and quasi-experimental methods when randomized experiments are not feasible. • Develop experiment pipelines and instrumentation: logging, dashboards, monitoring, and automated analysis to ensure measurement integrity. • Perform failure-mode and error analysis to guide product iteration and model improvement. • Translate experiment outcomes into clear product decisions, influencing roadmap prioritization and system design. • Drive experimentation at scale in a fast-moving environment, balancing speed, rigor, and business impact. • Lead and mentor data scientists and analysts, setting standards for experimentation, modeling, and evaluation across the organization.

• Design, evaluate, and interpret experiments in the presence of network effects, delayed outcomes, and imperfect randomization—balancing speed with statistical rigor. • Influence product direction by translating insights into clear recommendations that shape roadmap prioritization. • Develop key strategic insights through exploratory data analysis, to inform future investments or pivot in strategy • Build scalable metrics and dashboards to empower efficient decision-making • Own the definition and evolution of success metrics for core engagement surfaces, including tradeoffs between short-term and long-term member value.

• Leverage your technical and business acumen to become a specialist in our models, products, business and how the models interact with the business. • Identify potential problems or opportunities for improvement, communicate these issues to the team and suggest solutions. • Involves self-directed investigation into Upstart’s data and product, offline analyses using creative thinking as well as presentation to the ML team. • Receive mentorship from experienced DS/ML practitioners and collaborate closely with ML scientists to improve our products and models via data-driven insights. • Understand the underpinnings of ML models and how they create value for the business. • Identify data sources, use statistical methods and write clean code to conduct offline analyses about ML models and/or Upstart products. • Communicate findings clearly to get buy-in for recommended next steps. • Work with your mentor to align stakeholders, and drive next steps.

• Manage & oversee the work of Product Analytics team through stakeholder relationship management, defining projects’ scope of work, project prioritization & allocation, and ongoing people leadership & mentorship • Be a thought partner to department leadership by proactively proposing analytical solutions and data insights that drive improvement in product functionality, user experience, pricing and other product development and / or marketing decisions • Collaborate with department leadership and stakeholders to understand their business objectives, translate business objectives into proposed solutions (e.g. analysis, KPIs, BI, Data Science) • Perform (hands-on) analysis to drive insights, and inform business recommendations or marketing / PR content • Provide BI & Data Platform teams with requirements for product analytics data modelling, reports and dashboards • Collaborate with Data Science team on ML / AI developments supporting Product Analytics, GTM Marketing strategy and development of automated A/B testing capabilities • Advocate and propagate philosophy of impact-based opportunity assessment and quantify the impact assessment of all new ideas for product and experience’ improvement • Support experimentation initiatives by providing requirements for A/B design and executing success measurement and quantification