• Define and measure conversation quality KPIs that align with product, support, and business goals. • Conduct systematic quality audits of AI conversations using both quantitative metrics and qualitative assessment, flagging conversations that went well and those that need improvement. • Analyze conversation data to identify failure patterns, drop-off points, conversion blockers, and untapped sales opportunities. • Partner with product and engineering to translate analytical insights into specific improvements: prompt refinements, conversation flow redesigns or updates to logic for how our AI responds to queries. • Establish quality assurance processes and frameworks that scale with conversation volume. • Support the integration of conversational quality tools and frameworks into existing workflows and tooling. • Present findings and recommendations to leadership with clear narratives connecting AI performance to business outcomes. • Lead cross-functional initiatives. • Own the AI conversation quality testing strategy, define standards, performance benchmarks, and a long-term improvement roadmap.

• Analyze conversation transcripts and user feedback to identify areas of confusion, failure, and prompt leakage. • Work with the Customer Impact Team Product Lead to define and track conversational KPIs (e.g., resolution rate, containment rate, user satisfaction). • Optimize prompts and model selection for cost efficiency, response latency, and scalability in production environments. • Collaborate with the engineers to improve conversation-specific evaluation criteria (e.g., NLU accuracy, intent recognition). • Design and maintain evaluation frameworks to measure prompt performance using golden datasets and automated scoring (e.g., LLM-as-judge, rubric-based scoring, precision/recall of intent routing). • Implement guardrails to reduce hallucinations, prevent prompt injection, and ensure compliant, safe responses. • Collaborate on design, map, and implement complex conversation flows, including error recovery and contextual handoffs (escalation to human support). • Own the continuous optimization of system prompts and instructions for LLMs (Gemini, OpenAI) to ensure Velo's response is accurate, tone is consistent, and on-brand. • Design and optimize structured outputs, function calling, and tool-routing logic to ensure accurate data capture and downstream system integrations.

• Build and maintain ScorePlay's AI playbook: which tools, which workflows, what "good" looks like • Define AI strategy in partnership with the CEO: where we go deep, where we standardize, what we skip • Run a continuous horizon scan on frontier tooling and surface high-signal opportunities early • Design and deploy multi-agent systems that automate high-volume, high-friction workflows across Sales, CS, Marketing, Finance, and People • Own the technical architecture for how agents connect to our stack (HubSpot, Notion, Slack, Linear, Granola, n8n, Claude/Anthropic API) • Build and iterate on internal tools and Claude-powered automations that reduce manual work and improve output quality • Define benchmarks and run evals to measure agent performance: accuracy, reliability, cost, adoption • Run structured weekly syncs with each Head of Department to challenge workflows, validate AI plans, and pressure-test roadmaps before execution • Co-design function-specific agent strategies that get used, not just documented • Build training materials and onboarding pathways so new hires are productive in our AI stack from day one • Define the KPI framework for AI adoption across the company, with revenue per employee as the north star • Audit the agent fleet: map every active agent, its permissions, cost, and impact • Set guardrails to prevent agent sprawl — redundant tools, conflicting automations, unchecked API costs • Report impact to the CEO and leadership on a regular cadence

• Build and Lead the AI Transformation Practice • Define service offerings around: Agentic workflow design and deployment, AI automation of operational processes, Replacement of legacy SaaS tools with AI-native systems, AI copilots for internal teams, AI infrastructure integration with existing data systems • Develop repeatable frameworks for assessing AI opportunities inside client organizations • Bring an initial client network or book of business where possible • Develop new client relationships through thought leadership, partnerships, and targeted outreach • Identify AI transformation opportunities within Vulcury’s consulting and venture studio ecosystem • Lead the design of AI-enabled workflows and systems for client organizations • Oversee deployment, testing, and iteration of AI systems within client environments • Own the P&L of the AI transformation practice • Develop scalable delivery models combining advisory, engineering, and implementation • Recruit and mentor technical consultants and AI specialists as the practice grows