• Design, build, and maintain robust automation frameworks specifically tailored for testing AI features, including LLM-powered outputs, recommendation engines, and intelligent workflows. • Develop and execute test strategies for validating AI model outputs, including accuracy, relevance, consistency, and bias evaluation across diverse input scenarios. • Perform functional, regression, performance, and scalability testing on AI-driven features, including API-level testing of AI service integrations and end-to-end automated test suites. • Collaborate with AI engineers, data scientists, product managers, and UX designers to define acceptance criteria and quality standards for AI features throughout the development lifecycle. • Identify, document, and track defects in AI behavior, including edge cases, hallucinations, and unexpected model responses through resolution using appropriate tools and workflows. • Champion quality and responsible AI practices throughout the development lifecycle, contributing to prompt engineering reviews, model evaluation rubrics, and AI safety checklists. • Build and maintain CI/CD-integrated automation pipelines that continuously validate AI feature quality, using tools such as Playwright, Claude, or equivalent frameworks. • Participate in team stand-ups, sprint planning, retrospectives, and continuous improvement activities, bringing an AI-quality-first mindset to every discussion and helping establish best practices for testing intelligent systems.

• Lead the design and development of infrastructure, automation frameworks, and internal engineering tools. • Build scalable services, APIs, dashboards, workflow engines, and integrations that improve developer efficiency and operational visibility. • Develop LLM-based workflows for triage, summarization, code and log analysis, test workflow assistance, report generation, and knowledge retrieval. • Integrate tooling with CI/CD systems, source control, issue tracking, test infrastructure, dashboards, and internal engineering services. • Define architecture, coding standards, evaluation methods, and reliability practices for automation and LLM-enabled systems. • Mentor engineers, review designs, and provide technical leadership across infrastructure and tooling projects.

Role Description We are seeking a seasoned, battle-tested QA Lead & AI Brand Evaluation Strategist to take full ownership of the quality, integrity, and strategic evaluation of our rapidly expanding suite of agentic and AI-powered product features. This is not a traditional, checklist-driven QA role. You will operate in a highly dynamic, non-deterministic environment, leading the strategy to evaluate probabilistic software at scale. Your core mission is to transform complex, ambiguous business requirements into robust evaluation criteria and scoring systems. You will lead the design of tools and frameworks that process massive volumes of data, ensuring our brand scoring metrics are precise, reliable, and actionable. What you'll do - Strategic QA Leadership: Own the end-to-end testing and evaluation strategy for multi-agent architectures and high-volume data pipelines, defining direction even when product requirements are fluid or ambiguous. - Business-to-Evaluation Translation: Act as the primary bridge between Product, Data Science, and Engineering. Translate complex business objectives into concrete, data-driven evaluation criteria and operational rubrics. - Brand Scoring Architecture: Oversee the methodology for scoring brands against complex criteria, ensuring that automated systems accurately transform massive datasets into precise, reliable business metrics (e.g., Share of Model, Net Sentiment). - Framework & Tool Building: Drive the creation of internal tools, data-driven testing pipelines, and validation frameworks capable of handling non-deterministic AI outputs and large-scale information ingestion. - System Integrity & Governance: Establish risk-adaptive guardrails and checkpoints to ensure data precision, compliance (PII), and logical reasoning across complex, multi-step agentic workflows. - Lead & Orchestrate Evaluation Frameworks: Define, architect, and oversee validation processes for application layers, complex API workflows, and the data transformation engine. - Define Brand Scoring Criteria: Establish the mathematical and logical rules that operate over massive data volumes to audit whether brand evaluations match underlying raw metrics reliably. - Direct "Golden" Test Set Curation: Lead the strategy for compiling, synthesizing, and maintaining massive baseline validation datasets to test edge cases, user intent, and model drift. - Drive Adversarial & Ambiguity Testing: Design "red-teaming" scenarios and boundary testing to evaluate how gracefully the system handles prompt variations, conversational drift, and highly ambiguous data inputs. - Tooling Innovation: Collaborate with engineering to build proprietary testing tools or automation scripts that streamline the ingestion and analysis of high-volume data for QA purposes. - Defect Profiling & Root-Cause Strategy: Move beyond simple bug logging; analyze trends in statistical behavioral defects and guide cross-functional teams toward systemic root-cause resolution. - Platform Observability & Metrics: Monitor cloud logging and data observability dashboards to track execution latency, data drift, and accuracy trends, transforming insights into platform optimizations. Qualifications - 8+ years of proven experience in QA Engineering, Systems Analysis, or Software Testing, with at least 2+ years in a Lead or Strategic role managing complex, data-heavy digital platforms. - Proven "cancha" (battle-tested experience) leading QA initiatives in fast-paced environments where requirements are not fully defined, showing a high capacity to resolve ambiguity and establish order. - Strong data-driven background. Exceptional capability to shift away from purely binary (pass/fail) testing toward evaluating data trends, composite scoring, and statistical outcomes. - Comfortable working with big data contexts—understanding how information flows from raw ingestion (data lakes) through transformation layers to user-facing dashboards. - Deep expertise in traditional and modern software testing methodologies, Agile environments, risk management, and project tracking ecosystems (such as JIRA). - Practical experience with Python, JavaScript, or SQL to query databases, manipulate data structures, and trigger/build automated evaluation pipelines. - Deep operational awareness of LLM behaviors, agentic workflows, and typical AI challenges (hallucinations, context limits, instruction adherence). Preferred Skills and Qualifications - Experience with LLM evaluation frameworks (e.g., Ragas, LangSmith, TruLens) or prompt engineering playgrounds. - Familiarity with cloud data warehouses and analytics platforms (e.g., BigQuery, Databricks, Google Cloud Platform components). - Experience building internal scripts or lightweight tools specifically designed to automate QA data validation.

• Build a Universal Provisioning Engine and custom integration layers that satisfy external auditors while keeping internal teams focused on shipping features. • Eliminate the engineering "interruption tax," provide leadership with a real-time, data-driven view of our risk posture, and ensure continuous compliance with zero audit surprises. • Engineer a centralized security telemetry system that programmatically captures control evidence and health in real-time, transitioning from manual snapshots to continuous data streams for an 'always-on' view of our security posture. • Architect solutions for systems that lack native IGA support. • Partner with our risk management function to build a secure mechanism to generate agentic risk assessments workflows using the data and results from what is collected.