• Evaluation Strategy & Planning: Define and establish comprehensive evaluation strategies for new AI Agents. Prioritize the integrity and coverage of test data sets to reflect real-world usage and potential failure modes • LLM Output Integrity Assessment: Programmatically and manually evaluate the quality of LLM-generated content against predefined metrics (e.g., factual accuracy, contextual relevance, coherence, and safety standards) • Creating High-Fidelity Datasets: Design, curate, and generate diverse, high-quality test data sets, including challenging prompts and scenarios. Evaluate LLM outputs to proactively identify system biases, unsafe content, hallucinations, and critical edge cases • Automation of Evaluation Pipelines: Develop, implement, and maintain scalable automated evaluations to ensure efficient, continuous validation of agent behavior and prevent regressions with new features and model updates • Root Cause Analysis: Understand model behaviors and assist in the trace and root-cause analysis of identified defects or performance degradations • Reporting & Performance Metrics: Clearly document, track, and communicate performance metrics, validation results, and bug status to the broader development and product teams

• Define and establish comprehensive evaluation strategies for new AI Agents. Prioritize the integrity and coverage of test data sets to reflect real-world usage and potential failure modes • Programmatically and manually evaluate the quality of LLM-generated content against predefined metrics (e.g., factual accuracy, contextual relevance, coherence, and safety standards) • Design, curate, and generate diverse, high-quality test data sets, including challenging prompts and scenarios. Evaluate LLM outputs to proactively identify system biases, unsafe content, hallucinations, and critical edge cases • Develop, implement, and maintain scalable automated evaluations to ensure efficient, continuous validation of agent behavior and prevent regressions with new features and model updates • Understand model behaviors and assist in the trace and root-cause analysis of identified defects or performance degradations • Clearly document, track, and communicate performance metrics, validation results, and bug status to the broader development and product teams

• Design and implement efficient data pipelines using Python and AWS services (Lambda, S3, Glue, etc.) • Ensure data quality, reliability, and scalability across multiple sources and formats • Collaborate closely with data analysts, software engineers, and product teams to deliver actionable insights • Contribute to continuous improvement of our data infrastructure and best practices

• Design and implement efficient data pipelines using Python and AWS services (Lambda, S3, Glue, etc.) • Ensure data quality, reliability, and scalability across multiple sources and formats • Collaborate closely with data analysts, software engineers, and product teams to deliver actionable insights • Contribute to continuous improvement of our data infrastructure and best practices