Job Closed
This listing is no longer active.
Cresta is a software company using artificial intelligence and real-time coaching to transform the way sales and retention teams learn high-value skills. To do
QA Lead, AI Agent
Location
United States
Posted
93 days ago
Salary
0
Seniority
Senior
Job Description
QA Lead, AI Agent
Cresta
• Architect & Scale AI Evaluation Systems: Design and oversee the end-to-end framework for testing AI agent systems at scale. You will leverage LLM-driven methodologies—including automated simulations, "LLM-on-LLM" rubrics, and adversarial red-teaming—to ensure reliability, policy adherence, and logic across complex, multi-turn conversational flows. • Drive Deployment Excellence: Partner with Forward Deployed Engineers and PMs to triage issues, identify bottlenecks, and create new test cases on the fly to address real-world deployment challenges. • Be the Customer’s Voice: Conduct manual UAT and voice-call testing to represent the end-customer experience. You take it personally when an agent lacks empathy or clarity, and you excel at articulating these nuances to the engineering team and clients. • Lead and Scale the Team: lead a pod of QA analysts and partners. You will define the best practices, communication loops, and shared knowledge base that allow the QA function to scale alongside our rapidly growing product line.
Job Requirements
- 5+ years of experience in Quality Engineering, Deployments, or Technical QA, ideally within an AI or high-growth SaaS environment.
- Systems Thinking: A strong technical intuition and curiosity about how LLMs work. While you don't need to code, you must be comfortable navigating technical concepts like LLM, RAG, prompt logic, and multi-turn conversational flows.
- Operational Leadership: Proven ability to large E2E technical projects through partners, and a passion for building processes that improve efficiency between QA, Engineering, and Product.
- The "QA Nose": An uncanny ability to find the edge case and a bias toward action. You anticipate bottlenecks before they happen and deliver solutions with urgency.
- High Empathy: A consultative mindset with the ability to represent the "human element" of a customer support interaction.
- Startup Agility: You thrive in fast-paced environments, excel at turning ambiguity into execution, and are comfortable "rolling up your sleeves" to build.
Benefits
- Comprehensive medical, dental, and vision coverage with plans to fit you and your family
- Flexible PTO to take the time you need, when you need it
- Paid parental leave for all new parents welcoming a new child
- Retirement savings plan to help you plan for the future
- Remote work setup budget to help you create a productive home office
- Monthly wellness and communication stipend to keep you connected and balanced
- In-office meal program and commuter benefits provided for onsite employees
Related Guides
Related Categories
Related Job Pages
More QA Engineer Jobs
Quality Assurance Specialist
EmpowerWe are an equal opportunity employer with a commitment to diversity. All individuals, regardless of personal characteristics, are encouraged to apply. All qualified applicants will receive consideration for employment without regard to age, race, color, national origin, ancestry, sex, sexual orientation, gender, gender identity, gender expression, marital status, pregnancy, religion, physical or mental disability, military or veteran status, genetic information, or any other status protected by applicable state or local law.
• Monitor and provide feedback on the quality of phone interactions with participants by evaluating calls of representatives and advisors • Provide continuous call evaluation feedback to representatives and advisors through written evaluations • Document and escalate call monitoring issues to management as necessary • Evaluate representatives and advisors calls using the quality monitoring data management system to compile and track performance at an individual level • Extract and organize metrics to create month-end reporting and analysis on call monitoring results for internal quality reports • Facilitate call calibration exercises to validate calls are monitored consistently • Partner with internal resources (Compliance, Legal, Supervision, Training) on training and resources • Identify performance gap trends, conduct need/skill gap analysis • Contribute to appropriate learning solutions and documentation support • Participate in various training related projects, including developing training materials, attending meetings, and facilitating training for new representatives, advisors, and managers
Senior Manager, Marketing Data Ops, QA
EmpowerWe are an equal opportunity employer with a commitment to diversity. All individuals, regardless of personal characteristics, are encouraged to apply. All qualified applicants will receive consideration for employment without regard to age, race, color, national origin, ancestry, sex, sexual orientation, gender, gender identity, gender expression, marital status, pregnancy, religion, physical or mental disability, military or veteran status, genetic information, or any other status protected by applicable state or local law.
• Lead the audience testing and operational oversight of quality assurance framework for communications delivered through proprietary marketing automation platforms • Partner with Business Analysts and engineers reviewing technical documentation, foresee any data challenges and build test strategies to align to requirements. • Translate technical campaign requirements into structured test plans, timelines, resource planning and effort estimation. • Partner with data teams, solution architects, and IT to ensure proper system integration, data accuracy, and campaign execution readiness. • Identify operational risks, process constraints, and cross-functional dependencies impacting testing procedures and campaign delivery. And suggest mitigation plans. • Build, lead, and develop a team of QA analysts, set expectations, coach performance, and grow automation capabilities. • Partner cross-functionally to redesign testing process to meet evolving data infrastructure and architecture. • Investigate data discrepancies and complete root cause analysis on expedited timelines without impacting campaign deliverables.
• The Quality Lead oversees and actively participates in day-to-day embedded testing within the Agile Delivery team — balancing hands-on technical work with mentoring, workload delegation, and cross-team alignment. • You'll partner closely with your manager to review and improve QA artifacts, tools, and strategies. • Ensuring testing efforts across divisions are well-aligned and integrated into a comprehensive regression and performance suite.
QA Lead
Fulfillment IQeCommerce Fulfillment Product Studio that supports brands, retailers, and 3PLs with bespoke solutions.
• We are seeking an experienced Automation QA Lead to join our dynamic and growing team. • In this hybrid role, you will work hands-on with automation frameworks while also leading and mentoring a QA team of up to 4–5 Automation Engineers. • You will be responsible for driving quality strategies, overseeing automation efforts, and ensuring best‑in‑class delivery across our enterprise applications. • This role collaborates closely with development teams in India and North America, as well as business and customer stakeholders. • Perform end-to-end testing across FUT, SIT, and UAT phases. • Build, enhance, and maintain robust automation suites using Playwright + Python. • Lead validation testing and regression automation for critical business workflows. • Validate EDI transactions, business-critical reports, and ensure data accuracy across integrated systems. • Translate technical specifications into detailed test scenarios, test cases, and automation scripts. • Drive continuous improvements in automation frameworks, QA processes, and tooling. • Lead, mentor, and manage a team of 4–5 QA Automation Engineers, conducting regular check-ins, code reviews, and skill-building sessions. • Define and enforce QA best practices, automation standards, and coding guidelines across the team. • Oversee task allocation, workload balancing, and timely delivery across multiple parallel projects. • Partner with engineering leads, product stakeholders, and project managers to shape quality roadmap, risk assessments, and release strategies. • Provide leadership on test planning, test strategy, estimation, and quality metrics for all releases. • Ensure the team maintains high standards of test coverage, automation stability, and defect prevention. • Represent QA in cross‑functional discussions, release readiness reviews, and customer-facing updates.



