Job Closed

This listing is no longer active.

Cresta logo
Cresta

Cresta is a software company using artificial intelligence and real-time coaching to transform the way sales and retention teams learn high-value skills. To do

QA Lead, AI Agent

Location

United States

Posted

94 days ago

Salary

0

No structured requirement data.

Job Description

QA Lead, AI Agent

Cresta

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more. Role Description At Cresta, shipping AI is only half the story. Ensuring that AI interacts with humans reliably, accurately, and empathetically at scale is where the real challenge lies. As the QA Lead, AI Agent , you will be the ultimate guardian of the customer experience for our AI Agent product line. This role is perfect for a strategic quality expert who loves the intersection of human psychology and machine logic. You will own the end-to-end quality strategy, from designing complex test plans for non-deterministic LLMs to building automated and scalable testing environments using Cresta's proprietary no-code test and evaluation tools. You aren't just looking for bugs; you are building the framework that allows Cresta to deploy world-class AI agents for the world's largest enterprises with total confidence. What You’ll Do: - Architect & Scale AI Evaluation Systems: Design and oversee the end-to-end framework for testing AI agent systems at scale. You will leverage LLM-driven methodologies—including automated simulations, "LLM-on-LLM" rubrics, and adversarial red-teaming—to ensure reliability, policy adherence, and logic across complex, multi-turn conversational flows. - Drive Deployment Excellence: Partner with Forward Deployed Engineers and PMs to triage issues, identify bottlenecks, and create new test cases on the fly to address real-world deployment challenges. - Be the Customer’s Voice: Conduct manual UAT and voice-call testing to represent the end-customer experience. You take it personally when an agent lacks empathy or clarity, and you excel at articulating these nuances to the engineering team and clients. - Lead and Scale the Team: Lead a pod of QA analysts and partners. You will define the best practices, communication loops, and shared knowledge base that allow the QA function to scale alongside our rapidly growing product line. Qualifications - 5+ years of experience in Quality Engineering, Deployments, or Technical QA, ideally within an AI or high-growth SaaS environment. - Strong technical intuition and curiosity about how LLMs work. - Proven ability to lead large E2E technical projects through partners. - Consultative mindset with the ability to represent the "human element" of a customer support interaction. - Thrives in fast-paced environments and excels at turning ambiguity into execution. Requirements - An uncanny ability to find the edge case and a bias toward action. - Comfortable navigating technical concepts like LLM, RAG, prompt logic, and multi-turn conversational flows. - Experience with CCaaS (Contact Center as a Service), telephony, or STT/TTS (Speech-to-Text) technologies (bonus). - Background in Conversation Design or SDET roles (bonus). - Experience leading a team with direct reports (bonus). Benefits - Comprehensive medical, dental, and vision coverage with plans to fit you and your family. - Flexible PTO to take the time you need, when you need it. - Paid parental leave for all new parents welcoming a new child. - Retirement savings plan to help you plan for the future. - Remote work setup budget to help you create a productive home office. - Monthly wellness and communication stipend to keep you connected and balanced. - In-office meal program and commuter benefits provided for onsite employees. Compensation at Cresta Cresta’s approach to compensation is simple: recognize impact, reward excellence, and invest in our people. We offer competitive, location-based pay that reflects the market and what each individual brings to the table. Compensation for this position includes a Base salary + Bonus + Equity. Actual base salaries will be based on candidate-specific factors, including experience, skillset, and location, and local minimum pay requirements as applicable. Your recruiter can provide further details. We have noticed a rise in recruiting impersonations across the industry, where scammers attempt to access candidates' personal and financial information through fake interviews and offers. All Cresta recruiting email communications will always come from the @cresta.ai domain. Any outreach claiming to be from Cresta via other sources should be ignored. If you are uncertain whether you have been contacted by an official Cresta employee, reach out to recruiting@cresta.ai.

Job Requirements

  • 5+ years of experience in Quality Engineering, Deployments, or Technical QA, ideally within an AI or high-growth SaaS environment.
  • Strong technical intuition and curiosity about how LLMs work.
  • Proven ability to lead large E2E technical projects through partners.
  • Consultative mindset with the ability to represent the "human element" of a customer support interaction.
  • Thrives in fast-paced environments and excels at turning ambiguity into execution.
  • An uncanny ability to find the edge case and a bias toward action.
  • Comfortable navigating technical concepts like LLM, RAG, prompt logic, and multi-turn conversational flows.
  • Experience with CCaaS (Contact Center as a Service), telephony, or STT/TTS (Speech-to-Text) technologies (bonus).
  • Background in Conversation Design or SDET roles (bonus).
  • Experience leading a team with direct reports (bonus).

Benefits

  • Comprehensive medical, dental, and vision coverage with plans to fit you and your family.
  • Flexible PTO to take the time you need, when you need it.
  • Paid parental leave for all new parents welcoming a new child.
  • Retirement savings plan to help you plan for the future.
  • Remote work setup budget to help you create a productive home office.
  • Monthly wellness and communication stipend to keep you connected and balanced.
  • In-office meal program and commuter benefits provided for onsite employees.
  • Compensation at Cresta
  • Cresta’s approach to compensation is simple: recognize impact, reward excellence, and invest in our people. We offer competitive, location-based pay that reflects the market and what each individual brings to the table.
  • Compensation for this position includes a Base salary + Bonus + Equity. Actual base salaries will be based on candidate-specific factors, including experience, skillset, and location, and local minimum pay requirements as applicable. Your recruiter can provide further details.
  • We have noticed a rise in recruiting impersonations across the industry, where scammers attempt to access candidates' personal and financial information through fake interviews and offers. All Cresta recruiting email communications will always come from the @cresta.ai domain. Any outreach claiming to be from Cresta via other sources should be ignored. If you are uncertain whether you have been contacted by an official Cresta employee, reach out to recruiting@cresta.ai.

Related Categories

Related Job Pages

More QA Engineer Jobs

Full TimeRemoteTeam 10,001+Since 1899H1B Sponsor

• Lead continuous improvement at the factory through quality initiatives • Directly influence factory QMS through vendor and factory quality personnel collaboration • Deliver consistently high-quality product as measured through QM KPIs • Act as a coach and mentor for VF QA team and factory QA • Evaluate new vendors for factory quality setups • Support and monitor factory accredited lab testing SOP against Brand standards • Manage vendor performance and drive continuous improvement

Vietnam
Job Closed
Deltek logo

Senior Performance Test Engineer

Deltek

Deltek is a leading provider of information solutions and enterprise software. Founded in 1983 by father-and-son team Donald and Kenneth deLaski, Deltek serves more than 15,000 cli

QA Engineer94 days ago

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more. Role Description We are looking for a highly skilled Senior Performance Test Engineer with strong experience in API and UI performance testing , deep understanding of CI/CD integration , and hands-on exposure to AI-powered testing tools . The ideal candidate will be responsible for designing, executing, and optimizing performance testing strategies to ensure scalability, reliability, and optimal user experience across distributed systems and microservices architectures. Key Responsibilities - Performance Strategy & Planning - Define performance testing strategy aligned with business and non-functional requirements. - Identify critical user journeys, API endpoints, and system bottlenecks. - Establish SLAs, SLOs, and performance benchmarks. - Conduct risk assessment for scalability and system capacity. - API Performance Testing - Design and execute performance tests for REST/GraphQL APIs and microservices. - Perform load, stress, spike, soak, and endurance testing. - Validate throughput, response time, concurrency, and resource utilization. - Experience with tools such as: - Apache JMeter - k6 - LoadRunner - UI / Front-End Performance Testing - Analyze and optimize web application performance. - Measure metrics such as page load time, Time to First Byte (TTFB), First Contentful Paint (FCP), and Largest Contentful Paint (LCP). - Experience with tools such as: - Lighthouse - WebPageTest - Playwright - CI/CD Integration - Integrate performance tests into CI/CD pipelines using tools such as: - Jenkins - GitHub Actions - Azure DevOps - Configure automated performance regression tests in pipelines. - Implement performance gates and threshold-based build failures. - Generate automated performance reports and dashboards. - Monitoring & Observability - Analyze application performance using APM and monitoring tools such as: - Dynatrace - New Relic - Grafana - Identify memory leaks, CPU spikes, thread contention, DB bottlenecks, and network latency issues. - Work closely with DevOps and development teams for root cause analysis. - AI-Driven Testing & Optimization - Leverage AI tools for: - Test data generation - Test script optimization - Log analysis and anomaly detection - Predictive performance trend analysis - Experience with AI-assisted coding/testing tools such as: - ChatGPT - GitHub Copilot Qualifications - 6–10+ years of experience in performance testing. - Strong experience in API performance testing in microservices architecture. - Hands-on experience in UI performance analysis and optimization. - Experience integrating performance tests into CI/CD pipelines. - Strong understanding of HTTP protocol, caching, CDN. - Experience with cloud platforms (AWS/Azure/GCP/OCI) is preferred. - Strong analytical and troubleshooting skills. - Knowledge of scripting (Java, JavaScript, Python, or similar). Soft Skills - Excellent communication and collaboration skills. - Ability to manage multiple tasks/projects concurrently. - Strong interpersonal skills for working in a team-oriented environment. - Excellent problem-solving skills and attention to detail. - Ability to work independently and as part of a team in a fast-paced environment. - Proactive and self-motivated with a strong desire for continuous learning and improvement.

United States
Job Closed
Cro Metrics logo

Web Quality Assurance Analyst

Cro Metrics

We help companies grow through strategic, data-driven experimentation.

QA Engineer94 days ago
OtherRemoteTeam 51-200H1B No Sponsor

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more. Role Description As a Web Quality Assurance Analyst for Cro Metrics, you will serve as a partner to a team of engineers, program managers, and growth strategists. You’ll be the last line of defense before an A/B test is launched on the client’s website for the world to experience. This is not a “check‑the‑box” QA tester role — you’ll need to be comfortable analyzing many different websites across a wide range of complexities, navigating the technical ins and outs of various web experimentation tools, and ensuring all issues are addressed before your final stamp of approval. This Web QA position is also a launchpad into other paths at Cro Metrics, including Program Management and Growth Strategy. We invest heavily in teammates who show a strong desire to learn, stretch, and grow—pairing you with mentors and dedicating regular professional‑development time so you can chart your next step inside the organization. Key Responsibilities - Partnering closely with the Engineering and Account teams to interpret needs, implement A/B tests, and deliver timely results to clients. - Manually performing cross browser, functional, visual, and smoke testing daily across a wide variety of websites, and diving deep into each A/B test to ensure reliable functionality of the web experimentation tools. - Perform QA execution on all low or medium levels of complexity more independently with little to no guidance. - Perform QA execution on higher levels of complexity, escalations, and troubleshooting with guidance from more senior team members. - Ensuring all A/B test issues are identified, understood, and resolved or addressed with the team through regular written check ins, video chats, and retrospectives. - Documenting learnings and observations regularly to help disseminate information the team needs to succeed. - Continuously expanding your experimentation knowledge through self‑study, mentorship, and cross‑functional collaboration. Qualifications - Growth mindset — You’re motivated by curiosity, eager to acquire new skills, and view this role as the first step toward a longer career here (think Program Management or Growth Strategy). - Empathy — You put yourself in the end user’s shoes, communicate with kindness, and collaborate effectively across disciplines. - 1+ year quality assurance experience in a professional environment focused on websites or web applications or 2+ years customer service experience or other experience in a high communication role. - Technically-minded — Although no prior experience in experimentation is required, the position can get fairly technical when it comes to the web functionality, experimentation, and tools, resembling a Tier 2 IT Support role. - Critical thinking — You can analyze a problem, ask probing questions, and iterate on your own processes. - Attention to detail — You have high attention to visual detail, can communicate clear and specific step by step instructions, and leave no stone unturned. - High communication — You enjoy writing (a lot), collaborative problem solving, providing clear plans of action, and jumping on a video call. Nice to haves - Experience with Jira, Slack, or Zoom. - Experience working fully remotely. - Technical focus in schooling (e.g., Code School, CS Degree). - Knowledge of User Experience concepts. - Knowledge of Agile concepts. Benefits - 401(k) with 100% match up to 4% - Medical, dental, and vision insurance with 90%+ employer contribution - HSA, HRA, FSA, wellness and pet care benefits - 4 weeks PTO (5 weeks after 3 years) + paid bereavement and parental leave - Professional development support with customized career planning - Laptop and gear of your choice - Fully remote work culture with welcome packages, surprises, and an annual retreat - Access to group performance coaching sessions Compensation Range 45-65K + Benefits

United States
Job Closed
Impiricus logo

QA Analyst

Impiricus

The future of HCP-Pharma connectivity. Impiricus is the HCP-preferred platform to engage with Pharma.

QA Engineer94 days ago
OtherRemoteTeam 11-50Since 2020H1B No Sponsor

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more. Role Description As a QA Analyst at Impiricus, you will play a key role in ensuring the quality, reliability, and usability of our digital products. You will work closely with product managers, developers, and scrum teams to analyze requirements, design test strategies, and execute manual and exploratory testing to validate functionality and user experience. In this role, you will focus on identifying defects, validating product requirements, and ensuring that software solutions meet both business needs and regulatory standards. Your work will contribute directly to delivering high-quality healthcare technology that supports physicians and pharmaceutical partners. Duties & Responsibilities - Testing and Quality Assurance - Develop and execute manual test cases across web, backend, and mobile applications. - Review product requirements and user stories to ensure testability and completeness. - Create and maintain test plans, test cases, and testing documentation. - Perform functional, regression, smoke, and exploratory testing to ensure software quality. - Conduct API validation testing using appropriate tools to verify system integrations. - Identify, document, and track software defects through resolution while collaborating with developers and product managers. - Validate bug fixes and ensure proper implementation before release. - Process Improvement and Collaboration - Work closely with product managers and development teams to clarify requirements and acceptance criteria. - Participate in Agile ceremonies, including sprint planning, daily stand-ups, and retrospectives. - Provide feedback on product usability, performance, and potential risk areas. - Support the QA team in identifying opportunities to improve testing processes and coverage. - Assist in maintaining testing standards and QA best practices across projects. - Compliance and Documentation - Ensure software meets relevant healthcare compliance requirements, including HIPAA and other regulatory standards where applicable. - Maintain detailed documentation of test cases, test results, and defect reports. - Provide status updates on testing progress, risks, and quality metrics to stakeholders. Qualifications - Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience). - 2+ years of experience in software QA or quality assurance analysis, preferably in healthcare or health tech environments. - Strong understanding of software development lifecycle (SDLC) and Agile methodologies. - Experience with manual testing, test planning, and defect tracking tools. - Familiarity with API testing tools (such as Postman or similar). - Basic SQL knowledge for validating database data and troubleshooting issues. - Exposure to CI/CD environments. - Knowledge of automation testing is helpful. - Knowledge of healthcare regulations (HIPAA, FDA, GDPR) is preferred. - Strong analytical, problem-solving, and communication skills. Compensation and Benefits - The base salary range for this role is $85,000-$95,000. - Healthcare: Medical, dental, and vision coverage for you and your dependents + on-demand healthcare concierge. - HSA, FSA & DCFSA: Pre-tax savings options for healthcare and dependent care, with monthly employer contributions to HSA (if enrolled in a high-deductible plan). - Coverage & Protection: 100% paid short- and long-term disability, plus life and AD&D insurance. - Flexible Time Off: Take the time you need with a flexible vacation policy — recharge your batteries your way. - Parental Leave: Paid parental leave to spend time with your newborn, adopted, or foster child (available after 6 months). - Your Work, Your Way: If you’re close to an office, we encourage spending some time in-person to collaborate and connect. If not, remote is always an option. - Home Office Setup: We’ll ship you the gear you need to create a comfortable workspace at home. - 401(k): Save for your future with tax advantages (and company match!).

United States + 1 moreAll locations: United States | Canada
Job Closed