Senior QA Engineer

Location

United States

Posted

39 days ago

Salary

$120K - $140K / year

Seniority

Senior

Job Description

Senior QA Engineer

Ellipsis Health

Role Description As a Forward Deployed QA Engineer, you will occupy a critical, high-impact role dedicated to ensuring the reliability, stability, and quality of our core conversational AI product, Sage, across diverse client workflows. This role bridges the gap between Quality Assurance, AI Engineering, and Production Operations. You will focus heavily on automated testing, prompt engineering validation, and rapid root cause analysis (RCA) of Large Language Model (LLM)-driven behaviors in fast-paced, real-world deployments. - Workflow Mapping & Test Case Generation: Deeply analyze assigned client workflows to design robust, comprehensive positive and negative test cases that safeguard system stability. - AI-Driven Test Automation: Build and execute automated test scenarios by configuring shadow agents. - Prompt Evaluation & Optimization: Apply a strong understanding of prompt awareness to draft, refine, and evaluate prompts used within the testing framework to accurately simulate user behaviors and edge cases. - End-to-End Testing Execution: Strategically deploy specific testing methodologies including Sanity, Smoke, Regression, and Functional testing - determining the exact environment (staging, pre-production, production) and timing for each execution. - Deployment Cadence & Cross-Functional Collaboration: Partner closely with engineering teams during release cycles to proactively identify, triage, and unblock technical roadblocks, ensuring the product is continuously deployment-ready. - Daily LLM Defect RCA: Perform rigorous, daily root cause analysis on LLM-specific failures inherent to generative AI, including hallucinations, high latency, and logic deviations. - Live Production Call Debugging: Investigate live customer calls and production incidents in real time to unblock critical production use cases. - Audio & Transcription Validation: Query and analyze historical call transcripts, system behaviors, and audio data pipelines to pinpoint where a conversational workflow broke down. - Speech-to-Speech (S2S) Pipeline Monitoring: Monitor and evaluate the end-to-end voice AI pipeline, analyzing Automatic Speech Recognition (ASR) accuracy, managing audio-to-text latency issues, and understanding general Speech-to-Speech mechanics alongside the stability of the core Knowledge Base feeding the AI. - Advanced Evaluation Frameworks: Maintain a strong conceptual understanding of advanced LLM evaluation paradigms and tools such as LLM-as-a-judge to remain aware of how AI response quality and accuracy are programmatically graded at scale. - Telephony & Call Flow Awareness: Possess a foundational understanding of real-world call management and telephony routing concepts, including how the system is expected to navigate warm transfers, blind transfers, and voicemail detection workflows. Qualifications - Experience in QA Engineering: Strong background in software quality assurance, with a proven track record of designing, executing, and managing end-to-end test strategies (Smoke, Sanity, Regression, Functional). - LLM & Generative AI Expertise: Hands-on experience or deep technical familiarity with troubleshooting LLM behaviors, diagnosing hallucinations, managing latency, and using LLM call-tracing tools. - Technical & Logging Proficiency: Ability to comfortably write SQL queries (specifically PostgreSQL) to pull data logs and navigate cloud infrastructure logs (such as GCP) to perform rapid root-cause analysis. - Voice & Conversational AI Domain Knowledge: Foundational understanding of Speech-to-Speech pipelines, including Automatic Speech Recognition (ASR), audio-to-text workflows, and core knowledge base integrations. - Telephony Foundations: Basic familiarity with enterprise telephony routing, call management mechanics (warm/blind transfers), and voicemail detection systems. - Client-Facing Capability: Strong communication skills and the professional agility required to manage UAT timelines, coordinate with client stakeholders, and support rapid production deployments. Benefits - Competitive salary and benefits. - 401(k) matching. - Health, vision, and dental insurance. - Very flexible paid time off. - Typical salary range for this role is $120,000 to $140,000 USD, depending on skills, qualifications, and relevant experience. Background Checks As a health technology company, we reserve the right to run background checks on candidates to whom we extend offers, in compliance with applicable laws. We evaluate candidates holistically and comply with all “ban the box” regulations. Assistance If you have a disability or require accommodations during the application or recruitment process, please contact careers@ellipsishealth.com .

Related Categories

Related Job Pages

More QA Engineer Jobs

QA Engineer39 days ago
Part TimeRemoteTeam 201-500Since 1996H1B No Sponsor

• PFMEA and PPAP reviews • Assess completeness and technical accuracy of: • Failure modes and **failure mechanisms** • Cause-effect relationships tied to process steps • Evaluate: • Severity, Occurrence, Detection rankings • Risk prioritization methodology (RPN or equivalent) • Challenge robustness of: • Preventive vs. detection controls • Verify alignment with: • Actual manufacturing process flow • Known risks identified during assessments (RCR, commodity review, FAI) • Ensure full linkage between PFMEA and Control Plan • Evaluate: • Control methods (error-proofing, SPC, inspection) • Reaction plans for out-of-spec (OOS) conditions • Confirm: • Coverage of critical characteristics • Adequacy of inspection frequency and methods • Review statistical capability studies (Cp, Cpk) • Validate: • Data integrity and collection methodology • Sample size and statistical relevance • Process stability (variation analysis) • Confirm readiness against thresholds (e.g., Cpk ≥ 1.33) • Develop structured engineering review reports including: • Gap assessment (Critical / Major / Minor) • Risk implications • Recommended corrective actions • Provide clear **DTS readiness recommendation**

California
$70 / hour
Job Closed
Full TimeRemoteTeam 501-1,000Since 1997H1B No Sponsor

• Analysis of project requirements; • Drawing up and maintaining checklists, test cases; • Testing: functional, integration, regression; • Analysis of the conducted testing and creation of Bug reports; • Preparation of test reports, test plans (result, test coverage, etc.).

Serbia
Job Closed
Pfizer logo

QA Systems Associate ( 2 year fixed term)

Pfizer

Pfizer is a leading biopharmaceutical company headquartered in New York, New York. Over the years, Pfizer has received numerous awards and accolades for its bus

QA Engineer39 days ago

Use Your Power for Purpose Everything we do every day is driven by an unwavering commitment to delivering safe and effective products to patients. Our quality culture, rooted in science and risk-based compliance, is both flexible and innovative, always putting the customer first. Whether your role involves development, maintenance, compliance, or analysis through research programs, your contributions will have a direct impact on patient care. Your focus and commitment will be essential in achieving project tasks and goals, fostering a collaborative environment for your colleagues. Your hard work and dedication will help Pfizer reach new milestones and make a significant impact on patients worldwide. What You Will Achieve The primary purposes and key accountabilities of the QA Systems Associate at the PGS Melbourne site are: - The administration of Change Control, Risk Management, and Documentation systems, and in particular: - The evaluation of the impact and risk presented by proposed changes to the Pharmaceutical Quality System - The review and approval of GxP documents, including procedures, operating instructions, master batch records, protocols, reports and logbooks - Support computerized systems deployment and maintenance as required - The Quality support of changes, ranging from major site projects to relatively simple initiatives and adoption of global procedures. - The preparation of Product Quality Review reports on the designated topic within the responsibility of Quality Systems - Approving changes to product artwork, procedures and material master. Where necessary, support quality investigations associated with the design or administration of Quality Systems The essential responsibilities of the Quality Systems Associate are: - To assure that company practices align with the Pfizer requirements and the codified rules adopted by the Australian Therapeutic Goods Administration (TGA), United States Food and Drug Administration (FDA), Medicines and Healthcare products Regulatory Agency (MHRA), European Medicines Agency (EMA), Health Canada and other regulators - Collaborate with colleagues to enable efficient progression of tasks and seek advice from management as needed to ensure risks are understood and decisions well-informed - The timely completion of assigned work to support business needs - Participate in tiered accountability huddles for the assignment of work and the determination of priorities - Participate in the site's continuous improvement culture - Adopt a mindset of continuous improvement initiatives in day to day work Conduct self in accordance with Pfizer's code of conduct and company policies Here Is What You Need (Minimum Requirements) - Minimum Bachelor's Degree in an Engineering, Scientific or related discipline. - Self motivated with excellent verbal, written, presentation and interpersonal communication skills. Bonus Points If You Have (Preferred Requirements): - Knowledge and understanding of the technical regulatory and commercial aspects of pharmaceutical products - Proven ability to plan, implement, and achieve goals and objectives. - Demonstrable knowledge of current Good Manufacturing Practice (cGMP) and regulatory requirements for Australia, Europe, and the USA. Work Location Assignment: On Premise At Pfizer we care about our colleagues' wellbeing and offer a range of great benefits for them, including: - Paid parental leave - Access to Health & Wellness apps - Career Growth Experiences program - Recognition & rewards program - Paid volunteer days - Life Insurance Benefits - Pfizer Learning Academy access to top content providers - Access to flu vaccines & skin checks - Options to purchase additional leave - Salary packaging & novated lease options *Benefits listed may vary depending on your position and location and may be subject to change. Pfizer Australia and New Zealand's diverse workforce represents the patients we serve and the communities in which we operate. With a focus on Diversity, Equity & Inclusion (DE&I), Pfizer ANZ ensures our people are heard and cared for. We offer engagement opportunities in both Colleagues Resource Groups (CRGs) and workstreams across: Culture and Tradition, Reconciliation @ Pfizer (RAP), Gender Equity (GEN), Disability and through the Out Pfizer Employee Network (OPEN) for our LGBT + colleagues and allies. If you require reasonable adjustment during your application, please reach out to the Talent Acquisition Manager. To learn more about acceptable and prohibited uses of AI during the recruitment process, please review our candidate AI-use guidelines available on Pfizer Careers . Quality Assurance and Control

Australia
Job Closed
Verity Group logo

Senior QA

Verity Group

Somos Humanos. Somos Digitais. Somos Verity!

QA Engineer39 days ago
Full TimeRemoteTeam 51-200Since 2010H1B No Sponsor

• Verity is seeking new talent! • The QA Analyst will be responsible for ensuring the quality of the solutions developed by the squad, working from requirements refinement through validation of releases in production. • Play an important role in preventing defects, structuring test scenarios, and continuously improving quality processes. • Actively participate in agile ceremonies, refinement sessions, and technical discussions, contributing to the definition of acceptance criteria. • Analyze functional and technical requirements, identifying risks, impacts, and critical scenarios for validation. • Plan, design, and execute functional, regression, integration, and exploratory test cases. • Validate frontend and backend applications and integrations between systems. • Perform API and service testing, considering REST, SOAP, queues, and other architecture components. • Log, document, and track defects, supporting the team in root cause analysis and prioritization of fixes. • Support log analysis and technical evidence collection to identify failures and inconsistencies. • Participate in release acceptance, ensuring adherence to acceptance criteria and the quality standards defined by Porto. • Compile test evidence and help maintain documentation of validated scenarios. • Contribute to the evolution of the squad’s test strategy by proposing improvements to processes, tools, and quality metrics. • Support the implementation and evolution of automated tests where applicable. • Work closely with Developers, Technical Leads, Functional Analysts, and Product Owners to ensure reliable, maintainable deliveries.

Brazil
Job Closed