Senior / Principal QA

QA Automation EngineerQA Automation EngineerFull TimeRemoteLeadTeam 51-200

Location

Europe

Posted

18 hours ago

Salary

0

Seniority

Lead

No structured requirement data.

Job Description

Senior / Principal QA

Acclaim

Role Description We're growing our team and looking for a Senior/Principal QA Engineer to own quality across our AI voice assistant: from prompt and agent behavior testing to mobile, TV, and backend. Qualifications - 5+ years of manual QA experience. - High level of ownership and self-direction. - Experience testing AI agents/assistants. - Experience with AI observability/tracing tools. - Experience with mobile and/or TV-app testing. - Experience testing backends and integrations. - Solid QA fundamentals: test design, documentation, defect handling. - Hands-on experience using AI tools in QA processes. Requirements - Experience testing speech technologies (ASR/TTS) and working with audio. - Building and running AI-assistant quality evaluations. - CI/CD/deployment/test environments experience. - Device testing experience/ADB. - Willingness to move into test automation over time. Responsibilities - Assessing assistant quality: prompt/instruction following, compliance with requirements. - Testing the assistant "under the hood" via observability/tracing tools. - End-to-end manual testing of the assistant in mobile app and devices (TV, speaker). - Testing voice interaction with the assistant, recognition (ASR) and synthesis (TTS) quality. - Testing the assistant's product scenarios. - Testing the platform/backend part the assistant runs on. - Verifying the request's end-to-end path through components. - Maintaining test cases and bug reports. - Close collaboration with the team: TPM, Prompt Engineer, DEV, DevOps, ML. - First-line QA of critical scenarios ahead of demos. Benefits - Award-winning AI products for tech corporations. - Cutting-edge tech stack: Speech Technologies, NLP, Generative AI. - High engineering bar and real ownership. - Fast career progression. - Startup pace with enterprise stability. - Fully remote across Europe. - 21 vacation days + public holidays + 5 sick days. - Private English lessons via Preply.

Related Categories

Related Job Pages

More QA Automation Engineer Jobs

Full TimeRemoteTeam 51-200Since 2010H1B No Sponsor

Role Description We are currently seeking a Quality Assurance Engineer for our client in the Lucerne area. In this role, you will be responsible for the development, implementation, and verification of Quality Assurance requirements and act as the main interface between Process Engineering and Operations for all quality and safety topics. You will ensure the successful execution of the Quality Program and compliance with Product Assurance requirements at the manufacturing site. - Develop and promote Quality Assurance requirements and a strong quality culture - Ensure robust quality processes and cost-efficient program execution in collaboration with Engineering, Operations, and Supply Chain - Act as main interface for Process Engineering and Operations on quality and safety topics - Define and implement Critical-to-Quality (CTQ) characteristics and control plans - Lead non-conformance management, root cause analyses (RCA), and failure investigations - Moderate and lead Process FMEA activities - Supervise Manufacturing, Assembly, Integration and Testing (MAIT) quality aspects - Define and monitor Measuring & Test Equipment (M&TE), Measurement System Analysis (MSA), and process capability - Perform quality inspections and manage KIP/MIP documentation - Establish and maintain process quality documentation and support process qualification - Ensure compliance with customer, regulatory, and internal requirements - Contribute to proposals (RFI/RFQ), audits, and quality organization development (training & coaching) - Support product and qualification data packages, especially for space applications Qualifications - Degree (BSc/MSc) in Engineering or a related field - Relevant experience in Quality Assurance and Continuous Improvement - Knowledge of standards such as EN9100, ECSS, NASA, and MIL - Strong technical understanding and ability to analyze complex systems - Hands-on mindset combined with solid theoretical knowledge - Experience with quality tools and methodologies (e.g., 5Why, Ishikawa/Fishbone, FMEA) - Proficiency in MS Office - Experience in the space industry is an advantage - Strong analytical, critical thinking, and interpersonal skills - Independent and team-oriented working style - Proactive, communicative, and confident personality - Passion for quality and continuous improvement - Willingness to travel (approx. 5–10%) - Excellent communication skills in English (written and spoken) - Additional languages (German, French, or Swedish) are a plus Company Description Die Coopers Group AG ist eine agile Schweizer Recruiting Agentur, die Spezialisten und Führungskräfte in den Bereichen IT, Life Sciences, Engineering und Finance vermittelt. Mit flexiblen Ansätzen bringen wir Kandidat:innen und Unternehmen zusammen, die nicht nur fachlich, sondern auch menschlich zusammenpassen.

Switzerland
Spyrosoft logo

AI Automation Engineer – Microsoft Copilot Studio

Spyrosoft

We enable our clients to thrive, thanks to a combination of technical proficiency and domain-specific knowledge.

Full TimeRemoteTeam 1,001-5,000Since 2016H1B No Sponsor

• Design, build and maintain AI-powered copilots using Microsoft Copilot Studio • Develop intelligent automation solutions that streamline business processes • Configure conversational AI experiences and generative AI capabilities • Design and implement automation workflows using Microsoft Power Automate or other automation platforms • Integrate Copilot Studio with Microsoft 365, Dynamics 365, Power Platform and enterprise applications • Develop API integrations and custom connectors • Gather business requirements and translate them into technical solutions • Troubleshoot, optimize and continuously improve AI automation solutions • Collaborate with Solution Architects, Developers and business stakeholders • Follow engineering best practices and contribute to technical improvements • For Lead-level candidates: provide technical guidance, support solution design and mentor other engineers

Poland
zł110 - zł150 / hour

Role Description At Hupo, AI quality is the product - a wrong score, hallucinated feedback, or a broken voice turn is a customer-facing defect, and in a regulated BFSI context, a compliance risk. This role owns and scales the testing of Hupo’s AI pipelines, working closely with QA, Engineering, and Product as the platform grows. It’s a hands-on, high-ownership role in a fast-moving environment - you’ll be a key part of deciding, with evidence, whether each release is safe to ship. What You'll Do - Build and maintain automated testing for AI voice and chat agents, from single conversational turns to full roleplay flows. - Test LLM output quality - correctness, consistency, structured output, language fidelity, prompt regression - using evaluation harnesses (LLM-as-judge, golden datasets, tolerance-based assertions for non-determinism). - Validate RAG pipelines: retrieval relevance, grounding/faithfulness, and answer quality. - Test voice pipelines: STT/TTS accuracy and real-time, low-latency behavior across languages. - Automate across UI/E2E, API, and AI output layers, and build CI/CD from scratch - lint, type-check, tests, evals, coverage gates, deployment checks. - Own release quality: regression strategy, catching breakages early, and clear go/no-go calls. - Cover database and load/performance testing; extend automation to desktop and mobile clients. - Take over and extend existing QA automation, and strengthen shared frameworks, tooling, and QA processes/standards. What Success Looks Like Within your first few months, you'll: - Ramp on Hupo’s existing QA automation suites and take clear ownership of them. - Extend test coverage for LLM and voice-agent behavior, including new evaluation harnesses for non-deterministic output. - Contribute CI/CD improvements - coverage gates, deployment checks - that catch regressions before they reach customers. You’ll also be expected to continuously: - Adapt and expand impact as priorities shift. - Take ownership of new problems as they emerge. Qualifications - 5+ years as a QA Automation Engineer, with a proven track record testing AI systems - not just traditional software. - Builder mindset: established testing frameworks, standards, and CI/CD-integrated automation from scratch. - Hands-on experience testing conversational AI - voice and/or chat agents - on real, shipped projects. - Strong grasp of LLMs, prompt engineering, and RAG - designing tests for nondeterministic output and evaluating retrieval/generation quality. - Understanding of voice pipelines (speech-to-text, text-to-speech) and how to test them in automation. - Hands-on with LLM evaluation/observability tooling - e.g. Langfuse, LangSmith, DeepEval, RAGAS. - Proficient in UI/E2E (e.g. Playwright) and API test automation; experience building CI/CD pipelines from scratch. - Comfort with database testing, load/performance testing, and desktop/mobile test automation. - Solid Git discipline, basic cloud knowledge, and familiarity with the agile sprint lifecycle. - Track record in fast-paced, fast-shipping environments - ramps quickly on unfamiliar systems, pragmatic about process. Nice-to-Haves - TypeScript - for reading the codebase and contributing test/automation code directly. - Voice & real-time: LiveKit / WebRTC; STT/TTS via Azure Speech, Google Cloud Speech, Deepgram, or self-hosted Whisper. - AI/ML infra & eval tooling: GPU inference (vLLM / TGI / Triton), Azure OpenAI, golden datasets, HITL pipelines, Braintrust, Promptfoo. - QA & cloud tooling: Cypress, Postman, multi-cloud (AWS/Azure/GCP), Infrastructure-asCode (Terraform/Pulumi/CloudFormation), Docker/Kubernetes. - Also useful: product analytics familiarity (e.g. PostHog), multilingual/localized testing experience, and production monitoring/error-tracking tools.

Asia Pacific
Weekday (YC W21) logo

Agentic QA Engineer

Weekday (YC W21)

We are a Y-Combinator-backed startup building your AI-powered Recruiter Agent

Full TimeRemoteTeam 11-50Since 2021H1B No Sponsor

Role Description We are looking for an innovative Agentic QA Engineer to define and build the next generation of quality assurance practices for AI-native platforms. This is far more than a traditional QA role—you will combine software testing, automation engineering, and AI system validation to ensure platform reliability, scalability, and intelligent decision-making. You'll be responsible for validating the platform's behavior, including its AI-driven orchestration, routing, and multi-agent workflows. Rather than evaluating customer-built AI agents, your focus will be on ensuring the platform consistently makes reliable, secure, and efficient decisions under real-world conditions. Key Responsibilities - Platform Functional Testing - Design and execute end-to-end functional tests covering agent onboarding, deployment, orchestration, and request routing. - Validate platform behavior across multiple supported agent frameworks and deployment scenarios. - Test failure handling, recovery mechanisms, and edge cases throughout the platform lifecycle. - QA Automation & Reliability - Build and maintain automated regression test suites integrated into CI/CD pipelines. - Develop automated quality gates that prevent deployments when reliability metrics fall below acceptable thresholds. - Perform resilience testing by intentionally introducing failures such as service outages, database interruptions, network failures, and high traffic loads. - Define and monitor Service Level Objectives (SLOs) for platform reliability and availability. - Agentic Testing - Develop testing methodologies for AI-driven platform behavior where deterministic outputs are not possible. - Validate intelligent routing decisions across multiple AI agents. - Test multi-agent orchestration to ensure correct information flow, execution limits, cost controls, and successful task completion. - Verify platform safeguards against infinite agent loops, resource overconsumption, security risks, and cross-tenant data leakage. - Build and maintain a suite of deterministic test agents with predictable behaviors to evaluate platform decision-making accurately. - Quality Engineering - Create adversarial ("red team") testing strategies to uncover weaknesses in AI workflows and orchestration logic. - Convert execution traces into measurable platform health metrics such as routing accuracy, task success rate, execution cost, failure rates, and stuck-loop detection. - Investigate failure patterns and develop clear troubleshooting guides and operational playbooks. - Own quality assurance processes for platform releases and continuously improve testing strategies as the platform evolves. - Contribute toward developing platform testing capabilities that customers can leverage for validating their own AI agents. Qualifications - Strong experience in QA Engineering, SDET, Software Testing, or Site Reliability Engineering. - Proven expertise in building automated testing frameworks and integrating them into CI/CD pipelines. - Hands-on experience with test automation using Python or similar scripting languages. - Strong understanding of software quality principles, distributed systems, and cloud-native architectures. - Familiarity with AI/ML concepts and an understanding of how AI-powered systems behave in production environments. - Excellent analytical and problem-solving skills with an adversarial mindset for identifying edge cases and failure scenarios. - Strong communication skills with the ability to explain technical quality issues to engineering, product, and leadership teams. Preferred Qualifications - Experience working with Large Language Models (LLMs), AI agents, or autonomous AI systems. - Familiarity with agent frameworks such as LangGraph, CrewAI, AutoGen, or similar orchestration platforms. - Knowledge of distributed systems reliability and control plane architectures. - Experience with observability platforms such as OpenTelemetry, Grafana, Datadog, or similar monitoring tools. - Contributions to open-source QA, AI, or developer tooling projects. Benefits - Help define an entirely new discipline at the intersection of Quality Engineering, AI Systems, and Platform Reliability. - Build innovative testing methodologies for intelligent, AI-native platforms. - Work in a highly collaborative environment with significant ownership and technical autonomy. - Make a direct impact on the quality, scalability, and reliability of next-generation AI products. - Enjoy a flexible, remote-friendly work environment with competitive compensation and growth opportunities. Must-Have Skills - QA Automation - Agentic Testing - Test Automation - Python Good-to-Have Skills - Large Language Models (LLMs) - LangGraph - CrewAI - AutoGen - Site Reliability Engineering (SRE) - Distributed Systems - OpenTelemetry - Grafana - Datadog

India
₹2,500K - ₹4,000K / year