Role Description Buscamos un(a) AI QA Engineer apasionado(a) por la calidad de sistemas de inteligencia artificial generativa, con experiencia en evaluación de modelos LLM, automatización de pruebas y métricas de desempeño para agentes conversacionales. Será responsable de garantizar la confiabilidad, precisión, seguridad y estabilidad de agentes de IA en producción, implementando frameworks de evaluación automatizados, métricas auditables y procesos de validación continua. - Diseñar, validar y mejorar frameworks de evaluación para agentes de IA. - Implementar suites de pruebas automatizadas y regresión para modelos generativos. - Definir y monitorear métricas de calidad relacionadas con: Relevancia, Fidelidad, Coherencia, Precisión y Hallucinations. - Construir sistemas de evaluación tipo “LLM-as-a-Judge”. - Establecer benchmarks de desempeño para nuevos modelos y agentes existentes. - Validar actualizaciones de prompts, modelos y pipelines RAG. - Colaborar con equipos de IA y desarrollo para definir criterios de aceptación (pass/fail). - Analizar resultados de evaluación y proponer mejoras continuas. - Generar reportes métricos y trazabilidad sobre la calidad de los agentes. Qualifications - Mínimo 3 años de experiencia en QA automation, Data/AI Quality o evaluación de sistemas de IA. - Experiencia avanzada en Python. - Experiencia trabajando con frameworks de evaluación de IA como: RAGAS, DeepEval, Vertex Gen AI Evaluation Service. - Experiencia evaluando sistemas RAG y modelos LLM. - Capacidad para diseñar sistemas “LLM-as-a-Judge”. - Experiencia en automatización de pruebas y validaciones. - Conocimiento en: Evaluación de prompts, Calidad de respuestas, Benchmarking de modelos y Testing de IA generativa. - Conocimiento de métricas de: Groundedness, Faithfulness, Context relevance, Answer relevance. - Experiencia trabajando con sistemas no deterministas. Requirements - Deseable: Experiencia en plataformas de IA conversacional. - Conocimiento en pipelines RAG. - Experiencia con APIs de modelos generativos. - Manejo de herramientas de observabilidad y monitoreo. - Conocimiento en MLOps o LLMOps. - Experiencia en ambientes cloud (GCP, AWS o Azure). Benefits - Modalidad: 100% Remoto. - Salario: A convenir. - Excelente ambiente laboral, equipo joven, dinámico y comprometido, oportunidad de crecimiento y participación en proyectos innovadores. - Si quieres transformar información en impacto y crecer junto a un equipo que impulsa la innovación, este es tu lugar. ¡Postúlate! 🌐 📊 Company Description Be.Change Consulting Impulsamos tu Transformación Digital. Potenciamos tu empresa con innovación tecnológica y consultoría estratégica. Desde desarrollo de software e inteligencia artificial hasta transformación digital y talento especializado, te ayudamos a crecer en un mundo digital.

• Develop and enforce quality standards and goals across our software systems, services, and applications. • Design, build, and maintain an efficient automated testing environment while continuously educating the software engineering team on best practices. • Collaborate closely with Software Engineers, System Architects, Organizational Developer, UI/UX Designers, and product owners to deliver a high-quality product. • Advocate for a strong testing mindset throughout the organization to ensure quality remains a top priority. • Work with the support team to assess, evaluate, and prioritize incoming issues based on their severity and potential impact.

• Define and Drive QA Strategy based on AI • Establish testing processes, methodologies, and best practices from scratch, ensuring they align with our product roadmap and business priorities. • Secure our codebase by mixing manual exploratory testing with robust automation (primarily using Playwright). • Validate that Backend, Frontend (Nuxt 4/Vue.js), and Infrastructure logic communicate correctly through rigorous integration testing. • Integrate QA into our continuous delivery pipelines and collaborate on Infrastructure as Code (Terraform/Ansible) to enable reliable releases. • Act as the gatekeeper for releases, triaging bugs during critical incidents, and providing monthly/quarterly quality reporting to the board. • Lead the effort to recruit, mentor, and scale the QA department.

Company Description NBCUniversal is one of the world's leading media and entertainment companies. We create world-class content, which we distribute across our portfolio of film, television, and streaming, and bring to life through our global theme park destinations, consumer products, and experiences. We own and operate leading entertainment and news brands, including NBC, NBC News, NBC Sports, Telemundo, NBC Local Stations, Bravo, and Peacock, our premium ad-supported streaming service. We produce and distribute premier filmed entertainment and programming through our powerhouse film and television studios, including Universal Pictures, DreamWorks Animation, and Focus Features, and the four global television studios under the Universal Studio Group banner, and operate industry-leading theme parks and experiences around the world through Universal Destinations & Experiences, including Universal Orlando Resort, home to Universal Epic Universe, and Universal Studios Hollywood. NBCUniversal is a subsidiary of Comcast Corporation. Visit www.nbcuniversal.com for more information. Our impact is rooted in improving the communities where our employees, customers, and audiences live and work. We have a rich tradition of giving back and ensuring our employees have the opportunity to serve their communities. We champion an inclusive culture and strive to attract and develop a talented workforce to create and deliver a wide range of content reflecting our world. Job Description The QA Coordinator is responsible for coordinating the day-to-day QA operations for Universal+ across mobile, CTV, and web. Partner with Product, Program/Project Managers, Vendors, and Distribution Partners to plan test cycles, manage certification readiness, and keep documentation, environments, and defect workflows tight-so releases ship on time with the right coverage. As the QA Coordinator you will review and analyze system requirements, while keeping a user's perspective in mind; You will be responsible for translating these requirements into test cases, identifying defects, and driving them to resolution. Ideally, the candidate will possess a drive for thinking creatively to resolve complex problems and enjoy contributing to solutions. This role will be involved in monitoring all platforms and devices to ensure the product is being delivered as intended. Responsibilities - Plan & Schedule: Own the QA calendar-smoke, regression, certification, and UAT cycles; align test milestones to the release plan. - Coverage Management: Maintain the requirements-to-test traceability matrix; keep regression suites current and risk-based per release. - Device & Environment Readiness: Manage the device/OS matrix and test accounts; coordinate build intake, configuration flags, test data, and environment availability. - Partner Certifications: Prepare evidence packages (checklists, screenshots, logs, videos), track certification gates, and coordinate re-submissions with partners. - Defect Workflow: Drive daily triage with clear reproduction steps, severity/priority hygiene, ownership, and follow-through to fix verification and closure. - Cross-Functional Coordination: Sync with PMs (Commerce, Integrations, Data) and the Product Owner on auth/SSO, IAP, promotions, metadata/search, analytics tagging, and SSAI-related test needs. - Documentation: Maintain QA runbooks, test plans, regression lists, and release checklists; publish QA sign-off notes for Go/No-Go. - Incidents & Postmortems: Support fast diagnosis with repro steps, logs, and impact assessment; capture QA actions in retros. - Quality Signals: Track and share lightweight QA status (coverage executed, open blockers, high-severity defects, readiness risks). Qualifications Basic Requirements: - 3-5+ years in QA coordination or QA project support for consumer apps (multi-platform experience: mobile, CTV, web) - Solid understanding of software testing methods and device/app certification processes. - Hands-on experience with test case management and defect tracking systems; able to write clear test cases and bug reports. - Excellent organization, follow-through, and written/spoken communication. - Bilingual English/Spanish preferred. Additional Requirements: - Interested candidate must submit a resume/CV through www.nbcunicareers.com to be considered - Must be 18 years or older - Must be willing to submit to a background investigation - Must be covered by Solutions, NBCU's Alternative Dispute Resolution Program Desired Characteristics: - Exposure to automation pipelines (can schedule runs and interpret results with engineers). - Comfortable reviewing API responses and basic logs to aid triage. - Awareness of accessibility, localization, and privacy-by-design testing considerations. - Experience with test management and defect tracking tools. - Novice to intermediate level of experience with automation or performance testing tools such as Selenium and WebDriverIO. - Postman or other API testing experience. - Familiarity with JIRA and the Atlassian suite of tools. - Understanding of Software Development Life Cycle processes and the QA lifecycle and methodology. - Knowledge of the OTT ecosystem. - Innovative thinker with acute awareness of managing competing initiatives. - Strong analytical and interpersonal skills with ability to work independently and collaboratively. - Ability to set and self-manage goals, deadlines and multiple assignments concurrently in a project-based environment. - Familiarity with emerging video and mobile product landscape. - Ability to thrive in a continuously changing environment. Additional Information As part of our selection process, external candidates may be required to attend an in-person interview with an NBCUniversal employee at one of our locations prior to a hiring decision. NBCUniversal's policy is to provide equal employment opportunities to all applicants and employees without regard to race, color, religion, creed, gender, gender identity or expression, age, national origin or ancestry, citizenship, disability, sexual orientation, marital status, pregnancy, veteran status, membership in the uniformed services, genetic information, or any other basis protected by applicable law. If you are a qualified individual with a disability or a disabled veteran and require support throughout the application and/or recruitment process as a result of your disability, you have the right to request a reasonable accommodation. You can submit your request to AccessibilitySupport@nbcuni.com.