Job Closed
This listing is no longer active.
Be.Change Consulting Impulsamos tu Transformación Digital. Potenciamos tu empresa con innovación tecnológica y consultoría estratégica. Desde desarrollo de software e inteligencia artificial hasta transformación digital y talento especializado, te ayudamos a crecer en un mundo digital.
AI QA Engineer
Location
Colombia
Posted
20 days ago
Salary
0
Seniority
Mid Level
Job Description
AI QA Engineer
Be.change Consulting
Role Description Buscamos un(a) AI QA Engineer apasionado(a) por la calidad de sistemas de inteligencia artificial generativa, con experiencia en evaluación de modelos LLM, automatización de pruebas y métricas de desempeño para agentes conversacionales. Será responsable de garantizar la confiabilidad, precisión, seguridad y estabilidad de agentes de IA en producción, implementando frameworks de evaluación automatizados, métricas auditables y procesos de validación continua. - Diseñar, validar y mejorar frameworks de evaluación para agentes de IA. - Implementar suites de pruebas automatizadas y regresión para modelos generativos. - Definir y monitorear métricas de calidad relacionadas con: Relevancia, Fidelidad, Coherencia, Precisión y Hallucinations. - Construir sistemas de evaluación tipo “LLM-as-a-Judge”. - Establecer benchmarks de desempeño para nuevos modelos y agentes existentes. - Validar actualizaciones de prompts, modelos y pipelines RAG. - Colaborar con equipos de IA y desarrollo para definir criterios de aceptación (pass/fail). - Analizar resultados de evaluación y proponer mejoras continuas. - Generar reportes métricos y trazabilidad sobre la calidad de los agentes. Qualifications - Mínimo 3 años de experiencia en QA automation, Data/AI Quality o evaluación de sistemas de IA. - Experiencia avanzada en Python. - Experiencia trabajando con frameworks de evaluación de IA como: RAGAS, DeepEval, Vertex Gen AI Evaluation Service. - Experiencia evaluando sistemas RAG y modelos LLM. - Capacidad para diseñar sistemas “LLM-as-a-Judge”. - Experiencia en automatización de pruebas y validaciones. - Conocimiento en: Evaluación de prompts, Calidad de respuestas, Benchmarking de modelos y Testing de IA generativa. - Conocimiento de métricas de: Groundedness, Faithfulness, Context relevance, Answer relevance. - Experiencia trabajando con sistemas no deterministas. Requirements - Deseable: Experiencia en plataformas de IA conversacional. - Conocimiento en pipelines RAG. - Experiencia con APIs de modelos generativos. - Manejo de herramientas de observabilidad y monitoreo. - Conocimiento en MLOps o LLMOps. - Experiencia en ambientes cloud (GCP, AWS o Azure). Benefits - Modalidad: 100% Remoto. - Salario: A convenir. - Excelente ambiente laboral, equipo joven, dinámico y comprometido, oportunidad de crecimiento y participación en proyectos innovadores. - Si quieres transformar información en impacto y crecer junto a un equipo que impulsa la innovación, este es tu lugar. ¡Postúlate! 🌐 📊 Company Description Be.Change Consulting Impulsamos tu Transformación Digital. Potenciamos tu empresa con innovación tecnológica y consultoría estratégica. Desde desarrollo de software e inteligencia artificial hasta transformación digital y talento especializado, te ayudamos a crecer en un mundo digital.
Related Guides
Related Categories
Related Job Pages
More QA Engineer Jobs
Lead/Senior QA Engineer – AI
CodiLimeA strategic partner for technology-driven companies | Network engineering | Software engineering
• Define and Drive QA Strategy based on AI • Establish testing processes, methodologies, and best practices from scratch, ensuring they align with our product roadmap and business priorities. • Secure our codebase by mixing manual exploratory testing with robust automation (primarily using Playwright). • Validate that Backend, Frontend (Nuxt 4/Vue.js), and Infrastructure logic communicate correctly through rigorous integration testing. • Integrate QA into our continuous delivery pipelines and collaborate on Infrastructure as Code (Terraform/Ansible) to enable reliable releases. • Act as the gatekeeper for releases, triaging bugs during critical incidents, and providing monthly/quarterly quality reporting to the board. • Lead the effort to recruit, mentor, and scale the QA department.
QA Coordinator
Scratch FinancialScratch Financial is the world's simplest patient financing solution.
Company Description NBCUniversal is one of the world's leading media and entertainment companies. We create world-class content, which we distribute across our portfolio of film, television, and streaming, and bring to life through our global theme park destinations, consumer products, and experiences. We own and operate leading entertainment and news brands, including NBC, NBC News, NBC Sports, Telemundo, NBC Local Stations, Bravo, and Peacock, our premium ad-supported streaming service. We produce and distribute premier filmed entertainment and programming through our powerhouse film and television studios, including Universal Pictures, DreamWorks Animation, and Focus Features, and the four global television studios under the Universal Studio Group banner, and operate industry-leading theme parks and experiences around the world through Universal Destinations & Experiences, including Universal Orlando Resort, home to Universal Epic Universe, and Universal Studios Hollywood. NBCUniversal is a subsidiary of Comcast Corporation. Visit www.nbcuniversal.com for more information. Our impact is rooted in improving the communities where our employees, customers, and audiences live and work. We have a rich tradition of giving back and ensuring our employees have the opportunity to serve their communities. We champion an inclusive culture and strive to attract and develop a talented workforce to create and deliver a wide range of content reflecting our world. Job Description The QA Coordinator is responsible for coordinating the day-to-day QA operations for Universal+ across mobile, CTV, and web. Partner with Product, Program/Project Managers, Vendors, and Distribution Partners to plan test cycles, manage certification readiness, and keep documentation, environments, and defect workflows tight-so releases ship on time with the right coverage. As the QA Coordinator you will review and analyze system requirements, while keeping a user's perspective in mind; You will be responsible for translating these requirements into test cases, identifying defects, and driving them to resolution. Ideally, the candidate will possess a drive for thinking creatively to resolve complex problems and enjoy contributing to solutions. This role will be involved in monitoring all platforms and devices to ensure the product is being delivered as intended. Responsibilities - Plan & Schedule: Own the QA calendar-smoke, regression, certification, and UAT cycles; align test milestones to the release plan. - Coverage Management: Maintain the requirements-to-test traceability matrix; keep regression suites current and risk-based per release. - Device & Environment Readiness: Manage the device/OS matrix and test accounts; coordinate build intake, configuration flags, test data, and environment availability. - Partner Certifications: Prepare evidence packages (checklists, screenshots, logs, videos), track certification gates, and coordinate re-submissions with partners. - Defect Workflow: Drive daily triage with clear reproduction steps, severity/priority hygiene, ownership, and follow-through to fix verification and closure. - Cross-Functional Coordination: Sync with PMs (Commerce, Integrations, Data) and the Product Owner on auth/SSO, IAP, promotions, metadata/search, analytics tagging, and SSAI-related test needs. - Documentation: Maintain QA runbooks, test plans, regression lists, and release checklists; publish QA sign-off notes for Go/No-Go. - Incidents & Postmortems: Support fast diagnosis with repro steps, logs, and impact assessment; capture QA actions in retros. - Quality Signals: Track and share lightweight QA status (coverage executed, open blockers, high-severity defects, readiness risks). Qualifications Basic Requirements: - 3-5+ years in QA coordination or QA project support for consumer apps (multi-platform experience: mobile, CTV, web) - Solid understanding of software testing methods and device/app certification processes. - Hands-on experience with test case management and defect tracking systems; able to write clear test cases and bug reports. - Excellent organization, follow-through, and written/spoken communication. - Bilingual English/Spanish preferred. Additional Requirements: - Interested candidate must submit a resume/CV through www.nbcunicareers.com to be considered - Must be 18 years or older - Must be willing to submit to a background investigation - Must be covered by Solutions, NBCU's Alternative Dispute Resolution Program Desired Characteristics: - Exposure to automation pipelines (can schedule runs and interpret results with engineers). - Comfortable reviewing API responses and basic logs to aid triage. - Awareness of accessibility, localization, and privacy-by-design testing considerations. - Experience with test management and defect tracking tools. - Novice to intermediate level of experience with automation or performance testing tools such as Selenium and WebDriverIO. - Postman or other API testing experience. - Familiarity with JIRA and the Atlassian suite of tools. - Understanding of Software Development Life Cycle processes and the QA lifecycle and methodology. - Knowledge of the OTT ecosystem. - Innovative thinker with acute awareness of managing competing initiatives. - Strong analytical and interpersonal skills with ability to work independently and collaboratively. - Ability to set and self-manage goals, deadlines and multiple assignments concurrently in a project-based environment. - Familiarity with emerging video and mobile product landscape. - Ability to thrive in a continuously changing environment. Additional Information As part of our selection process, external candidates may be required to attend an in-person interview with an NBCUniversal employee at one of our locations prior to a hiring decision. NBCUniversal's policy is to provide equal employment opportunities to all applicants and employees without regard to race, color, religion, creed, gender, gender identity or expression, age, national origin or ancestry, citizenship, disability, sexual orientation, marital status, pregnancy, veteran status, membership in the uniformed services, genetic information, or any other basis protected by applicable law. If you are a qualified individual with a disability or a disabled veteran and require support throughout the application and/or recruitment process as a result of your disability, you have the right to request a reasonable accommodation. You can submit your request to AccessibilitySupport@nbcuni.com.
ISTQB Certified QA Tester (Mobile Apps)
Louco Event Media GmbHLouco is the game changer in live entertainment. The first platform that truly understands what users want and delivers hyper-personalized event experiences.
Role Description Louco is building a new generation event discovery platform that connects users, creators, and the entertainment industry. Our goal is not just to build another event app – we want to rethink how people find & attend events. We are looking for an experienced Backend Developer (PHP / Laravel) who wants to build a real product and work closely with our founders and tech lead. Responsibilities - Perform manual QA testing for mobile applications - Validate new releases and features before production deployment - Identify, document, and track bugs/issues - Collaborate closely with developers and designers - Test onboarding, guest mode, event discovery flows, authentication, and performance - Support product quality improvements and release stability Qualifications - 4+ years of experience in Mobile App QA Testing - Strong experience with Manual Testing for iOS and Android apps - Experience testing marketplace, discovery, booking, or event platforms is highly preferred - Good understanding of user flows, onboarding, authentication, and mobile UX behavior - Ability to create and maintain detailed test cases and bug reports - Experience with regression testing, smoke testing, and release validation - Familiarity with tools such as Jira, TestFlight, Firebase, Postman, or similar QA tools - Attention to detail and strong communication skills - Fluent English communication Nice to Have - Experience with Automated Testing (Appium, Detox, Selenium, Cypress, etc.) - Experience with CI/CD QA workflows - Experience working in agile startup/product teams - Understanding of analytics and user-retention focused products Benefits - Work directly with founders - Real product ownership - Fast-paced startup environment - Possibility for long-term collaboration - Performance-based bonuses Company Description Louco is the game changer in live entertainment. The first platform that truly understands what users want and delivers hyper-personalized event experiences.
Quality Surveyor - Intellectual and Developmental Disabilities
QlarantBest People. Best Solutions. Best Results.®
Title: Quality Surveyor – Intellectual and Developmental Disabilities Location: LA Home Based, Remote, LA, US Job Description: Qlarant is a not-for-profit corporation that partners with public and private sectors to create high quality, safe, and efficient delivery of health care and human services programs. We have multiple lines of business including population health, utilization review, managed care organization quality review, and quality assurance for programs serving individuals with developmental disabilities. Qlarant is also a national leader in fighting fraud, waste and abuse for large organizations across the country. In addition, our Foundation provides grant opportunities to those with programs for underserved communities. Best People, Best Solutions, Best Results We are seeking dedicated individuals who live in the state of Louisiana, to join our team as National Core Indicators (NCI) Surveyors. In this role, you will conduct in-person and/or virtual surveys with individuals with developmental or intellectual disabilities who receive services, their families, or guardians. Your primary responsibility will be to gather data on the quality of services provided by the state of Louisiana. Survey topics include employment, rights, service planning, community inclusion, choice, and health and safety. If you are passionate about working with people, have experience in the field of intellectual and developmental disabilities and enjoy helping people make an impact on the quality of their services, we encourage you to apply! Key Responsibilities: - Conducting Surveys: Administer surveys to individuals (sometimes with support from someone who knows them best) receiving services either in-person or virtually. - Data Collection and Entry: Ensure accurate and timely documentation of all survey data in the designated online data entry system. - Communication and Engagement: Contact providers, individuals, families and legal guardians to explain the purpose and importance of the NCI survey and answer their questions in a clear and professional manner. - Maintaining Confidentiality: Protect the privacy and confidentiality of all survey participants. - Professionalism: Conduct interviews with respect and sensitivity, maintaining a professional demeanor throughout the process. - Understanding Survey Techniques: Demonstrate proficiency in surveying techniques, including using a conversational tone and rephrasing questions as needed. - Following Protocols: Adhere to all established protocols and procedures for survey administration and data management. - Training and Reliability: Complete 4 required training courses at the time of onboarding (3 are online and self-paced, 1 is live virtual training), maintain inter-rater reliability, and participate in annual refresher training as needed. Training time is compensated. - Surveyors are provided with necessary equipment (laptop and mobile hotspot) to ensure surveys are conducted efficiently and confidentially. Time & Travel Commitment: - Hours: 10-20 total hours per week. Daytime availability between 9am-3pm is preferred at least 1-2 days per week. - Surveys: Complete a minimum of 8 surveys per month (each survey typically takes 1-2 hours to complete) - Travel: Travel is required for all in-person surveys, typically ranging from 10-40 miles each way. Travel is included in the compensation for the survey completed. Education & Experience: - Bachelor’s Degree + at least 1 year of experience working with individuals who have developmental / intellectual disabilities OR - Associates Degree + at least 2 years of experience working with individuals who have developmental / intellectual disabilities. Qualifications: - Communication Skills: Strong communication and interpersonal skills, with the ability to interact respectfully and effectively with individuals with disabilities. Ability to build rapport and establish a positive connection with survey participants. - Reliability: Dependable transportation for all in-person surveys. - Computer Literacy: Proficiency in using computers and navigating online data systems with specific experience in Outlook and Microsoft Excel Spreadsheets. - Attention to Detail: Meticulous attention to detail for accurate data collection. - Flexibility: Ability to work a flexible schedule, including some evenings and weekends, as needed. - Professionalism: Ability to interact with team members, supervisors, individuals, families and providers in a respectful and sensitive manner. - Experience: Experience with survey administration and data input is preferred. - Bi-lingual: Preferred but not required. Qlarant is an Equal Opportunity Employer of Minorities, Females, Protected Veterans, and Individuals with Disabilities. Qlarant is a drug-free workplace. All offers of employment are contingent upon successful completion of pre-employment background and drug screens.



