Job Closed

This listing is no longer active.

CaseWare logo
CaseWare

Founded in 1988, CaseWare is a global software provider for corporations, government agencies, and accounting firms, including individual accountants and audito

AI Test Architect

Location

Colombia

Posted

140 days ago

Salary

0

Seniority

Lead

Job Description

AI Test Architect

CaseWare

• Architect a comprehensive "Quality Intelligence" platform using generative AI to predict defect hotspots, intelligently optimize regression suites, auto-generate tests, and enable self-healing automation. • Define enterprise-wide AI-first testing strategy, including non-deterministic evaluation paradigms, continuous monitoring for drift/hallucination, and integration across the full SDLC. • Establish governance for ethical AI testing, aligning with emerging standards. • Design and implement advanced benchmarks, red teaming protocols, and adversarial testing for internal AI agents and generative features—focusing on hallucination rates, bias/fairness, prompt injection, jailbreaks, and goal misalignment. • Build evaluation pipelines with statistical rigor (e.g., multi-trial runs, LLM-as-judge, human-in-the-loop) using tools like LangFuse, LangSmith, DeepEval, RAGAS, or Arize Phoenix for metrics such as faithfulness, context precision, and safety compliance. • Partner with DevOps to embed AI-based testing into GitHub-based CI/CD pipelines (e.g., AI-generated tests, predictive flakiness detection, automated gating with quality signals). • Lead design of self-healing test frameworks (integrating AI plugins with Playwright/Cypress or similar) that adapt to UI/model changes with minimal maintenance. • Architect synthetic data generation, maintain golden data-sets and AI-powered data masking solutions to enable privacy-compliant, high-fidelity testing at scale. • Collaborate with product, data science, ML engineering, and security teams to influence AI feature design with quality guardrails from day one. • Evangelize and mentor: Upskill traditional QA engineers into AI-augmented testers through workshops, playbooks, and communities of practice. • Drive adoption of AI quality best practices organization-wide, including metrics dashboards for DORA + AI-specific indicators (e.g., hallucination rate, red team success rate, self-healing coverage).

Job Requirements

  • 8+ years in Quality Engineering/Test Architecture within cloud-native SaaS environments, with 2+ years focused on AI/ML/LLM testing and validation.
  • Deep expertise in AWS (serverless, microservices, IaC with Terraform/CloudFormation) and GitHub CI/CD ecosystems.
  • Proficiency architecting LLM-based applications and testing frameworks (LangChain/LangGraph/LangSmith strongly preferred; equivalents acceptable).
  • Mastery of modern automation (Playwright, Cypress) with hands-on experience integrating self-healing AI plugins or generative test tools.
  • Strong programming skills in JavaScript/TypeScript and/or Python; solid understanding of foundational AI concepts (transformers, embeddings, RAG, evaluation trade-offs).
  • Experience with LLM evaluation tools like Bedrock Evaluations, Prompt Management, Guardrails, DeepEval, RAGAS, Arize Phoenix, Langfuse.
  • Experience with Red teaming frameworks/tools (Cobalt Strike, Sliver, Nmap) and knowledge of adversarial testing methodologies is a bonus.
  • Proven leadership: Mentoring teams, defining standards, and driving cross-functional change in ambiguous, high-growth settings.
  • Bachelor's/Master's in Computer Science, AI/ML, or equivalent; relevant certifications a strong plus.
  • Strong English language communication and collaboration skills.

Benefits

  • Contrato a termino Indefinido with all the legal benefits
  • Prepaid Medicine
  • Life insurance and funeral assistance
  • Internet allowance
  • Home office stipend
  • Competitive compensation — above the market average
  • 100% remote work environment and an excellent work-life balance
  • Opportunity to work for a growing global SaaS leader company
  • A culture that promotes independence, innovation, trust, and accountability
  • Open space to be creative, innovative, and strategize for the future
  • Mentorship by a highly experienced professional
  • Budget for training, we want you to grow
  • 5 Personal Time Off days per year
  • Sick Leave Top up to total 100% of salary paid by the employer from Day 3 to 90.
  • Recognition Award, additional paid time off in recognition of the corresponding year of service
  • Upgrade vacation starting at 5 years of service

Related Job Pages

More Artificial Intelligence Jobs

OtherRemoteTeam 5,001-10,000H1B Sponsor

• Collaborate with the AI team on innovative projects • Participate in the development of AI models • Support the implementation of AI solutions in Biogen's processes • Engage with cross-functional teams to drive initiatives

United States
$23 - $29 / hour
Job Closed
Volga Partners logo

AI Evaluation & Annotation Specialist

Volga Partners

Smart and Reliable Technology Solutions

ContractRemoteTeam 1,001-5,000Since 2020H1B Sponsor

Role Description Are you curious, detail-oriented, and excited about shaping the future of artificial intelligence? We’re looking for AI Evaluation & Annotation Specialists to help train and improve Large Language Models (LLMs). In this role, you’ll review AI-generated responses, provide corrections, evaluate quality, and follow structured guidelines to ensure accuracy and consistency. No engineering background required — if you enjoy problem-solving, analyzing language, and following structured tasks, this role is a great fit. This is a hands-on, production-based role where accuracy, focus, and consistency matter. What You’ll Do - Review AI-generated responses and rate them for clarity, correctness, and relevance. - Annotate and label content based on project-specific guidelines. - Follow detailed written instructions and apply them consistently. - Generate or evaluate prompts depending on assignment type. - Work with QA Leads to apply feedback and continuously improve task quality. - Report completed work daily and meet productivity and quality standards. Qualifications - You’re detail-oriented and enjoy accuracy-based work. - You can follow instructions carefully and apply them consistently. - You’re comfortable working independently with minimal supervision. - You have strong reading comprehension and critical thinking skills. - You communicate clearly and respond to feedback professionally. - Experience with annotation, evaluation, translation, linguistics, or QA is helpful, but not required — training and guidance is provided. Requirements - Fluency in Korean, with strong written and verbal communication skills. - Bachelor's degree in Linguistics, Computer Science, or a related field; or equivalent experience. - Experience or interest in AI, Machine Learning, or data annotation preferred. - Strong attention to detail and ability to work with complex data sets. - Familiarity with translation and localization processes is an advantage. - Ability to work independently and collaboratively in a fast-paced environment. - Proficient in using Artificial Intelligence and data annotation tools, with a willingness to learn new technologies. Benefits - Work with a global team. - Entry point into the growing AI and language technology industry. - Exposure to real-world AI model training. - Skill development in annotation, QA/evaluation, and structured AI tasks. Compensation Range Rates vary by language and experience level (L1/L2). Below are current approved ranges in USD: - $11.00 USD to $14.00 USD per hour. Assessment Requirement All applicants are required to complete a short skills-based assessment as part of the selection process. This assessment is unpaid and is used solely to confirm eligibility and alignment with project quality standards. Completing the assessment does not guarantee selection; however, it is mandatory in order to be considered for this role.

Canada + 9 moreAll locations: Canada | Brazil | Colombia | Argentina | Chile | China | Bolivia | Cuba | Ecuador | Iran
$11 - $14 / hour
Job Closed

• Provide quality instruction in each assigned course within the approved academic program curriculum. • As appropriate, the Dean/Department Chair may solicit assistance in the development, implementation, and continuous evaluation of the curriculum, including sequence of courses and student learning outcomes. • Should respond, in a timely manner, to specific and general information requests from the institution and administrative officials, prospective employers, professional organizations, public agencies, civic organizations, private foundations, general public, and students, as appropriate. • At all times, promote appropriate standards of linguistic expression in both written and oral communications. • Ensure that all academic program requirements and forms of documentation (e.g., clinical evaluations, competency documentation) are completed as required for each student and submitted per established deadlines. • Ensure all faculty expectations are met on a weekly basis. • Appropriately manage all classroom activities. • Be reasonably accessible to students for questions and assistance. • Monitor educational and professional literature for the best practices in areas related to courses taught. • Assist with the development of curricular and policy changes as requested by the Dean/Department Chair. • Assist with the selection of new textbooks as appropriate. • Adjunct (part-time) faculty members teaching on a regular basis are required to participate in professional development activities – minimum of 1 per year.

United States
DocPlanner logo

Specjalist ds. sprzedaży AI – klient biznesowy

DocPlanner

At Docplanner Group, we’re on a mission to help people live longer, healthier lives. As the world’s largest healthcare platform, each month, we connect 24 million patients with 280k doctors across 13 countries. Our marketplaces, SaaS and AI tools simplify daily tasks and help doctors, clinics and hospitals work more efficiently. Real impact – We help doctors help patients. Your work truly makes a difference. At scale, yet agile – 3,000+ employees, but still fast, flexible, and hands-on. Shape the future, sustain growth – Make a difference now and build for long-term success.

Full TimeRemoteTeam 1,001-5,000Since 2012H1B No Sponsor

• Prowadzeniem kompleksowego procesu sprzedażowego B2B – od pierwszego kontaktu, przez analizę potrzeb biznesowych, aż po przekazanie klienta do zespołu wdrożeniowego (Customer Success) • Obsługą dostarczonej bazy leadów – Twoim zadaniem będzie kontakt z potencjalnymi klientami zgodnie z naszym standardem sprzedaży doradczej • Kluczowe w naszym procesie jest zrozumienie wyzwań, z jakimi mierzą się placówki i zaproponowanie najlepszego rozwiązania • Budowaniem relacji z decydentami – menedżerami, właścicielami, dyrektorami medycznymi i członkami zarządów • Prowadzeniem rozmów sprzedażowych telefonicznie, mailowo oraz w formie spotkań online • Analizą celów i wyzwań placówek medycznych, by jak najlepiej dopasować naszą ofertę • Prezentowaniem ofert dostosowanych do potrzeb różnych struktur organizacyjnych • Realizacją celów sprzedażowych, opartych na wartości kontraktu, liczbie wdrożeń i aktywowanych użytkowników • Ścisłą współpracą z zespołami Customer Success, Product i Marketingu, by zapewnić płynne wdrożenie i długofalową satysfakcję klienta

Poland
zł6K - zł6.5K / month