#sejaSysMap #SysMap #soulSysMap
Mid-level AI Engineer – AI Systems QA
Location
Brazil
Posted
48 days ago
Salary
0
Seniority
Senior
Job Description
Mid-level AI Engineer – AI Systems QA
SysMap Solutions
• Design and implement evaluation and testing frameworks for AI systems (LLMs, RAG, agents); • Create quality metrics, benchmarks, and validation datasets for models in production; • Automate QA pipelines integrated into the development cycle (CI/CD); • Identify and document failures, hallucinations, and behavioral regressions in models; • Collaborate with engineering and product teams to define acceptance criteria for AI features; • Monitor the quality of production systems and propose continuous improvements.
Job Requirements
- Experience with Python and libraries in the AI/ML ecosystem (LangChain, LlamaIndex, Hugging Face, or similar);
- Practical experience in testing and evaluating LLM-based systems;
- Basic software engineering knowledge (version control, unit testing, REST APIs);
- Familiarity with prompting concepts, RAG, and agent orchestration;
- Experience with AWS services (Lambda, S3, SageMaker, Bedrock);
- Knowledge of observability and model monitoring tools;
- Experience with evaluation frameworks such as Ragas, DeepEval, or similar.
Related Guides
Related Categories
Related Job Pages
More QA Engineer Jobs
• Revisar y auditar código fuente en PHP, Java y C++ • Validar estándares de calidad y buenas prácticas de desarrollo • Ejecutar análisis de código estático utilizando SonarQube • Detectar vulnerabilidades, errores y oportunidades de mejora • Documentar hallazgos técnicos y asegurar su trazabilidad • Colaborar con el equipo de QA en criterios de calidad • Apoyar en la automatización de pruebas con Selenium, JUnit y TestNG
Ads Quality Rater - Thai (Thailand)
WelocalizeReach, Grow, and Engage Global Audiences with Multilingual Content
Welo Data is an award-winning localization and data transformation company. We run one of the world’s largest Ads Rating Programs and we want you to join! As an Ads Quality Rater, you will review and grade internet advertisements to help shape how ads are delivered to the end users. In other words, you’ll tell the AI program how good or bad its advertisement suggestions are based on the user’s keywords. You will review ads in both English and Thai This role is great for people who: - Surf the internet daily - Use major search engines frequently - Know what people want based on a few keywords - Enjoy researching topics online - Want a flexible and fun side job Project Details Job Title: Ads Quality Rater Pay Rate: Based on tasks (approximately $6 per hour) Location: Remote/work from home - must be based in Thailand Hours: Set your schedule based on the following – Minimum commitment is 5 hours per week. You can choose to work up to 20 hours per week. Start Date: ASAP Employment Type: Independent Contractor/Freelance/Self-Employed Project Duration: Long-term Note: - Even though the position is WFH, you must reside in the country that is noted in this description. This will be automatically checked during the hiring process. Recruitment Process - There is no formal interview for this job! No phone calls, no waiting, and no wasting time wondering whether you got the job or not. - Instead, you will be guided through a self-paced and automated recruitment process. IMPORTANT NOTE: Welo Data uses numerous identity checks to ensure that everyone who makes the team is real, qualified, and ready to work. To avoid any misunderstandings, do not use IP masking programs (such as VPNs). Experience & Requirements - Fluency in Thai - Strong understanding of English (written and spoken) - Comfortable knowledge of modern popular culture - Reliable computer system and internet connection - Familiar with using online search engines - Sign a standard Non-Disclosure Agreement and Service Level Agreement Please note that this work is based on project needs. Due to this, weekly hours may vary and flexibility with change is expected. You must legally be able to work on freelance projects in Thailand. To know more details (Click here)
Ads Quality Rater - Bulgarian (Bulgaria)
WelocalizeReach, Grow, and Engage Global Audiences with Multilingual Content
Welo Data is an award-winning localization and data transformation company. We run one of the world’s largest Ads Rating Programs and we want you to join! As an Ads Quality Rater, you will review and grade internet advertisements to help shape how ads are delivered to the end users. In other words, you’ll tell the AI program how good or bad its advertisement suggestions are based on the user’s keywords. You will review ads in both English and Bulgarian This role is great for people who: - Surf the internet daily - Use major search engines frequently - Know what people want based on a few keywords - Enjoy researching topics online - Want a flexible and fun side job Project Details Job Title: Ads Quality Rater Pay Rate: Based on tasks (approximately $7 per hour) Location: Remote/work from home - must be based in Bulgaria Hours: Set your schedule based on the following – Minimum commitment is 5 hours per week. You can choose to work up to 20 hours per week. Start Date: ASAP Employment Type: Independent Contractor/Freelance/Self-Employed Project Duration: Long-term Note: - Even though the position is WFH, you must reside in the country that is noted in this description. This will be automatically checked during the hiring process. Recruitment Process - There is no formal interview for this job! No phone calls, no waiting, and no wasting time wondering whether you got the job or not. - Instead, you will be guided through a self-paced and automated recruitment process. IMPORTANT NOTE: Welo Data uses numerous identity checks to ensure that everyone who makes the team is real, qualified, and ready to work. To avoid any misunderstandings, do not use IP masking programs (such as VPNs). Requirements - Fluency in Bulgarian - Strong understanding of English (written and spoken) - Comfortable knowledge of modern popular culture - Reliable computer system and internet connection - Familiar with using online search engines - Sign a standard Non-Disclosure Agreement and Service Level Agreement Please note that this work is based on project needs. Due to this, weekly hours may vary and flexibility with change is expected. You must legally be able to work on freelance projects in Bulgaria. To know more details (Click here)
Ads Quality Rater - Tamil (India)
WelocalizeReach, Grow, and Engage Global Audiences with Multilingual Content
Welo Data is an award-winning localization and data transformation company. We run one of the world’s largest Ads Rating Programs and we want you to join! As an Ads Quality Rater, you will review and grade internet advertisements to help shape how ads are delivered to the end users. In other words, you’ll tell the AI program how good or bad its advertisement suggestions are based on the user’s keywords. You will review ads in both English and Tamil This role is great for people who: - Surf the internet daily - Use major search engines frequently - Know what people want based on a few keywords - Enjoy researching topics online - Want a flexible and fun side job Project Details Job Title: Ads Quality Rater Pay Rate: Based on tasks (approximately $3 per hour) Location: Remote/work from home - must be based in India Hours: Set your schedule based on the following – Minimum commitment is 5 hours per week. You can choose to work up to 20 hours per week. Start Date: ASAP Employment Type: Independent Contractor/Freelance/Self-Employed Project Duration: Long-term Note: - Even though the position is WFH, you must reside in the country that is noted in this description. This will be automatically checked during the hiring process. Recruitment Process - There is no formal interview for this job! No phone calls, no waiting, and no wasting time wondering whether you got the job or not. - Instead, you will be guided through a self-paced and automated recruitment process. IMPORTANT NOTE: Welo Data uses numerous identity checks to ensure that everyone who makes the team is real, qualified, and ready to work. To avoid any misunderstandings, do not use IP masking programs (such as VPNs). Experience & Requirements - Fluency in Tamil - Strong understanding of English (written and spoken) - Comfortable knowledge of modern popular culture - Reliable computer system and internet connection - Familiar with using online search engines - Sign a standard Non-Disclosure Agreement and Service Level Agreement Please note that this work is based on project needs. Due to this, weekly hours may vary and flexibility with change is expected. You must legally be able to work on freelance projects in India. To know more details (Click here)


