AI Evaluation Specialist
Location
United States
Posted
1 day ago
Salary
$70 / hour
Seniority
Senior
Job Description
AI Evaluation Specialist
Weekday
• Review AI-generated content for accuracy, logical reasoning, completeness, and clarity. • Identify factual errors, reasoning gaps, inconsistencies, and unsupported conclusions. • Assess responses using structured evaluation frameworks and detailed quality guidelines. • Write clear, concise, and evidence-based rationales explaining evaluation decisions. • Highlight both strengths and areas for improvement in AI-generated outputs. • Apply consistent judgment across a wide range of evaluation tasks. • Follow detailed project instructions and standardized assessment criteria. • Ensure evaluations are objective, accurate, and reproducible. • Complete assignments independently while maintaining high quality standards.
Job Requirements
- Bachelor's degree from a globally recognized university (top-ranked institutions preferred).
- Excellent analytical thinking and problem-solving abilities.
- Strong written communication skills with the ability to explain complex reasoning clearly and precisely.
- Exceptional critical reading skills, including the ability to identify:
- Nuanced arguments
- Implicit meaning
- Logical inconsistencies
- Missing context
- Weak or unsupported reasoning
- Strong attention to detail and ability to consistently apply structured evaluation guidelines.
- Ability to work independently and manage assigned tasks efficiently.
- Native-level English fluency.
- Experience in content evaluation, research, quality assurance, editing, or analytical review (preferred).
- Familiarity with artificial intelligence, large language models, or AI evaluation methodologies (preferred).
- Experience working with structured annotation or assessment frameworks (preferred).
- Ability to produce thoughtful, objective, and well-supported written evaluations under defined quality standards (preferred).
Related Guides
Related Categories
Related Job Pages
More Artificial Intelligence Jobs
AI-Native Marketing Manager
Human IntelligenceAt Human Intelligence, we're building a future where artificial intelligence helps people better understand themselves, better understand each other, and ultimately create a happier, healthier, and more connected world. We rely on exceptional systems working behind the scenes. We're looking for an exceptional GoHighLevel Automation Specialist to help build, optimize, and continuously improve one of the world's most AI-first CRM ecosystems. Our culture emphasizes strategic systems, friction removal, quality improvement, and intentional customer journeys.
Role Description As our AI-Native Marketing Manager, you'll lead and execute integrated marketing initiatives across multiple products, brands, campaigns, and customer journeys. You'll combine strategic thinking with practical execution while continuously using AI to improve efficiency, creativity, personalization, and marketing performance. This is a hands-on leadership role requiring strong commercial thinking, outstanding communication skills, exceptional organization, and genuine curiosity about emerging technology. Qualifications - 5+ years of professional marketing experience. - Experience managing integrated digital marketing campaigns. - Strong understanding of customer acquisition, conversion optimization, and customer lifecycle marketing. - Experience with email marketing, marketing automation, CRM systems, SEO, and digital analytics. - Excellent written and verbal English communication skills. - Strong analytical and commercial thinking. - Outstanding organizational and project management skills. - Experience managing multiple campaigns simultaneously. - Ability to work independently within a fully remote environment. Requirements - Experience using AI for campaign planning, research, customer insights, copy generation, SEO, analytics, reporting, workflow automation, creative ideation, and marketing optimization. - Experience with ChatGPT and other AI productivity platforms is expected. Benefits - Employment Type: Independent Contractor - Location: Fully Remote - Monthly Base Compensation: USD $1,500–$2,500 per month, depending on your experience, marketing capability, AI proficiency, commercial results, and overall fit. - Performance Incentives: Monthly KPI and performance incentives based on measurable outcomes including marketing-qualified leads (MQLs), revenue contribution, campaign ROI, customer acquisition, conversion rate improvements, product launch success, marketing automation improvements, AI implementation and innovation, and cross-functional contribution. - Compensation reviews conducted following successful completion of the probationary period based on sustained performance, leadership, initiative, and business impact.
Senior Integrative Designer
Human IntelligenceAt Human Intelligence, we're building a future where artificial intelligence helps people better understand themselves, better understand each other, and ultimately create a happier, healthier, and more connected world. We rely on exceptional systems working behind the scenes. We're looking for an exceptional GoHighLevel Automation Specialist to help build, optimize, and continuously improve one of the world's most AI-first CRM ecosystems. Our culture emphasizes strategic systems, friction removal, quality improvement, and intentional customer journeys.
Role Description As our Senior Integrative Designer, you'll own the creative execution of a broad range of business, educational, marketing, and product assets. You'll combine strategic thinking, exceptional visual design, AI-assisted creativity, and outstanding attention to detail to produce consistently high-quality work across multiple brands and initiatives. This is an AI-first role. We're looking for someone who actively embraces modern AI creative tools to increase speed, quality, consistency, and creative exploration while maintaining exceptional design standards. You'll collaborate closely with marketing, leadership, product, operations, copywriters, developers, and automation specialists to bring ideas to life quickly and beautifully. What You'll Be Doing - Brand & Visual Identity: - Design premium visual assets that strengthen the Human Intelligence brand. - Maintain consistency across all digital and physical touchpoints. - Develop visual systems, templates, and reusable design libraries. - Ensure brand consistency across multiple products and initiatives. - Business Resources: - Design presentations and keynote decks. - Business proposals. - White papers. - Guides. - Reports. - Brochures. - Infographics. - Educational resources. - Downloadable PDFs. - Internal documentation. - Executive presentations. - Digital Marketing: - Social media graphics. - Landing pages. - Website visuals. - Advertising creative. - Email marketing assets. - Sales collateral. - Campaign graphics. - Product launches. - Promotional resources. - AI Creative Production: - Leverage AI tools to accelerate creative workflows, including: - Image generation. - Concept exploration. - Graphic enhancement. - Video editing assistance. - Presentation design. - Layout generation. - Creative ideation. - Rapid prototyping. - Asset variations. - Workflow automation. - Cross-Functional Collaboration: - Work closely with Marketing to improve campaign performance. - Support Product with UI assets and product graphics. - Collaborate with Copywriters to create compelling visual storytelling. - Assist Leadership with investor, partner, and executive presentations. - Partner with developers to ensure designs are implemented accurately. - Creative Quality: - Review your own work thoroughly before submission. - Maintain exceptional attention to detail. - Produce polished, production-ready assets. - Continuously improve design systems and creative workflows. - Help establish Human Intelligence as a world-class visual brand. Qualifications - 5+ years of professional graphic design, digital design, or integrated design experience. - A strong portfolio demonstrating premium-quality creative work across multiple mediums. - Advanced proficiency with Adobe Creative Cloud (Photoshop, Illustrator, InDesign and Acrobat). - Experience designing business presentations, reports, brochures, educational resources, and marketing collateral. - Strong typography, layout, hierarchy, and visual communication skills. - Experience designing for both print and digital environments. - Excellent written and spoken English. - Strong attention to detail and quality control. - Ability to manage multiple projects simultaneously. - Experience working within established brand guidelines while contributing to brand evolution. Requirements - Experience with tools such as ChatGPT, Midjourney, Adobe Firefly, Runway, Kling, Veo, Canva AI, Figma AI, Gamma, or similar platforms will be highly regarded. - You don't need to use every tool. What matters is that you genuinely enjoy learning new technologies and continually improving your creative workflow. Benefits - Employment Type: Independent Contractor - Location: Fully Remote - Compensation: USD $1000 - $1500 + performance incentives up to $500/mo - Competitive monthly USD retainer based on experience, capability, portfolio quality, and geographic location, plus performance incentives where applicable.
Patient Engagement Specialist – AI Quality
iSTA SolutionsStreamline your workflows with iSTA Solutions. Tailored remote staffing solutions and comprehensive support services.
• AI Oversight & Quality Assurance: Monitor AI-generated patient communications across multiple channels; review and adjust conversations for accuracy, quality, tone, and professionalism • Tone & Empathy Optimization: Refine AI responses to be clear, conversational, empathetic, and culturally attuned to a U.S. audience — helping actively shape how the AI communicates, not just checking it • Human Intervention & Escalation: Spot errors, inconsistencies, or complex questions; step in on nuanced conversations, gather context, and route critical details to the right team members • Patient Outreach: Send accurate messages, updates, and routine reminders per established workflows • Documentation & Support: Help create and maintain patient-facing support documentation and materials • Systems & Platform Adaptation: Work across Airtable and the client's proprietary in-house application, which is currently mid-migration — expect to operate across both platforms in the near term before the team moves fully onto the native app • Process Improvement & Reporting: Work with internal teams to spot messaging trends, categorize information, and suggest improvements to AI performance and workflow • Compliance & Service Levels: Keep meticulous records of patient interactions in line with healthcare policy, while meeting SLAs — with quality prioritized over raw speed
Patient Engagement Specialist (AI Quality)
ISTA Personnel SolutionsISTA Personnel Solutions is a dynamic, fast-growing BPO company. We are not a recruitment agency — we operate as a dedicated extension of our clients’ teams, delivering high-quality service with precision, efficiency, and a personal touch.
Role Description We're currently hiring a perceptive, proactive, and tech-savvy Patient Engagement Specialist (AI Quality) to join an exciting, forward-thinking medical campaign supporting a U.S.-based healthcare business. In this role, you'll oversee and refine AI-generated communications between patients and the healthcare team. The AI handles the bulk of the messaging, but you'll review, correct, and shape it — not just monitoring for errors, but actively influencing how the AI "speaks" to patients, so it reads as natural, empathetic, and culturally appropriate for a U.S. audience. You'll step in personally whenever a conversation needs a human touch. This is a long-term role with genuine growth potential. It starts out process-heavy and hands-on, evolving into an oversight and quality-control function — and as the business scales, there's a clear path toward a leadership position managing others doing the same work. PLEASE NOTE - Working Hours: Monday–Friday, 14:00–23:00 SAST (subject to daylight savings). - Public Holidays: This role requires working on both South African and U.S. public holidays (compensation for SA public holidays in accordance with the BCEA). - Internet Requirements: A fixed fibre line with a minimum speed of 25 Mbps (upload & download) and wired Ethernet capability is mandatory. Applicants without a fixed fibre line cannot be considered. - Power Backup: A reliable backup is required to manage load shedding or outages. Applicants without a power backup cannot be considered. - Work Environment: Fully remote. Qualifications - 6 months–1 year of experience in the U.S. healthcare industry preferred (EHR systems or healthcare admin experience a plus). - Excellent English communication skills, with the ability to handle sensitive and nuanced conversations professionally. - Naturally empathetic, highly detail-oriented, organized, and proactive. - Computer literate, proficient in Microsoft Office (Outlook and Word; no advanced Excel required). - Proven ability to work independently in a fast-paced remote environment, navigating multiple databases and digital tools. Requirements - Monitor AI-generated patient communications across multiple channels; review and adjust conversations for accuracy, quality, tone, and professionalism. - Refine AI responses to be clear, conversational, empathetic, and culturally attuned to a U.S. audience — helping actively shape how the AI communicates, not just checking it. - Spot errors, inconsistencies, or complex questions; step in on nuanced conversations, gather context, and route critical details to the right team members. - Send accurate messages, updates, and routine reminders per established workflows. - Help create and maintain patient-facing support documentation and materials. - Work across Airtable and the client's proprietary in-house application, which is currently mid-migration — expect to operate across both platforms in the near term before the team moves fully onto the native app. - Work with internal teams to spot messaging trends, categorize information, and suggest improvements to AI performance and workflow. - Keep meticulous records of patient interactions in line with healthcare policy, while meeting SLAs — with quality prioritized over raw speed. Benefits

