We are a Y-Combinator-backed startup building your AI-powered Recruiter Agent
AI Evaluation Specialist
Location
United States
Posted
2 days ago
Salary
$70 / hour
Seniority
Mid Level
No structured requirement data.
Job Description
AI Evaluation Specialist
Weekday (YC W21)
Role Description Join a cutting-edge AI research initiative focused on improving the quality, accuracy, and reasoning capabilities of next-generation artificial intelligence systems. We are seeking analytical professionals with exceptional critical thinking and communication skills to evaluate AI-generated responses across a variety of topics. In this role, you will assess AI outputs, identify strengths and weaknesses in reasoning, and provide structured, evidence-based feedback that helps improve model performance. This opportunity is ideal for individuals who enjoy careful analysis, attention to detail, and working independently on intellectually challenging tasks. This is a fully remote, contract-based opportunity with flexible working hours. Key Responsibilities - Evaluate AI Responses - Review AI-generated content for accuracy, logical reasoning, completeness, and clarity. - Identify factual errors, reasoning gaps, inconsistencies, and unsupported conclusions. - Assess responses using structured evaluation frameworks and detailed quality guidelines. - Provide High-Quality Feedback - Write clear, concise, and evidence-based rationales explaining evaluation decisions. - Highlight both strengths and areas for improvement in AI-generated outputs. - Apply consistent judgment across a wide range of evaluation tasks. - Maintain Evaluation Quality - Follow detailed project instructions and standardized assessment criteria. - Ensure evaluations are objective, accurate, and reproducible. - Complete assignments independently while maintaining high quality standards. Qualifications - Bachelor's degree from a globally recognized university (top-ranked institutions preferred). - Excellent analytical thinking and problem-solving abilities. - Strong written communication skills with the ability to explain complex reasoning clearly and precisely. - Exceptional critical reading skills, including the ability to identify: - Nuanced arguments - Implicit meaning - Logical inconsistencies - Missing context - Weak or unsupported reasoning - Strong attention to detail and ability to consistently apply structured evaluation guidelines. - Ability to work independently and manage assigned tasks efficiently. - Native-level English fluency. Preferred Qualifications - Experience in content evaluation, research, quality assurance, editing, or analytical review. - Familiarity with artificial intelligence, large language models, or AI evaluation methodologies. - Experience working with structured annotation or assessment frameworks. - Ability to produce thoughtful, objective, and well-supported written evaluations under defined quality standards. Engagement Details - Independent contractor engagement. - Fully remote with flexible working hours. - Work completed on your own schedule. - Project duration may be extended, shortened, or concluded based on business needs and performance. - Weekly payments processed through supported payment platforms. Why Join - Contribute to the development of next-generation AI technologies. - Help improve the reasoning, accuracy, and reliability of advanced AI systems. - Work on intellectually engaging projects with real-world impact. - Collaborate indirectly with leading AI researchers through high-quality evaluation work. Equal Opportunity Statement All qualified applicants will be considered without regard to legally protected characteristics. Reasonable accommodations are available upon request. Contract Information - Independent contractor engagement. - Fully remote work completed on your own schedule. - Weekly payments are processed based on approved work completed. - Work does not involve access to confidential or proprietary information from any employer, client, or institution. - Please note that visa sponsorship is not available for this opportunity.
Related Guides
Related Categories
Related Job Pages
More Artificial Intelligence Jobs
Video Data Collector - AI Training
Toloka AnnotatorsBe a key player in crafting the high-quality data essential for AI innovation. Perfect for aspiring freelancers
Role Description This is a project-based opportunity on an AI training platform. No fixed hours, no commitment beyond what fits your schedule. You record, you get paid. We're looking for people to record point-of-view videos of everyday household activities. You mount your smartphone on your head, go about your routine, and the camera captures exactly what you see. These videos help train AI systems and robots to understand how people interact with objects in real-world environments. Do you wash dishes every day? Vacuum the rug? Walk the dog? Fold laundry? What if you could get paid just for recording it? Responsibilities - Record everyday household activities from a first-person perspective — cooking, cleaning, tidying up, folding laundry, gardening, walking the dog, and more. - Record in your own home, yard, garage, or workspace. - Follow short recording guidelines per session. - Upload videos through the project app. Requirements - Equipment: Own or use a smartphone or alternative camera device capable of 30 FPS or greater and 1080p resolution or greater. iPhone 12 or newer, Google Pixel 6+, or Samsung Galaxy S21+ (must have a wide-angle / 0.5x lens). - Capture method: Mount the device at forehead or eye-level for an egocentric perspective, maintaining a downward angle to maximize hand visibility. - Software: Utilize the designated "D&A" application for all data capture and upload. - Quality control: Ensure all video footage is clear, stable, well-lit, at least 15 seconds long, and accurately depicts useful, intentional tasks without unnecessary pauses or contrived movements. Benefits - Earn while you do your household chores: up to $6 per hour equivalent depending on the project. - This is project-based freelance work — pick up recording sessions when they fit your schedule. - Influence the future of how AI and robots understand and interact with the real world.
• Take operational ownership of the configuration, reliability, and standards of Storyteller’s AI tools and AI platforms. • Support the AI Operations team’s response when AI tools, platforms, models, or underlying capabilities change. • Set, maintain, and improve operational standards for how AI tooling is configured, accessed, used, and monitored. • Improve visibility around AI tool configuration, platform usage, access administration, permissions, dependencies, and operational performance. • Identify weak, unstable, or high-friction areas in our AI tooling setup and help develop practical solutions that are durable and scalable. • Work with internal teams to understand where AI tooling is helping, where it is creating friction, and what needs to improve. • Give leadership clearer visibility on where the AI tooling stack is performing well, where it is fragile, and what platform or workflow changes are needed next. • Help ensure Storyteller’s AI tooling remains useful, controlled, reliable, and aligned with how the business actually works.
• Take operational ownership of the configuration, reliability, and standards of Storyteller’s AI tools and AI platforms. • Support the AI Operations team’s response when AI tools, platforms, models, or underlying capabilities change. • Set, maintain, and improve operational standards for how AI tooling is configured, accessed, used, and monitored. • Improve visibility around AI tool configuration, platform usage, access administration, permissions, dependencies, and operational performance. • Identify weak, unstable, or high-friction areas in our AI tooling setup and help develop practical solutions that are durable and scalable. • Work with internal teams to understand where AI tooling is helping, where it is creating friction, and what needs to improve. • Give leadership clearer visibility on where the AI tooling stack is performing well, where it is fragile, and what platform or workflow changes are needed next. • Help ensure Storyteller’s AI tooling remains useful, controlled, reliable, and aligned with how the business actually works.
• Take operational ownership of the configuration, reliability, and standards of Storyteller’s AI tools and AI platforms. • Support the AI Operations team’s response when AI tools, platforms, models, or underlying capabilities change. • Set, maintain, and improve operational standards for how AI tooling is configured, accessed, used, and monitored. • Improve visibility around AI tool configuration, platform usage, access administration, permissions, dependencies, and operational performance. • Identify weak, unstable, or high-friction areas in our AI tooling setup and help develop practical solutions that are durable and scalable. • Work with internal teams to understand where AI tooling is helping, where it is creating friction, and what needs to improve. • Give leadership clearer visibility on where the AI tooling stack is performing well, where it is fragile, and what platform or workflow changes are needed next. • Help ensure Storyteller’s AI tooling remains useful, controlled, reliable, and aligned with how the business actually works.


