Elevating Autism & IDD Care through Technology
Clinical AI Evaluation Specialist
Location
United States
Posted
26 days ago
Salary
$90K - $115K / year
Seniority
Mid Level
Job Description
Clinical AI Evaluation Specialist
CentralReach
• Contribute to evaluation frameworks for clinical AI products, including defining acceptance criteria, test plans, clinically relevant rubrics, and performance benchmarks. • Conduct structured sampling reviews of AI-generated outputs, assessing across criteria alignment with ABA principles and BACB ethical standards. • Design and implement monitoring automation in collaboration with engineering, including automated quality checks, alerting thresholds, and drift detection for AI systems. • Contribute to the creation and maintenance of reusable governance artifacts including system cards, monitoring playbooks, evaluation of SOP templates, and risk assessment documentation. • Assist with pre-deployment validation testing for new AI products and features, executing defined testing and results of documentation. • Participate in risk tiering assessments for new and evolving AI products, providing clinical and evaluative perspectives. • Support incident investigation for AI-related issues if/as needed and in alignment with clinical best practice and CR Development Policy. • Collaborate with product management and engineering teams to provide input during product planning and development, ensuring governance considerations are integrated early. • Contribute to the development of evaluation methodology and standards for clinical AI use cases, documenting approaches and refining processes based on findings and emerging best practices. • Contribute to AI literacy and training content for internal teams and customers, drawing on hands-on clinical evaluation experience. • Stay current with developments in AI evaluation methodology, LLM behavior, ABA best practices, and relevant regulatory and ethical standards. • Support CentralReach's thought leadership in responsible AI through contributions to conference presentations, professional publications, or applied research as opportunities arise.
Job Requirements
- Board Certified Behavior Analyst (BCBA) in good standing
- 2+ years of demonstrated experience in quality assurance, evaluation, data analysis, or technical assessment roles, preferably within healthcare or behavioral health settings.
- Demonstrated experience with large language models, generative AI, and their applications and limitations in clinical or professional contexts.
- Experience with prompt/context engineering, AI output evaluation, or AI workflow design.
- Experience with statistical sampling methodologies or healthcare quality assurance frameworks.
- Familiarity with AI governance frameworks (NIST AI RMF, EU AI Act risk categories, AMA governance guidance).
- Experience developing evaluation rubrics, scoring frameworks, or quality measurement tools.
- Background in clinical informatics, research methodology, or systematic quality improvement.
- Demonstrated proficiency with data analysis tools and comfort working with quantitative and qualitative evaluation data.
- Strong analytical skills with the ability to develop systematic, evidence-based evaluation approaches.
- Familiarity with HIPAA compliance, healthcare data standards, and ethical considerations in AI.
- Excellent written and verbal communication skills with the ability to document methodology clearly and convey findings to clinical, technical, and leadership audiences.
- Demonstrated ability to work collaboratively across clinical, technical, and product teams.
- Experience in a remote work environment with strong self-direction and accountability.
Benefits
- Health benefits
- Generous PTO
- 401(k) matching
- Paid parental leave
- Hybrid work schedules
- Career development support
- Wellness programs
Related Guides
Related Categories
Related Job Pages
More Artificial Intelligence Jobs
Mathematician
MercorCincinnatus is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for full-time and long-term contingent roles. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives. Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve full-time or fixed-term commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows. Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus. Equal Employment Opportunity Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic. Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.
Role Description - Solve advanced mathematical problems across competition math, proof-based mathematics, and quantitative reasoning to generate frontier AI training data. - Evaluate and provide expert feedback on AI-generated mathematical reasoning and proofs. - Develop original, research-grade problems and solutions in areas such as number theory, combinatorics, algebra, real analysis, and probability. - Apply quantitative and statistical methods to assess and improve model performance on mathematical benchmarks. - Collaborate with AI research teams to identify gaps in mathematical reasoning and design targeted training tasks. Qualifications - Demonstrated participation or achievement in elite competitive mathematics (IMO, USAMO, Putnam, or equivalent national/international olympiads). - Deep proficiency across multiple mathematical disciplines including proof-based mathematics, combinatorics, number theory, real and complex analysis, linear algebra, and probability. - Strong quantitative background with experience applying mathematical rigor to research or industry problems (e.g., quantitative finance, theoretical CS, or academic research). - Bachelor's degree or higher in Mathematics, Applied Mathematics, Statistics, or a related field (advanced degree preferred). - Ability to produce precise, well-structured mathematical writing. Requirements - Contract position. - Compensation: $40–$60/hour. - Location: Remote. Application Process - Upload resume. - AI interview based on your resume. - Submit form. Resources & Support - For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome . - For any help or support, reach out to: support@mercor.com . - Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Visual AI Photo Contributor
MercorCincinnatus is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for full-time and long-term contingent roles. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives. Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve full-time or fixed-term commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows. Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus. Equal Employment Opportunity Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic. Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.
Role Description - Submit photos to enhance visual AI systems. - Provide a variety of photos showing different looks, angles, and qualities. - Ensure all photos are clear, recent, and high quality. - Maintain authenticity with no AI-generated or digitally altered images. - Collaborate on licensing agreements for selected photos. Qualifications - Residency in the United States, excluding California, Illinois, Texas, and Washington. Requirements - Application Process (Takes 20–30 mins to complete): - Upload resume - AI interview based on your resume - Submit form Benefits - Compensation: $20/hour - Location: Remote Company Description Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
• Produce high-quality webstore product images (3000 x 3000) • Execute Product laydowns (clean, consistent, high-resolution) • Ensure logo accuracy and visibility • Generate realistic AI lifestyle images using tools such as Gemini • Create front and back studio images from lifestyle inputs • Maintain lighting consistency and garment accuracy • Apply logos to garments with realistic texture and placement • Create 3–5 second lifestyle animations from still images using Runway or similar • Follow structured prompts and workflows • Manage requests via Asana • Store and organize assets in SharePoint
Oncologic Radiologist
Florida Cancer SpecialistsSince 1984, Florida Cancer Specialists & Research Institute (FCS) has built a national reputation for excellence. With over 250 physicians, 220 nurse practitioners and physician assistants and nearly 100 locations in our network. Utilizing innovative clinical research, cutting-edge technologies, and advanced treatments, we are committed to providing world-class cancer care. Recognized by the American Society of Clinical Oncology (ASCO) with a national Clinical Trials Participation Award. Offers patients access to more clinical trials than any private oncology practice in Florida. Patients have access to ground-breaking therapies in a community setting. Partnership with Sarah Cannon, one of the largest clinical research organizations in the United States.
Role Description Florida Cancer Specialists & Research Institute is currently expanding our team of radiologists in a remote setting. We are currently hiring full-time radiologists who will be responsible for assigned clinics. This is an excellent opportunity to join a dynamic team committed to providing superior patient and community-centered care from the comfort of your home! - 100% remote - Outpatient oncology PET and CT imaging only - Patient continuity, assigned to specific regions to read previous imaging - Multi-disciplinary approach with internal medical oncology team Qualifications - Full unrestricted Florida MD/DO License - Board-eligible/certified in Diagnostic Radiology or Nuclear Medicine, dual board-certified preferred - Experience and expertise reading PET and CT scans in an oncology population - CT certification to read diagnostic CTs, unattached to PET/CT or SPECT/CT Requirements - Screenings – Background, drug, and nicotine screens - All offers of employment are contingent upon clear results of a thorough background screening - New hires must receive various vaccinations, including the influenza vaccine, barring an approved exemption - FCS is a drug-free workplace, and all new hires will be subject to drug/nicotine testing - Medical Marijuana cards are not recognized Benefits - No FL income tax - Comprehensive benefits including 401k match and generous PTO - Positive impact on community with white-glove service - No weekend or call coverage
