Cincinnatus is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for full-time and long-term contingent roles. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives. Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve full-time or fixed-term commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows. Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus. Equal Employment Opportunity Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic. Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.
AI Evaluator - Software Expert
Location
Worldwide
Posted
5 days ago
Salary
$80 - $120 / hour
Seniority
Mid Level
No structured requirement data.
Job Description
AI Evaluator - Software Expert
Mercor
Role Description - Evaluate AI-generated artifacts against domain-specific quality rubrics. - Identify factual, aesthetic, and presentation errors in documents, spreadsheets, and slide decks. - Provide clear, structured written feedback to improve AI model outputs. - Apply deep subject-matter expertise to grade outputs for accuracy and rigor. - Work independently and asynchronously to meet deadlines. Qualifications - Must-Have: 5+ years of relevant professional experience in Software / AI / IT / data. - Native or professional fluency in English. - Highly proficient in Microsoft Office and Google Workspace, especially Slides. - Preferred: Advanced degree (Master's or higher) from a reputable institution. Company Description Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Related Guides
Related Categories
Related Job Pages
More DevOps Engineer Jobs
Senior DevOps Engineer
BorrowellBorrowell is a financial technology company based in Toronto, Ontario, Canada, that provides free credit score reporting and monitoring and a range of other fin
• We are seeking a Senior DevOps Engineer to design, automate, and manage the mission-critical cloud infrastructure supporting our marketplace. • As a key member of our DevOps team, you will drive our Infrastructure-as-Code (IaC) strategy, develop tools to simplify our deployment processes, and integrate automated testing into our delivery pipelines. • This is a collaborative, cross-functional role requiring close partnership with development, security, and QA teams. The ideal candidate is a proactive engineer with a strong foundation in cloud platforms, CI/CD automation, containerization, and security best practices, dedicated to helping us deliver reliable services to 4 million members.
• Design and manage scalable, secure, and highly available infrastructure on Google Cloud Platform (GCP) • Develop and maintain Infrastructure as Code (IaC) using Terraform or Deployment Manager • Build and optimize CI/CD pipelines using tools like Cloud Build, GitHub Actions, or Jenkins • Implement and manage GKE (Google Kubernetes Engine) clusters, including autoscaling, monitoring, and security • Set up and manage Cloud Monitoring, Cloud Logging, and Error Reporting for observability • Automate operational tasks using Python • Ensure compliance with security best practices using IAM, VPC Service Controls, and Cloud Armor • Collaborate with development teams to support microservices architecture and service mesh (e.g., CloudRun, CloudFunction) • Lead incident response and root cause analysis for production issues • Mentor junior DevOps engineers and contribute to DevOps best practices and documentation
• Design, build, and operate cloud infrastructure (GCP preferred) using infrastructure-as-code, with an emphasis on repeatability, security, and cost efficiency. • Own and continuously improve CI/CD pipelines. Automated integration and unit testing, provisioning, deployments, and rollbacks to keep delivery fast and safe. • Build and maintain observability across the platform, including monitoring, logging, tracing, alerting, and meaningful dashboards that surface issues before customers do. • Improve and advance our security posture: secrets management, encryption in transit and at rest, IAM and least-privilege access, network segmentation, and vulnerability management across all infrastructure. • Drive compliance readiness by partnering with security and leadership to maintain, automate, and provide evidence for controls across frameworks such as SOC 2, ISO 27001, GDPR, (HIPAA a plus), including audit support and continuous control monitoring. • Lead incident response and the on-call rotation; drive blameless postmortems, reduce mean-time-to-recovery, and turn lessons learned into lasting fixes. • Define and uphold reliability targets (SLOs/SLIs), capacity planning, and performance tuning as we scale across countries and industries. • Leverage AI-powered tooling (Claude Code, Cursor, GitHub Copilot, and others) to accelerate infrastructure-as-code, automation, and internal tooling, and to improve incident triage and response. • Partner with engineering to improve developer experience and deployment velocity, removing friction and automating away toil. • Drive a culture of operational excellence, reliability, security, and continuous improvement across the engineering organization. • Set technical direction for platform and infrastructure, and mentor engineers on DevOps, reliability, and security best practices. • Continuously evaluate and adopt emerging AI-powered tools and workflows to improve infrastructure speed, quality, and security.
Senior DevOps Engineer
TrueMLTrueML is a fintech company building software to create positive experiences for consumers seeking financial health.
• Implement the technical roadmap for Infrastructure as Code (IaC), CI/CD evolution, and cloud-native architecture to support TrueML’s scaling needs. • Design, develop, and maintain self-service internal platforms to reduce developer cognitive load, enabling feature teams to deploy and manage services with minimal friction at increased velocity. • Act as a core steward for cloud spend (AWS), proactively identifying and driving cost-optimization initiatives across our infrastructure. • Build and maintain infrastructure architecture that supports strict High Availability (HA) requirements and robust Disaster Recovery (DR) protocols across multiple regions. • Implement and evolve comprehensive monitoring, logging, and distributed tracing systems, leveraging AIOps to move from reactive to predictive system maintenance.



