Cincinnatus is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for full-time and long-term contingent roles. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives. Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve full-time or fixed-term commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows. Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus. Equal Employment Opportunity Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic. Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.
Software Engineer - Evaluation Author - AI Trainer
Location
United States
Posted
3 days ago
Salary
$35 - $120 / hour
Seniority
Mid Level
Job Description
Software Engineer - Evaluation Author - AI Trainer
Mercor
Role Description - Author non-trivial coding tasks with golden solutions and automated verifiers. - Design rubrics and grade agent trajectories and model outputs. - Improve task and rubric quality through structured review. - Evaluate the accuracy and depth of AI-generated content to strengthen reasoning and rigor in model outputs. - Work independently and asynchronously to meet deadlines while improving AI model performance. Qualifications - Must-Have: 5+ years of software engineering at a real product organization (big tech or venture-backed startup). - Strong code quality, systems design, debugging, and testing discipline. - Clear written communication (you write instructions others follow). - Preferred: Familiarity with AI coding tools and evals. Requirements - Short Mercor Technical Screen. - Live Code Review Session. - Domain Expert Interview. - You're paid $200 for completing all three, regardless of outcome. Benefits - Compensation: $35–$120/hour. - Location: Remote. - Commitment: 30+ hours/week. Application Process - Upload resume. - AI interview based on your resume. - Submit form. Resources & Support - For details about the interview process and platform information, please check: Mercor Resources . - For any help or support, reach out to: support@mercor.com . - Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Related Guides
Related Job Pages
More Full-stack Engineer Jobs
Senior Full-Stack Engineer
DistroDistro is a marketplace to find, hire, and pay technical talent in over 200 countries. Join now for free.
• Design and develop complex web applications using the latest front-end and back-end technologies. • Provide backend support and rapid bug fixes for daily updates, patches, and team-identified issues. • Be on-call for backend troubleshooting and quick resolution. • Create and maintain efficient, reusable, and reliable code. • Diagnose and troubleshoot technical issues. • Integrate data storage solutions such as databases and cloud storage. • Develop and implement AI-driven solutions to streamline business processes, including: • Designing and building AI document comparison functionality to compare applications against supporting documents and receipts. • Creating automated validation systems that flag mismatches for manual review and allow matching documents to proceed automatically. • Leveraging AI to reduce manual audit work and identify automation opportunities for repetitive, detailed processes. • Collaborate with developers, designers, and stakeholders to deliver quality applications. • Ensure applications meet performance, scalability, and security requirements. • Build and manage weekly plans, keep estimates current, and ensure steady, demo-able progress.
• Develop Software Implementation Designs – The Software Developer will create documentation that will explain the design of the software solution. • Develop Software Unit Tests – Create Unit Tests that prove the solution works as designed using Test Driven Development practices. • Create Software Installation Packages – Create installation packages that will allow our customers to install software on workstations and servers. • Create Software Implementations – Create source code that meets the needs of the design requirements by satisfying the rule established in a unit test. • Participate in Code Reviews – All code produced will be subject to a code review prior to check in to the source code control system. The Software Developer will be responsible for making modifications to the code based on the outcome of the code review. • Participate in Software Implementation Planning – The Software developer will participate in software planning sessions by contributing estimates of effort required to accomplish specific software design implementations. • Participate in Implementation Review Sessions – The Software Developer will participate in implementation review sessions. These sessions allow the developer to receive feedback from key stakeholders by demonstrating the software implementation. • Deliverables: Software source code software unit tests installation packages. • Architecture documents database scheme diagrams entity relationship diagrams sequence diagrams
• Design and implement multi-agent AI systems using frameworks such as LangChain, AutoGen, and CrewAI. • Build RAG (Retrieval-Augmented Generation) pipelines with vector databases like Pinecone or Weaviate. • Implement agent orchestration patterns and inter-agent communication protocols for distributed AI workflows. • Develop LLM fine-tuning workflows and optimize prompt engineering for production use. • Design and deploy Microsoft Copilot Foundry solutions with custom knowledge bases and integration workflows. • Develop backend services using C# /.NET (ASP.NET Core, Entity Framework, SignalR, Minimal APIs) and Python (FastAPI, Django, Flask). • Implement microservices and containerized deployments using Docker and event-driven architectures with Azure Service Bus or RabbitMQ. • Integrate Azure services including Azure Functions, Azure Logic Apps, Azure Cognitive Services, Azure OpenAI Service, and Azure Data Factory. • Build and maintain CI/CD pipelines and infrastructure as code using Azure DevOps, ARM/Bicep, or Terraform. • Mentor junior developers, document architecture decisions, and collaborate with product managers and data scientists. • Balance technical debt with feature delivery while considering scalability, maintainability, code quality, testing, and documentation. • Implement monitoring and observability using Application Insights. • Apply Git workflow management and branching strategies in collaborative development environments.
• Design, develop, and implement AI-powered features across healthcare product platforms and operational systems. • Build full-stack applications using modern AI engineering workflows and tools such as Cursor, Claude, Codex, OpenAI APIs, and related LLM technologies. • Collaborate with product managers, designers, engineers, and clinical stakeholders to deliver scalable, user-focused solutions that improve patient engagement, care coordination, and operational efficiency. • Develop APIs, backend services, automation pipelines, and AI workflows that support intelligent product capabilities and production-ready deployments. • Apply software engineering, DevOps, and MLOps best practices to testing, deployment, monitoring, and optimization of AI solutions. • Research emerging AI technologies, LLM implementation patterns, and modern product engineering frameworks to continuously improve platform capabilities. • Navigate ambiguity, solve complex technical challenges, and collaborate effectively across technical and non-technical teams in a fast-paced Agile environment. • Take on additional tasks and responsibilities as needed to support team objectives and ensure the success of the project.



