Cincinnatus is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for full-time and long-term contingent roles. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives. Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve full-time or fixed-term commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows. Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus. Equal Employment Opportunity Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic. Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.
Language Model Evaluator
Location
India
Posted
10 days ago
Salary
$15 - $20 / hour
Seniority
Mid Level
Job Description
Language Model Evaluator
Mercor
Role Description - Conduct fact-checking using trusted public sources and external tools. - Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies. - Assess reasoning quality, clarity, tone, and completeness of responses. - Ensure model responses align with expected conversational behavior and system guidelines. - Work independently and asynchronously to meet deadlines while improving AI model performance. Qualifications - Must-Have: - Bachelor's degree - Native speaker in Punjabi - Significant experience using large language models (LLMs) - Excellent writing skills in English - Strong attention to detail - Background or experience in domains requiring structured analytical thinking - Preferred: - Experience with RLHF, model evaluation, or data annotation work - Experience writing or editing high-quality written content - Experience comparing multiple outputs and making fine-grained qualitative judgments Company Description Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Related Guides
Related Categories
Related Job Pages
More Data Analyst Jobs
Role Description At CSpring, we believe in the power of people and data to drive real-world impact. We’re a purpose-driven consulting firm that helps organizations solve complex problems, gain insights, and achieve measurable results. Our clients span the public and private sectors, and our work — from data strategy and engineering to workforce transformation — improves lives across Indiana and beyond. We’re seeking talented professionals who are collaborative, curious, and committed to making a difference. - Engage with business and technical stakeholders to gather, analyze, document, define, and standardize business/data requirements (2%). - Perform or lead Gap Analysis between as-is and to-be systems to identify shortcomings (5%). - Develop and maintain documentation including requirement specifications, RTMs, acceptance criteria, process flows, and data mapping documents, using SharePoint and Confluence for version control and collaboration (2%). - Design, develop, and optimize complex data models and validation frameworks using SQL and Teradata (5%). - Perform/conduct comprehensive root cause analysis for data quality issues and develop/implement solutions or automated preventive controls (5%). - Represent the State/organization in interactions with the CMS federal team (alongside manager or as lead SME), addressing queries and serving as subject matter expert on CMS reporting standards (5%). - Oversee or ensure timely, accurate, error-free, and compliant submission of monthly extracts and quarterly CMS reports (claims, provider, eligibility, CMS-64.9 forms) (5%). - Lead enterprise level analytical initiatives supporting CMS reporting systems and Indiana healthcare programs (5%). - Conduct advanced data analysis, modeling, and automation using Excel (Power Query, Power Pivot, VBA macros, complex formulas, dynamic arrays, and scenario modeling) (10%). - Develop and maintain interactive dashboards and visualizations using Tableau and Power BI to identify trends, anomalies, compliance risks, and key performance metrics for executive and federal reporting (7%). - Define and implement enterprise data quality governance frameworks, KPIs, automated validation rules, and monitoring systems (5%). - Lead and coordinate complex testing/QA/UAT activities by establishing strategies, overseeing execution, and reviewing test cases prepared by junior analysts (5%). - Provide mentorship and technical leadership to junior analysts on advanced SQL, Informatica ETL, Power BI/Tableau development, and CMS processes (10%). - Recommend and drive implementation of process enhancements, tool upgrades, and automation initiatives for efficiency and compliance (10%). - Manage and analyze Treatment Episode Data Set (TEDS) Admission and Discharge data submissions for the Division of Mental Health and Addiction (DMHA) (4%). - Collaborate with the CMS State Support Team to analyze, document, and implement TMSIS (Transformed Medicaid Statistical Information System) updates and changes required by CMS (4%). - Research, interpret, and ensure application of HHS/CMS rules and regulations to maintain ongoing compliance with Indiana Medicaid/Medicare systems (7%). - Coordinate with the State Board of Accounts (SBOA) to design and deliver customized annual data extracts for audits and financial reporting, using automated workflows and secure transmission processes (4%). - Remote work requests will be considered consistent with company’s remote work policy. - Domestic travel required less than 2% of the time. Qualifications - Bachelor’s degree in Computer Science, Information Systems, Systems Engineering, or a related field, or foreign equivalent. - 7 years of relevant experience in data analysis, systems analysis, quality assurance, and CMS data reporting and compliance in the Business Systems Analyst or relevant position. Requirements - 7 years of experience with handling multiple projects/workstreams simultaneously. - 7 years of experience defining and executing test strategies, UAT, and QA methodologies. - 7 years of experience with Agile and Waterfall software development methodologies. - 7 years of experience using JIRA, Zephyr, and comparable tools for project tracking and test management. - 7 years of experience in SQL and Teradata for data extraction, modeling, and validation. - 7 years of experience developing data dashboards and analytical reports using Tableau and comparable BI tools. - 5 years of experience utilizing data quality frameworks and data governance best practices. - 5 years of experience with Medicaid including CMS reporting compliance experience. - Will accept any suitable combination of education, training, and experience. Benefits - At CSpring, we unlock the potential of people and data. - If you’re ready to lead meaningful work, collaborate with passionate teams, and grow your career in a people-first consulting environment — apply today!
• Develop advanced analyses, dashboards, analytical models, and data-driven intelligent solutions, turning information into actionable insights to support strategic decision-making; • Build, evolve, and maintain analytical solutions, automations, internal products, and data- and AI-driven applications, using AI-assisted development and tools such as Windsurf or similar; • Operate within an AI-driven operating model, using LLMs, Generative AI, prompt engineering, intelligent agents, and automations to accelerate analysis, code generation, documentation, investigations, and the continuous improvement of the team's solutions; • Work on data extraction, cleansing, integration, and preparation for analytical consumption and intelligent models, interacting with data pipelines, data architectures, and enterprise analytics environments; • Apply best practices in governance, versioning, MLOps, data quality, and data security, also contributing to the advancement of the team's analytical culture and the use of Artificial Intelligence; • Digital mindset, curiosity about new technologies, and an interest in understanding how Artificial Intelligence can transform processes, experiences, and outcomes.
Senior Data Analyst
Setton Industries Inc.Empowering Manufacturing Innovation through Engineering Excellence and Tailored Solutions
• Analyze performance data across a large and diverse client portfolio to identify trends, patterns, and opportunities • Own and evolve portfolio level performance reporting, including dashboards and scorecards • Build and maintain benchmarking frameworks to compare performance across key verticals • Define and standardize KPIs to ensure consistency across teams • Evaluate the effectiveness of strategies and product capabilities, identifying what drives the strongest outcomes • Conduct post launch analyses for product and strategy updates impacting performance • Support emerging areas like AI driven search such as AI assisted search visibility, entity coverage, and local signals as measurement capabilities evolve • Partner with Product, Data, and Engineering teams to improve data quality and expand access to meaningful signals • Translate complex data into clear narratives and recommendations for both technical and non technical stakeholders • Proactively surface insights and opportunities that influence product direction and business strategy
Senior Data Scientist, Data Analytics
BrillioTurning technological disruptions into the advantages. Let's create something Brillian(t) together!
• Perform deep data analysis to identify trends, anomalies, variances, and forecasting improvement opportunities. • Support senior data scientists with model inputs, backtesting, validation, and performance analysis. • Monitor forecast accuracy, bias, and overall health of forecasting solutions. • Analyze data quality issues and understand business drivers. • Build analytical datasets and reports to support forecasting and supply chain planning. • Develop dashboards and KPI reporting using Power BI or similar analytics platforms. • Collaborate with business teams and data scientists to improve decision-making. • Automate recurring analysis using SQL and Python. • Ensure strong data governance and reliable data availability.



