Research institute investigating the trajectory of AI
Researcher – Benchmark Reviews
Location
United States
Posted
7 days ago
Salary
$100K - $200K / year
Seniority
Senior
Job Description
Researcher – Benchmark Reviews
Epoch AI
• Review benchmarks: Assess a new benchmark at least every two weeks. You’ll evaluate their methodology and what a 'good performance' would imply about AI capabilities. • Publish and maintain reports: Write up public-facing reports on new benchmarks and periodically update them as new versions release and models make progress. • Dig into data: Examine individual tasks within benchmarks in detail, using coding agents to assist while retaining your own critical oversight.
Job Requirements
- You excel at written communication. Your first drafts rarely need much polishing; they’re clear and publication-quality with minimal revision.
- You think critically about research methodologies. You have strong opinions about what benchmarks and evaluations should look like, and can recognize when something seems wrong.
- You’re already familiar with existing AI benchmarks, their methodologies, strengths, and weaknesses. You have ideas for new benchmarks that don’t exist yet, but should.
- You’re comfortable working with dense data. You’ll often be breaking down benchmarks into their component parts, which can be overwhelming without familiarity.
- Experience writing about AI/ML for a public or research audience is nice to have.
- Hands-on experience running or building AI evaluations is nice to have.
Benefits
- Annual salary between $100,000 and $200,000 USD, depending on location, seniority, and experience.
- Salaries are not restricted to USD, and contracts and payments are usually in local currencies.
- Fully remote environment, including flexible work hours.
- Competitive global benefits program, including a comprehensive health insurance program—including supplemental benefits specific to a local country, as available and mandated by local law—and life insurance and a pension plan, if applicable in your country.
- Generous paid time off (PTO), including no specific annual limit, with 30 days PTO per year protected, unlimited personal and sick leave, and 4 months paid parental leave for permanent staff with at least 12 months of tenure.
- A flexible and generous expense policy for you to spend on equipment and a large range of productivity tools or learning/development opportunities, including unlimited spending on AI tools, subject to regulations and manager approval.
- Paid work trips, including 3 staff retreats per year and relevant conferences.
- Access to our very well-equipped offices in Berkeley, California, including paid meals, snacks, gym, and more.
Related Guides
Related Categories
Related Job Pages
More Researcher Jobs
• Own end-to-end options strategy research: hypothesis → data → modeling → backtesting → production → live monitoring and iteration • Work on Relative Value, Statistical Arbitrage, and Spread Trading strategies specific to the options universe (the stack above) • Build and own the volatility fitter the signals sit on – calibrating arbitrage-free, temporally stable surfaces (SVI/SSVI or a proposed alternative) on realistic data (wide bid/ask, missing strikes, gaps, latency), with attention to residual noise near expiry / illiquid strikes / events • Translate strategy output into execution – routing a target delta-order across option legs to minimize Greek risk, with inventory-aware quoting that shifts price/size against live vega/gamma/skew, and awareness of options microstructure (spreads, queue, adverse selection, latency) • Build and maintain mid-frequency (MFT), fully automated strategies with a strong live-performance focus • Track record of deploying fully automated strategies with Sharpe > 2 (or demonstrable equivalent risk-adjusted performance) • Design robust signal research pipelines (feature engineering, labeling, validation, regime analysis) • Develop realistic backtests and live-simulation frameworks accounting for slippage, spreads, latency, partial fills, and market impact • Work in tight feedback loops with trading and execution to improve PnL, robustness, and risk-adjusted performance • Debug and tune research outputs under live conditions: data issues, execution artifacts, microstructure noise, and changing market regimes
• Conduct in-depth research on Mantle, its ecosystem partners, and broader crypto, AI, and financial markets • Identify emerging trends early and translate them into actionable insights • Produce narrative-driven research tailored for crypto-native audiences • Represent Mantle at industry events and convert interactions into relationships • Leverage networks across crypto, AI, and developer ecosystems to expand presence
• Support projects related to the education of students with disabilities, with emphasis on state education agencies and special education systems-level work. • Assist with reviewing, synthesizing, and organizing research, policy, guidance, and other resources relevant to special education leadership and improvement efforts. • Contribute to the development of tools, materials, presentations, briefs, and other products for internal and external audiences. • Participate in team meetings, professional learning opportunities, and mentorship activities with leaders in special education and related fields. • Help analyze problems of practice and identify evidence-based strategies that support improved services, systems, and outcomes for children and youth with disabilities. • Collaborate with project staff to complete assigned tasks, monitor timelines, and communicate progress on internship activities. • Complete a culminating project aligned with the priorities of NCSI. • Perform other related duties in support of learning goals and organizational needs.
• Create and curate an evaluation suite. Find real-world tasks that serve as challenging tests for practical AI capabilities, and update the tasks over time as AI capabilities evolve. Devise rubrics for evaluating AI performance. - Evaluate AI systems. Regularly evaluate new, notable AI models and products on the task suite. Update tasks and rubrics to reflect the changing landscape of AI capabilities. - Communicate your research. Create public-facing reports, blog posts, and data visualizations with your observations. Ensure the evaluations feed into our other research topics and help keep our team informed. - Conduct data analysis. Analyze evaluation results and compare models across tasks. - Improve the process. You might automate parts of the workflow, and build out parts of the evaluation into standalone benchmarks.
