Newcode.ai logo
Newcode.ai

The AI for Forward-Thinking Firms

Head of Evaluations (Legal AI Benchmarking)

AI EngineerMachine Learning EngineerFull TimeRemoteLeadTeam 11-50Since 2021H1B No SponsorCompany SiteLinkedIn

Location

United States

Posted

3 days ago

Salary

0

Seniority

Lead

No structured requirement data.

Job Description

Head of Evaluations (Legal AI Benchmarking)

Newcode.ai

Role Description We are seeking a highly analytical professional with a strong statistical background to join our Head of Evaluations. In this role, you will design, implement, and scale the testing frameworks used to evaluate our platform. You will ensure our AI products meet the highest standards of legal reasoning, factual accuracy, and regulatory compliance while maintaining a near-zero hallucination rate. Key Responsibilities - Design Legal Benchmarks for: Contract Drafting, Information Extraction, Legal Research, and Contract Review - Build, source and maintain relevant datasets - Audit AI Output: Review and score complex AI-generated legal text, contract analyses, and statutory interpretations for accuracy and precision and lay out a strategy. - Define Evaluation Metrics: Establish clear criteria for grading model performance, specifically focusing on logical reasoning, citation accuracy, and the model's ability to safely abstain from answering. - Collaborate with Engineering: Partner directly with Engineering to translate legal errors into actionable technical feedback for model fine-tuning. Qualifications - PhD or Masters in statistics, mathematics, machine learning or equivalent - Analytical Skills: Proven ability to break down complex statutory frameworks and case law into structured, logical data points. - Tech-Savviness: python, panda, numpy, jupiter notebooks and similar statistical models

Related Job Pages

More AI Engineer Jobs

OpenRouter logo

Applied AI Engineer

OpenRouter

LLM router and marketplace

AI Engineer3 days ago
Full TimeRemoteTeam 1-10Since 2023H1B No Sponsor

• Build internal agentic workflows end-to-end on OpenRouter, from prototype to production, that automate support and GTM operations • Integrate our systems with the tools the company runs on, and steadily replace those tools where it makes sense to • Own the reliability of what you ship: evals, guardrails, and monitoring, so automation is trustworthy rather than flashy • Drive measurable impact — deflect support volume, accelerate GTM motions, cut manual toil — and prove it with data • Work directly with support, GTM, and engineering to find the highest-leverage workflows to automate next • Iterate on real usage: ship, measure, improve

United States
Full TimeRemoteTeam 10,001+H1B Sponsor

• Definir arquitetura e padrões para soluções conversacionais; • Liderar a evolução dos assistentes virtuais da empresa; • Apoiar tecnicamente times multidisciplinares; • Conduzir análises de desempenho e propor melhorias contínuas; • Atuar como referência em NLU, NLP, GenAI e automação conversacional; • Participar da implantação e da estratégia de adoção do IBM watsonx Orchestrate; • Avaliar novas funcionalidades da plataforma IBM watsonx e definir roadmap de evolução; • Garantir qualidade, escalabilidade, observabilidade e governança das soluções implementadas.

Brazil
Job Closed
Truelogic Software logo

Senior AI/ML Engineer – Technology

Truelogic Software

Premium boutique software development company that helps brands with big ideas to make a difference in people’s lives.

AI Engineer3 days ago
Full TimeRemoteTeam 501-1,000Since 2004H1B No Sponsor

• Design and execute advanced AI/ML models – including fine-tuned LLMs and RAG architectures – for integration into both internal tooling and external client-facing products. • Lead experimentation efforts: clearly define hypotheses, design interpretable tests, analyze results rigorously, and communicate findings effectively. • Serve as the technical SME for GenAI/LLM-powered tools, from foundational architecture to model optimization and deployment. • Work with software and data engineering to productionize models and contribute to robust CI/CD pipelines with MLOps/LLMOps best practices. • Stay current on emerging methods in AI – e.g., agentic workflows, memory-augmented systems, multimodal embeddings – and bring those ideas to life in real-world applications. • Promote AI adoption internally by educating and empowering cross-functional partners while improving processes and workflows. • Manage your time and projects effectively, ensuring clear progress updates and proactive communication with collaborators. • Write production-grade Python code to support models, pipelines, and experimental frameworks. • Work with the highest degree of ethics and empathy, recognizing that AI can be a potentially disruptive force for organizations and individuals

Dominican Republic
Job Closed
Truelogic Software logo

Senior AI/ML Engineer – Technology

Truelogic Software

Premium boutique software development company that helps brands with big ideas to make a difference in people’s lives.

AI Engineer3 days ago
Full TimeRemoteTeam 501-1,000Since 2004H1B No Sponsor

• Design and execute advanced AI/ML models – including fine-tuned LLMs and RAG architectures – for integration into both internal tooling and external client-facing products. • Lead experimentation efforts: clearly define hypotheses, design interpretable tests, analyze results rigorously, and communicate findings effectively. • Serve as the technical SME for GenAI/LLM-powered tools, from foundational architecture to model optimization and deployment. • Work with software and data engineering to productionize models and contribute to robust CI/CD pipelines with MLOps/LLMOps best practices. • Stay current on emerging methods in AI – e.g., agentic workflows, memory-augmented systems, multimodal embeddings – and bring those ideas to life in real-world applications. • Promote AI adoption internally by educating and empowering cross-functional partners while improving processes and workflows. • Manage your time and projects effectively, ensuring clear progress updates and proactive communication with collaborators. • Write production-grade Python code to support models, pipelines, and experimental frameworks. • Work with the highest degree of ethics and empathy, recognizing that AI can be a potentially disruptive force for organizations and individuals.

Brazil
Job Closed