GovWorx logo
GovWorx

AI That Elevates the Impact of ALL Responders

AI Evaluation Engineer

AI EngineerMachine Learning EngineerFull TimeRemoteSeniorTeam 11-50Since 2023H1B No SponsorCompany SiteLinkedIn

Location

Colorado

Posted

3 days ago

Salary

$120K - $170K / year

Seniority

Senior

Bachelor Degree3 yrs expEnglishPythonSQL

Job Description

AI Evaluation Engineer

GovWorx

• Design, build, and maintain automated AI evaluation pipelines for production LLM applications • Develop prompt engineering strategies and iterate on prompts and compare LLMs using quantitative evaluation methods • Build offline evaluation datasets and regression testing frameworks to measure AI performance over time • Analyze production AI behavior using Python, SQL, and statistical techniques to identify opportunities for improvement • Design experiments, A/B tests, and benchmarking methodologies for evaluating prompt and model changes • Develop dashboards and reporting that communicate AI quality, reliability, and performance metrics • Partner with engineering and product teams to safely deploy and monitor improvements to production AI systems • Investigate model failures through detailed error analysis and recommend improvements to prompts, evaluation datasets, and workflows • Help establish best practices for Responsible AI, evaluation methodologies, and continuous model improvement

Job Requirements

  • Must have US citizenship and pass FBI fingerprint and background check in multiple states
  • 3+ years of experience in software engineering, machine learning, data science, or a related technical field
  • Experience designing evaluation metrics and interpreting AI model performance
  • Understanding of statistical methods including hypothesis testing and experiment design
  • Strong Python development experience
  • Strong SQL skills with experience analyzing large datasets
  • Experience building or supporting production LLM or Generative AI applications
  • Experience with prompt engineering and systematic prompt evaluation

Benefits

  • Offers Bonus
  • Benefits (Health, Dental, Vision)
  • Flexible Time Off

Related Job Pages

More AI Engineer Jobs

Full TimeRemoteTeam 1,001-5,000H1B Sponsor

• You bring deep expertise in knowledge representation, semantic systems, and applied AI, and you are energized by translating complex enterprise domains into structured, machine-understandable models. • You are comfortable working through ambiguity and building early systems that demonstrate clear value. • You own the enterprise knowledge model including entities, relationships, actions, constraints, and how they evolve over time. • You define and govern ontology standards, ensuring clear layering, reuse, and consistency across systems. • You provide technical leadership while remaining hands-on in semantic architecture and knowledge graph development. • You design and evolve enterprise knowledge graph platforms that integrate structured, semi-structured, and unstructured data, enabling reasoning, inference, and contextual retrieval. • You design the underlying data and graph architecture, including ingestion pipelines, transformation and mapping, entity resolution, schema alignment, validation, and both batch and streaming updates. • You guide key technical decisions across ontology design, graph architecture, and AI integration, including trade-offs between materialized inference, constraint validation, and query-time reasoning at scale. • You partner closely with product and engineering to ensure models are practical, scalable, and aligned to real-world use cases. • You will mentor others and help build a culture of structured thinking, semantic clarity and innovation.

Canada
The Quality Group logo

AI Engineer, All Genders

The Quality Group

Marken: ESN, More Nutrition, Foodist

AI Engineer3 days ago
Full TimeRemoteTeam 1,001-5,000Since 2021H1B No Sponsor

• You translate requirements from business units into concrete, implementable AI solutions, working closely with Business, Security & Engineering. • You take use cases from the initial idea to production — including implementation, alignment, documentation and transparency. • You integrate AI / LLMs and agents into production workflows and contribute to building a central AI infrastructure/platform. • You work hands-on with our enterprise AI tools (Claude Enterprise, ChatGPT Enterprise) and help to further develop them and improve usability. • You build and maintain workflows and automations in n8n and support teams in using them independently. • You evaluate and introduce new AI tools, consolidate existing ones, and develop standards and governance together with Security (e.g., for n8n).

Germany
SandboxAQ logo

ML Research Engineer, AI for Life Sciences

SandboxAQ

Leveraging AQ - the powerful compound effects of AI + Quantum technology

AI Engineer3 days ago
Full TimeRemoteTeam 51-200Since 2021H1B No Sponsor

• Bring novel ideas and the content of scientific papers into high-performing and robust scientific code. • Lead the ideation, benchmarking, and execution of complex datasets and ML models, ensuring seamless integration into large-scale simulation frameworks. • Drive software through the entire product lifecycle—from foundational research and implementation to launch and long-term support—ensuring technical excellence at every stage.

Canada
CA$125.8K - CA$222K / year
SupportYourApp logo

Senior AI Innovation Engineer, Fluent Ukrainian, English

SupportYourApp

Support-as-a-Service that helps companies scale faster by taking care of their customers’ needs.

AI Engineer3 days ago
ContractRemoteTeam 1,001-5,000H1B No Sponsor

• Take ownership of the existing AI transformation strategy and drive its execution at pace; • Identify gaps, dependencies, and bottlenecks, resolving them proactively; • Continuously refine strategy based on market feedback, client needs, and real-world learnings. • Contribute to building AI-powered service offerings beyond headcount-based models; • Prototype, test, and scale solutions from proof-of-concept to client deployment; • Define reusable AI components (knowledge bases, copilots, QA automation, intelligent routing). • Own revenue targets for innovation-driven offerings; • Partner with Sales and Business Growth teams to position AI solutions in client conversations; • Develop pricing and packaging strategies for hybrid AI + human service models. • Build repeatable frameworks for evaluating, piloting, and scaling new solutions; • Establish workflows, governance, and reporting for the innovation function; • Define and track success metrics (e.g., speed to pilot, adoption, cost efficiency, revenue impact). • Drive implementation of AI agents into internal workflows; • Partner with internal teams to identify automation opportunities that improve efficiency and margins.

Georgia