AI That Elevates the Impact of ALL Responders
AI Evaluation Engineer
Location
Colorado
Posted
3 days ago
Salary
$120K - $170K / year
Seniority
Senior
Job Description
AI Evaluation Engineer
GovWorx
• Design, build, and maintain automated AI evaluation pipelines for production LLM applications • Develop prompt engineering strategies and iterate on prompts and compare LLMs using quantitative evaluation methods • Build offline evaluation datasets and regression testing frameworks to measure AI performance over time • Analyze production AI behavior using Python, SQL, and statistical techniques to identify opportunities for improvement • Design experiments, A/B tests, and benchmarking methodologies for evaluating prompt and model changes • Develop dashboards and reporting that communicate AI quality, reliability, and performance metrics • Partner with engineering and product teams to safely deploy and monitor improvements to production AI systems • Investigate model failures through detailed error analysis and recommend improvements to prompts, evaluation datasets, and workflows • Help establish best practices for Responsible AI, evaluation methodologies, and continuous model improvement
Job Requirements
- Must have US citizenship and pass FBI fingerprint and background check in multiple states
- 3+ years of experience in software engineering, machine learning, data science, or a related technical field
- Experience designing evaluation metrics and interpreting AI model performance
- Understanding of statistical methods including hypothesis testing and experiment design
- Strong Python development experience
- Strong SQL skills with experience analyzing large datasets
- Experience building or supporting production LLM or Generative AI applications
- Experience with prompt engineering and systematic prompt evaluation
Benefits
- Offers Bonus
- Benefits (Health, Dental, Vision)
- Flexible Time Off
Related Guides
Related Job Pages
More AI Engineer Jobs
• You bring deep expertise in knowledge representation, semantic systems, and applied AI, and you are energized by translating complex enterprise domains into structured, machine-understandable models. • You are comfortable working through ambiguity and building early systems that demonstrate clear value. • You own the enterprise knowledge model including entities, relationships, actions, constraints, and how they evolve over time. • You define and govern ontology standards, ensuring clear layering, reuse, and consistency across systems. • You provide technical leadership while remaining hands-on in semantic architecture and knowledge graph development. • You design and evolve enterprise knowledge graph platforms that integrate structured, semi-structured, and unstructured data, enabling reasoning, inference, and contextual retrieval. • You design the underlying data and graph architecture, including ingestion pipelines, transformation and mapping, entity resolution, schema alignment, validation, and both batch and streaming updates. • You guide key technical decisions across ontology design, graph architecture, and AI integration, including trade-offs between materialized inference, constraint validation, and query-time reasoning at scale. • You partner closely with product and engineering to ensure models are practical, scalable, and aligned to real-world use cases. • You will mentor others and help build a culture of structured thinking, semantic clarity and innovation.
• You translate requirements from business units into concrete, implementable AI solutions, working closely with Business, Security & Engineering. • You take use cases from the initial idea to production — including implementation, alignment, documentation and transparency. • You integrate AI / LLMs and agents into production workflows and contribute to building a central AI infrastructure/platform. • You work hands-on with our enterprise AI tools (Claude Enterprise, ChatGPT Enterprise) and help to further develop them and improve usability. • You build and maintain workflows and automations in n8n and support teams in using them independently. • You evaluate and introduce new AI tools, consolidate existing ones, and develop standards and governance together with Security (e.g., for n8n).
ML Research Engineer, AI for Life Sciences
SandboxAQLeveraging AQ - the powerful compound effects of AI + Quantum technology
• Bring novel ideas and the content of scientific papers into high-performing and robust scientific code. • Lead the ideation, benchmarking, and execution of complex datasets and ML models, ensuring seamless integration into large-scale simulation frameworks. • Drive software through the entire product lifecycle—from foundational research and implementation to launch and long-term support—ensuring technical excellence at every stage.
Senior AI Innovation Engineer, Fluent Ukrainian, English
SupportYourAppSupport-as-a-Service that helps companies scale faster by taking care of their customers’ needs.
• Take ownership of the existing AI transformation strategy and drive its execution at pace; • Identify gaps, dependencies, and bottlenecks, resolving them proactively; • Continuously refine strategy based on market feedback, client needs, and real-world learnings. • Contribute to building AI-powered service offerings beyond headcount-based models; • Prototype, test, and scale solutions from proof-of-concept to client deployment; • Define reusable AI components (knowledge bases, copilots, QA automation, intelligent routing). • Own revenue targets for innovation-driven offerings; • Partner with Sales and Business Growth teams to position AI solutions in client conversations; • Develop pricing and packaging strategies for hybrid AI + human service models. • Build repeatable frameworks for evaluating, piloting, and scaling new solutions; • Establish workflows, governance, and reporting for the innovation function; • Define and track success metrics (e.g., speed to pilot, adoption, cost efficiency, revenue impact). • Drive implementation of AI agents into internal workflows; • Partner with internal teams to identify automation opportunities that improve efficiency and margins.




