ML Engineer - LLM Evaluation & Automation

Location

United States

Posted

4 days ago

Salary

0

Seniority

Mid Level

No structured requirement data.

Job Description

ML Engineer - LLM Evaluation & Automation

Grid Dynamics Holdings

Role Description We are seeking a highly skilled Machine Learning Engineer who specializes in leveraging Large Language Models (LLMs) for automated evaluation and quality assessment. In this role, you will design and build systems that automatically measure and improve the accuracy, relevance, and consistency of model outputs. You will lead initiatives to create evaluation pipelines, develop metrics, and deliver actionable insights for continuous improvements. This position requires strong technical expertise, analytical problem-solving abilities, and the capacity to manage projects across multiple cross-functional teams. - Design and implement automated systems and pipelines for evaluating LLM outputs. - Develop metrics and KPIs to measure output quality, accuracy, and consistency using LLM-based evaluations. - Collaborate with Engineering teams to create automated logic checks and validation tools. - Partner with Data Scientists to analyze evaluation results and optimize prompt and task structures. - Provide feedback loops to ensure evaluation guidelines align with LLM-based assessments. - Investigate how LLM-derived evaluations can enhance product reliability and user experience. - Recommend refinements to prompt engineering, evaluation strategies, and automation tools. - Stay informed on emerging trends in LLM evaluation, automated quality assessment, and AI toolchains. - Continuously improve and expand automated evaluation processes based on industry best practices. Qualifications - 5+ years of experience in ML engineering, NLP, or AI/ML automation. - Hands-on experience in prompt engineering and designing LLM-based evaluation systems is preferred. - Strong understanding of machine learning principles with focus on NLP and advanced LLM capabilities (e.g., Chain-of-Thought, agentic workflows). - Expertise in building automated evaluation or QA pipelines. - Excellent analytical and problem-solving skills with experience in root cause and error pattern analysis. - Proven project management and cross-functional collaboration experience. - Excellent communication skills to convey complex insights to technical and non-technical audiences. - Detail-oriented mindset with a focus on evaluation metrics, prompt design, and automation. - Ability to quickly adapt to new business rules and evaluation guidelines across diverse product domains. - Strong programming skills in Python and SQL. - Experience with big data technologies like PySpark for data aggregation and sampling is a strong plus. - Bachelor's/Master's degree in Computer Science/Engineering or a related field. Benefits - Opportunity to work on cutting-edge projects. - Work with a highly motivated and dedicated team. - Competitive salary. - Flexible schedule. - Benefits package - medical insurance, vision, dental, etc. - Corporate social events. - Professional development opportunities. - Well-equipped office.

Related Categories

Related Job Pages

More QA Automation Engineer Jobs

QuickTeam logo

AI Automation Engineer – AI Implementer

QuickTeam

We provide businesses with virtual employees to help them save time and money!

Full TimeRemoteTeam 1-10H1B No Sponsor

• Build and connect AI agents and automations across the company • Set up outbound and growth systems: lead scraping, enrichment, and automated engagement — e.g. agents that monitor Reddit and forums for people looking to hire and engage them • Stand up a central "company memory" with specialized sub-agents that feed off it • Continuously evaluate new AI tools and frameworks, run experiments, and ship what works • Own automations end-to-end, from idea to working system

Dominican Republic
QuickTeam logo

AI Automation Engineer – AI Implementer

QuickTeam

We provide businesses with virtual employees to help them save time and money!

Full TimeRemoteTeam 1-10H1B No Sponsor

• Build and connect AI agents and automations across the company • Set up outbound and growth systems: lead scraping, enrichment, and automated engagement — e.g. agents that monitor Reddit and forums for people looking to hire and engage them • Stand up a central "company memory" with specialized sub-agents that feed off it • Continuously evaluate new AI tools and frameworks, run experiments, and ship what works • Own automations end-to-end, from idea to working system

Ethiopia

DevSecOps Engineer

Flexion Inc

At Flexion, our company culture is built on autonomy, trust, and transparency. We empower teams to remain autonomous and self-directed by hiring people who can solve complex problems collaboratively. Every member within a cross-functional team is a leader who takes responsibility for the entire team’s success, which mirrors the company’s overall approach of low hierarchy. Flexion team members are expected to help each other grow, challenge ideas respectfully, and integrate feedback as a normal part of the workday.

Role Description Flexion is looking for a DevSecOps Engineer for an enterprise-level application development project to help accelerate trusted AI adoption across government. In this role, you must possess excellent problem-solving skills and deep technical knowledge, with a strong aptitude for understanding the focus, scope, and impact of your work in a broader, forward-looking software landscape. We are looking for a Platform DevSecOps Engineer strong in infrastructure, cloud architecture, and security. This is not primarily an application or UI role, but you should be comfortable navigating our Python and TypeScript stack. Your center of gravity is infrastructure, identity, and secure operations. - Possess excellent problem-solving skills and technical knowledge. - Use test-driven development with full code coverage. - Engage in fully transparent, open source development. Qualifications - 10+ years of engineering experience, with depth in infrastructure, cloud, or platform work. - Strong AWS and cloud-architecture experience. - Extensive FedRAMP experience. - Infrastructure as code, particularly Terraform and Helm. - CI/CD with GitHub Actions and Jenkins. - Containerization (Docker) and orchestration (Kubernetes). - Identity and access management: SSO/OIDC, SAML, SCIM provisioning; Keycloak and/or enterprise ICAM a strong plus. - Expertise in observability (logging, monitoring, alerting). - Able to work effectively across Python and TypeScript codebases, with FastAPI, Astro, or SvelteKit experience a plus. - Test-driven development, automated unit testing, and end-to-end testing. - Comfortable using AI coding agents as force multipliers and structuring repos for them. - Cares about crafting thoughtfully-designed developer experiences. - Superb time-management, communication, and problem-solving skills. - Bachelor's degree and/or equivalent experience. Requirements - Speak openly and honestly with your colleagues and clients about problems and proposed solutions. - Welcome and handle changing requirements and priorities with little or no warning. - Collaborate online in small groups about 50% of the time. - Encourage simple and minimal solutions that keep options open. - Expect and vocally advocate for quality first. - Learn new practices and techniques as the situation demands. - Ensure the teams develop demonstrable software every week or two. - Do what needs to be done to deliver the product or project without ego or attitude. - Dig deep to find the root causes of problems so we can create the right solutions. - Relentlessly improve yourself, your team, and your processes. - Offer and welcome constructive feedback as a tool for continuous team and individual growth. - Embrace AI assistants (e.g., ChatGPT, NotebookLM) as force multipliers to accelerate innovation, iterate faster, and maintain high-quality output. Benefits - Comprehensive health, dental, and vision coverage. - Life and disability insurance. - Flexible spending accounts. - 401(k) plan. - Generous paid time off. - Referral bonuses. - Commitment to well-being and career success. Company Description At Flexion, our company culture is built on autonomy, trust, and transparency. We empower teams to remain autonomous and self-directed by hiring people who can solve complex problems collaboratively. Every member within a cross-functional team is a leader who takes responsibility for the entire team’s success, which mirrors the company’s overall approach of low hierarchy. Flexion team members are expected to help each other grow, challenge ideas respectfully, and integrate feedback as a normal part of the workday.

United States
$125K - $160K / year
Full TimeRemoteTeam 501-1,000H1B No Sponsor

• Responsible for designing, developing, documenting and supporting enterprise automation solutions for STCU using automation tools and modern program language. • Works with other members of an Agile/SCRUM team to create and construct automation solutions and performs Tier I support for maintenance, troubleshooting, automation, and scripting. • Breakdown assembled work order specifications into manageable tasks for Agile/SCRUM solution development by working with product, business, and application owners to understand business needs and requirements. • Develop foundational skills/ knowledge across all automation disciplines by using resources and initiative to assist with design, development, testing, deployment and support of workflow solutions, forms administration, eSign, and job automation, independently and integrated. • Automate tasks and processes for efficiency using automation software, PowerShell and other methods. • Contribute to operational efficiencies, accountability, and error reduction by designing, developing, configuring and implementing automation solutions, leveraging expertise to produce solutions that enable global consistency, standardization, optimized performance, and minimal technical debt. • Successfully deploy new automations by adhering to current Automation Development Lifecycle (ADLC) standards. • Take an active ownership role in the security and protection of member and organizational data/records.

Washington
$40 - $48 / hour