Job Closed

This listing is no longer active.

Gramian Consulting logo
Gramian Consulting

We get talents. You get results.

AI Evaluation Engineer – Agentic Coding, Software Engineering

Full-stack EngineerSoftware EngineerContractRemoteSeniorTeam 2-10Since 2025H1B No SponsorCompany SiteLinkedIn

Location

Colombia

Posted

85 days ago

Salary

0

Seniority

Senior

Job Description

AI Evaluation Engineer – Agentic Coding, Software Engineering

Gramian Consulting

• Execute coding tasks within **agentic coding environments**, maintaining strict evaluation protocols • Review and evaluate **model-generated code trajectories** for correctness and completeness • Validate outputs by reading code, running tests, analyzing logs, and inspecting artifacts • Perform targeted validation using **scripts, tests, and manual checks** • Write **clear, evidence-based rationales** for evaluations and rankings • Design realistic, multi-step **coding tasks and workflows** (offline work) • Create and refine **evaluation rubrics and scoring criteria** • Ensure consistency, quality, and compliance across evaluations • Identify issues in environments, instructions, or workflows and report with clear evidence

Job Requirements

  • 5+ years of experience in **software engineering, QA, developer tooling, or similar code-heavy roles**
  • Strong proficiency in at least one programming ecosystem (e.g., Python, JavaScript/TypeScript, Java, C/C++, Rust, SQL)
  • Ability to **read and understand unfamiliar codebases** and implement/debug changes
  • Experience running and interpreting **tests, scripts, and CLI tools**
  • Strong debugging and problem-solving skills, including handling edge cases
  • Comfortable working in **Linux/terminal environments**
  • Familiarity with **Git workflows** and standard development tooling
  • Experience with **AI coding tools or agentic coding environments** (e.g., Cursor, Claude Code, or similar)
  • Strong attention to detail and ability to produce **consistent, high-quality evaluations**

Related Job Pages

More Full-stack Engineer Jobs

Findem logo

Senior Staff Full Stack Engineer – Agentic

Findem

Talent acquisition powered by AI from search to hire.

Full TimeRemoteTeam 51-200H1B Sponsor

• Build AI-native products that reshape how HR teams source, hire, and operate. • Design, build, and orchestrate AI agents and agentic workflows to accelerate product delivery. • Ship features end-to-end, from customer insight to production-ready experience, using AI-augmented development. • Prototype rapidly with coding co-pilots and agentic tooling, validate with customers, and iterate based on real-world feedback. • Architect and evolve systems to support performance, reliability, and scale, designing for agent-assisted operation from the start. • Turn ambiguous opportunities into clear execution plans and deliver measurable outcomes. • Multiply team output by sharing agentic workflows, elevating engineering practices, and raising the bar for speed and quality across the team.

Canada
$200K - $220K / year
Cribl logo

Software Engineer – Core Platform

Cribl

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy.

Full TimeRemoteTeam 501-1,000Since 2017H1B Sponsor

• Develop software for projects and features with an emphasis on backend systems and APIs responsible for ingesting, processing, and routing data • Design, develop, test, and maintain clear, concise, and robust code that produces the desired outcomes for our customers • Partner with a cross functional team of engineers, designers, and product managers to translate feature specifications into product designs and implementable code • Ensure product features are working as expected by creating rich test plans paired with comprehensive automated tests • Have end-to-end ownership of the software you develop, regularly participating in your team’s on-call/support rotation • Be a driver, take the initiative to help the larger team reach desired outcomes even if it’s outside your job description • This position will require stand-by, on-call, or off-hours duties (if on call IS required)

California
$140K - $170K / year
Kaseya logo

Staff Software Engineer

Kaseya

Kaseya® is the leading provider of IT and security management solutions for managed service providers (MSPs) and SMBs.

Full TimeRemoteTeam 1,001-5,000H1B Sponsor

• Ship high-quality data/ML/AI powered features end-to-end • Integrate data/ML/AI capabilities into real customer workflows • Raise the bar on design, reliability, performance, and engineering practices across the org • Own the architecture and technical direction for key product areas in a multi-tenant SaaS platform. • Work with product, design, and data/ML teams to translate business problems into simple, robust technical solutions. • Drive the evolution of our system architecture (APIs, services, data flows, auth, tenancy, integrations) as the product and customer base scale. • Build and maintain backend services and APIs (REST/GraphQL/gRPC) that power those experiences. • Deliver backend features end-to-end including data schemas and business logic. • Collaborate with UX and product to ensure responsive and delightful product experiences. • Partner with data scientists and MLOps/platform teams to embed data, ML and AI capabilities into the product (recommendations, categorization, automation, routing, insights, LLM-powered workflows, etc.). • Design APIs, data contracts, and UX flows that make ML/AI features reliable, understandable, and safe for customers. • Ensure telemetry and feedback loops are in place so data/ML teams can measure performance, iterate models, and improve outcomes. • Help define and implement guardrails for AI features (fallbacks, explanations, error handling, permissions). • Champion operability: monitoring, alerting, logging, and runbooks for services you own. • Lead efforts to improve performance, scalability, and resilience of critical paths (e.g., onboarding, reporting, AI-assisted workflows). • Work with security and compliance to ensure features meet requirements around authentication, authorization, data privacy, and multi-tenancy. • Participate in and help evolve on-call/incident response processes as a technical leader. • Act as a technical mentor for multiple teams, raising the bar on code quality, reviews, testing, and design. • Lead technical design reviews and cross-team architecture discussions, especially where product, data, and ML intersect. • Help define engineering standards and best practices (API design, frontend patterns, error handling, observability, testing).

India
Job Closed
Google Fiber logo

Analytical Engineering Lead

Google Fiber

All internet should be this simple.

Full TimeRemoteTeam 501-1,000Since 2010H1B No Sponsor

• Maintain, add to, and improve foundational scripts and pipelines that join raw data from Engineering, setting analysts up for success to develop reports, visuals, and insights. • Provide technical leadership on Looker/GCP deployment architecture, development guidelines, data delivery layer, optimization efforts, access configurations, and AI usage. • Perform data modeling across multiple business domains to unlock efficient analytical problem-solving and deep dives - working with large, complex data sets. • Collaborate with a wide variety of cross-functional partners, including senior management, to understand business needs, formulate, and complete end-to-end data intelligence infrastructure. • Manage the team workload and bandwidth, communicating expected quarterly deliverables, adhoc asks, and updating timelines.

United States
$157.6K - $231.4K / year
Job Closed