Job Closed
This listing is no longer active.
We get talents. You get results.
AI Evaluation Engineer – Agentic Coding, Software Engineering
Location
Colombia
Posted
85 days ago
Salary
0
Seniority
Senior
Job Description
AI Evaluation Engineer – Agentic Coding, Software Engineering
Gramian Consulting
• Execute coding tasks within **agentic coding environments**, maintaining strict evaluation protocols • Review and evaluate **model-generated code trajectories** for correctness and completeness • Validate outputs by reading code, running tests, analyzing logs, and inspecting artifacts • Perform targeted validation using **scripts, tests, and manual checks** • Write **clear, evidence-based rationales** for evaluations and rankings • Design realistic, multi-step **coding tasks and workflows** (offline work) • Create and refine **evaluation rubrics and scoring criteria** • Ensure consistency, quality, and compliance across evaluations • Identify issues in environments, instructions, or workflows and report with clear evidence
Job Requirements
- 5+ years of experience in **software engineering, QA, developer tooling, or similar code-heavy roles**
- Strong proficiency in at least one programming ecosystem (e.g., Python, JavaScript/TypeScript, Java, C/C++, Rust, SQL)
- Ability to **read and understand unfamiliar codebases** and implement/debug changes
- Experience running and interpreting **tests, scripts, and CLI tools**
- Strong debugging and problem-solving skills, including handling edge cases
- Comfortable working in **Linux/terminal environments**
- Familiarity with **Git workflows** and standard development tooling
- Experience with **AI coding tools or agentic coding environments** (e.g., Cursor, Claude Code, or similar)
- Strong attention to detail and ability to produce **consistent, high-quality evaluations**
Related Guides
Related Job Pages
More Full-stack Engineer Jobs
Senior Staff Full Stack Engineer – Agentic
FindemTalent acquisition powered by AI from search to hire.
• Build AI-native products that reshape how HR teams source, hire, and operate. • Design, build, and orchestrate AI agents and agentic workflows to accelerate product delivery. • Ship features end-to-end, from customer insight to production-ready experience, using AI-augmented development. • Prototype rapidly with coding co-pilots and agentic tooling, validate with customers, and iterate based on real-world feedback. • Architect and evolve systems to support performance, reliability, and scale, designing for agent-assisted operation from the start. • Turn ambiguous opportunities into clear execution plans and deliver measurable outcomes. • Multiply team output by sharing agentic workflows, elevating engineering practices, and raising the bar for speed and quality across the team.
Software Engineer – Core Platform
CriblCribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy.
• Develop software for projects and features with an emphasis on backend systems and APIs responsible for ingesting, processing, and routing data • Design, develop, test, and maintain clear, concise, and robust code that produces the desired outcomes for our customers • Partner with a cross functional team of engineers, designers, and product managers to translate feature specifications into product designs and implementable code • Ensure product features are working as expected by creating rich test plans paired with comprehensive automated tests • Have end-to-end ownership of the software you develop, regularly participating in your team’s on-call/support rotation • Be a driver, take the initiative to help the larger team reach desired outcomes even if it’s outside your job description • This position will require stand-by, on-call, or off-hours duties (if on call IS required)
Staff Software Engineer
KaseyaKaseya® is the leading provider of IT and security management solutions for managed service providers (MSPs) and SMBs.
• Ship high-quality data/ML/AI powered features end-to-end • Integrate data/ML/AI capabilities into real customer workflows • Raise the bar on design, reliability, performance, and engineering practices across the org • Own the architecture and technical direction for key product areas in a multi-tenant SaaS platform. • Work with product, design, and data/ML teams to translate business problems into simple, robust technical solutions. • Drive the evolution of our system architecture (APIs, services, data flows, auth, tenancy, integrations) as the product and customer base scale. • Build and maintain backend services and APIs (REST/GraphQL/gRPC) that power those experiences. • Deliver backend features end-to-end including data schemas and business logic. • Collaborate with UX and product to ensure responsive and delightful product experiences. • Partner with data scientists and MLOps/platform teams to embed data, ML and AI capabilities into the product (recommendations, categorization, automation, routing, insights, LLM-powered workflows, etc.). • Design APIs, data contracts, and UX flows that make ML/AI features reliable, understandable, and safe for customers. • Ensure telemetry and feedback loops are in place so data/ML teams can measure performance, iterate models, and improve outcomes. • Help define and implement guardrails for AI features (fallbacks, explanations, error handling, permissions). • Champion operability: monitoring, alerting, logging, and runbooks for services you own. • Lead efforts to improve performance, scalability, and resilience of critical paths (e.g., onboarding, reporting, AI-assisted workflows). • Work with security and compliance to ensure features meet requirements around authentication, authorization, data privacy, and multi-tenancy. • Participate in and help evolve on-call/incident response processes as a technical leader. • Act as a technical mentor for multiple teams, raising the bar on code quality, reviews, testing, and design. • Lead technical design reviews and cross-team architecture discussions, especially where product, data, and ML intersect. • Help define engineering standards and best practices (API design, frontend patterns, error handling, observability, testing).
• Maintain, add to, and improve foundational scripts and pipelines that join raw data from Engineering, setting analysts up for success to develop reports, visuals, and insights. • Provide technical leadership on Looker/GCP deployment architecture, development guidelines, data delivery layer, optimization efforts, access configurations, and AI usage. • Perform data modeling across multiple business domains to unlock efficient analytical problem-solving and deep dives - working with large, complex data sets. • Collaborate with a wide variety of cross-functional partners, including senior management, to understand business needs, formulate, and complete end-to-end data intelligence infrastructure. • Manage the team workload and bandwidth, communicating expected quarterly deliverables, adhoc asks, and updating timelines.




