AI Quality Engineer
Location
United States
Posted
88 days ago
Salary
0
Seniority
Mid Level
No structured requirement data.
Job Description
AI Quality Engineer
Momentive
Role Description - Design and implement evaluation frameworks (evals) to assess LLM and agentic AI system quality, including accuracy, consistency, safety, and task completion rates. - Build and maintain automated test pipelines for AI features, covering unit, integration, and end-to-end scenarios across agentic workflows. - Develop tooling to detect regressions in model behavior, prompt outputs, and agent decision-making across releases. - Define and track quality metrics for AI systems (e.g., hallucination rates, tool-use accuracy, latency, failure recovery) and surface findings clearly to stakeholders. - Collaborate with engineers and product managers to identify edge cases, adversarial inputs, and failure modes specific to multi-step agentic pipelines. - Contribute to prompt evaluation strategies, including red-teaming, adversarial testing, and bias/fairness assessments. - Participate in design and code reviews with a quality-focused lens, raising concerns about testability and reliability early. - Help define and document quality standards and best practices for AI/ML features across the team. - Other duties as assigned. Qualifications - Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience. - 3–5 years of professional software engineering or quality engineering experience. - Hands-on experience working with LLMs or agentic AI systems (e.g., GPT-4, Claude, Gemini, or open-source models). - Proficiency in Python for scripting, test automation, and data analysis. - Experience designing and running evaluations (evals) for generative AI or LLM-powered features. - Solid understanding of software testing principles: unit, integration, regression, and end-to-end testing. - Familiarity with agentic frameworks and concepts (e.g., tool use, multi-step reasoning, retrieval-augmented generation, memory). - Experience with CI/CD pipelines and integrating automated tests into development workflows. - Strong analytical skills — able to interpret probabilistic outputs and distinguish meaningful regressions from expected variance. - Strong written and verbal communication skills; ability to clearly document findings and present quality data to non-technical stakeholders. - Detail-oriented, with a structured approach to exploring edge cases and failure scenarios. - Ability to work in a fast-paced environment and manage multiple priorities effectively. Requirements - Experience with prompt engineering and systematic prompt evaluation methodologies. - Familiarity with AI safety, alignment, or responsible AI concepts (e.g., hallucination mitigation, bias detection, guardrails). - Exposure to agentic orchestration frameworks (e.g., LangChain, LangGraph, AutoGen, CrewAI, or similar). - Experience with vector databases or RAG pipelines (e.g., Pinecone, Weaviate, pgvector). - Knowledge of observability and monitoring tools for AI systems (e.g., LangSmith, Weights & Biases, Arize). - Background in data science or ML experimentation practices. - Experience with version control systems (Git) and defect-tracking tools (e.g., Jira). - Exposure to cloud platforms (e.g., AWS, Azure, GCP) in the context of deploying or testing AI services. Benefits - Medical, Dental & Vision Benefits - 401(k) Savings Plan with Company Match - Flexible Planned Paid Time Off - Generous Sick Leave - Inclusive & Welcoming Environment - Purpose-Driven Culture - Work-Life Balance - Commitment to Community Involvement - Employer-Paid Parental Leave - Employer-Paid Short-Term Disability - Remote Work Flexibility
Related Guides
Related Job Pages
More AI Engineer Jobs
• Architect and develop robust backend systems that integrate with LLM APIs to power Clarium's AI features • Design and build scalable backend services in Python and FastAPI • Take full ownership of critical system components from REST APIs to AI agent implementation • Collaborate with product managers and customers to ensure what you build maps directly to real-world healthcare workflows • Participate actively in code reviews and sprint planning
Applied AI Engineer
BJAKBjak is a technology company focused on making financial services easy, fun and more rewarding for everyone
• Build and ship AI features end-to-end (model → system → user experience) • Design and iterate on prompts, tools, memory, and agent workflows • Turn raw model outputs into structured, reliable, and predictable behaviors • Debug issues across the full stack (model, orchestration, infra, UX) • Optimize for latency, cost, and production reliability • Develop lightweight evaluation frameworks to measure real-world performance • Work closely with product and engineering to translate ambiguous problems into working systems
AI Agents Engineer SR
NEORISNEORIS is a Digital Accelerator that helps companies step into the future.
Role Description En NEORIS estamos buscando un/a AI Agents Engineer SR para participar en proyectos de transformación digital y automatización inteligente, diseñando e implementando agentes de Inteligencia Artificial orientados a optimizar procesos a lo largo de todo el ciclo de vida del desarrollo de software (SDLC). El rol es 100% remoto para Argentina, con fuerte foco en orquestación de agentes, LLMs, prompt engineering avanzado e integración segura de IA en entornos productivos. Responsabilidades - Diseñar, desarrollar e implementar agentes de IA específicos por proyecto, alineados a objetivos de negocio y técnicos. - Orquestar múltiples agentes de IA y herramientas inteligentes, asegurando su correcta interacción e integración con plataformas existentes. - Aplicar prompt engineering avanzado para mejorar la precisión, consistencia y calidad de los resultados generados por modelos de IA. - Impulsar el desarrollo asistido por IA en actividades como análisis y resolución de bugs, generación de historias de usuario, testing y QA. - Crear, mantener y optimizar documentación técnica y funcional asistida por agentes de IA. - Implementar integraciones de IA contemplando seguridad, privacidad, compliance y gobierno del uso de modelos. - Automatizar procesos y flujos de trabajo mediante IA, incrementando la productividad de los equipos de desarrollo y gestión. - Colaborar de forma transversal con equipos de desarrollo, infraestructura, seguridad y management para asegurar una adopción efectiva de soluciones basadas en IA. Qualifications - Experiencia sólida en diseño y arquitectura de agentes de IA. - Conocimientos en orquestación de agentes y workflows inteligentes. - Experiencia comprobable en prompt engineering avanzado. - Trayectoria en AI-assisted development aplicado al SDLC. - Conocimiento profundo de las distintas etapas del ciclo de vida del desarrollo de software. - Experiencia en automatización de procesos y flujos de trabajo. - Conocimientos en integraciones seguras de IA, gobernanza y compliance. - Capacidad para trabajar de forma transversal con equipos técnicos y de negocio. - Nivel de inglés intermedio/avanzado. Benefits - Cobertura médica premium: Plan de salud de primera línea completamente cubierto para vos y tu familia. - Soporte para home office: Bono de conectividad para tus necesidades de trabajo remoto. - Capacitaciones: Descuentos en clases de inglés y acceso a Udemy. - Tiempo de deporte: Beneficios y descuentos en gimnasios. - Tiempo para vos: Vacaciones flexibles y día libre en tu cumpleaños. - Desarrollo profesional continuo: Oportunidades de aprendizaje y crecimiento constante.
Role Description We’re looking for a highly skilled AI Engineer to help us design, build, and scale intelligent systems across our next-generation global HR Tech platform. As an early-stage startup, we’re looking for someone who thrives in dynamic environments, is passionate about automation and AI-driven innovation, and brings hands-on experience developing AI/ML solutions within enterprise-grade SaaS platforms. This role reports directly to the Head of Engineering. - Design and implement machine learning models to solve HR-specific challenges (e.g., intelligent automation, anomaly detection, personalized experiences) - Collaborate with product managers and engineers to bring AI features to life across the platform - Develop and optimize data pipelines to support model training and inference - Research and apply NLP, LLMs, and other AI/ML technologies to streamline and improve HR workflows - Monitor model performance and retrain/improve based on feedback and usage - Build scalable AI solutions using cloud-native tools (preferably AWS) - Contribute to data governance, ethical AI use, and compliance Qualifications - 4–6+ years of experience as an AI/ML Engineer or Data Scientist, ideally with startup or SaaS exposure - Strong understanding of supervised/unsupervised learning, NLP, neural networks, and generative AI - Proficiency with Python and ML frameworks like TensorFlow, PyTorch, Hugging Face, or similar - Experience deploying models in production using cloud platforms like AWS Sagemaker, GCP Vertex AI, or Azure ML - Experience with data versioning, MLOps, and model lifecycle management - Ability to work autonomously and thrive in a startup environment - Familiarity with HR tech, payroll, or workforce data is a plus
