Job Closed

This listing is no longer active.

Netomi logo
Netomi

Netomi is a software development company utilizing artificial intelligence (AI) to help its clients deliver “the highest-quality customer experiences.” As an employer, the comp

Manager – AI Performance, Analytics

Location

India

Posted

88 days ago

Salary

0

Seniority

Lead

Bachelor Degree7 yrs expEnglishPythonSQLTableau

Job Description

Manager – AI Performance, Analytics

Netomi

• Analyze large volumes of chatbot and AI conversation data to identify performance gaps and improvement opportunities. • Define and track key AI performance metrics such as resolution rate, containment rate, response quality, and customer satisfaction. • Build dashboards and reporting frameworks using SQL, Python, and BI tools such as Tableau. • Design frameworks to evaluate the response quality of LLM responses across dimensions. • Build scalable processes for LLM-based automated evaluation and human review workflows. • Identify systematic failure patterns in AI responses and recommend improvements. • Partner with AI and engineering teams to evaluate model upgrades, prompt changes, and new AI capabilities. • Design and analyze experiments to measure the impact of model improvements. • Generate insights that inform model training data, prompts, and product features. • Identify trends in customer interactions to uncover new AI automation opportunities and gaps in intent coverage. • Translate analytics insights into actionable recommendations for product and AI teams. • Lead and mentor a team of data analysts focused on AI performance and conversational analytics. • Establish best practices for analytics, experimentation, and AI evaluation. • Collaborate closely with product, AI research, engineering, and customer success teams.

Job Requirements

  • 7–10 years of experience in data analytics, data science, or AI analytics.
  • Experience managing or mentoring analytics teams.
  • Experience analyzing conversational data, customer support data, or AI-driven products is a strong plus.
  • Strong SQL and data analysis skills.
  • Experience with BI tools such as Tableau or similar platforms.
  • Experience using Python for data analysis or experimentation.
  • Familiarity with machine learning or large language models (LLMs).
  • Experience evaluating AI or machine learning model performance.
  • Experience working with conversational AI platforms or chatbots.
  • Experience with experimentation frameworks and A/B testing.

Benefits

  • Netomi is an equal opportunity employer committed to diversity in the workplace.

Related Job Pages

More Artificial Intelligence Jobs

Northwestern Mutual logo

Senior Director, AI Strategy – Architecture

Northwestern Mutual

You’ll Like It Here At Northwestern Mutual, we believe that our lives and our work matter. And that doing what’s right is good for everyone. We follow through by designing tech that improves the community and cultivating creative ways to make finance accessible anywhere. These guiding principles have allowed our company to grow for more than 160 years. Here, you’ll be with a team who emphasizes integrity and prioritizes security to design experiences that better everyone. You’ll work in cross functional teams to create optimal solutions that are rooted in innovative strategy and thoughtful execution. And you’re provided development tools and opportunities to become a leader all with the support of a collaborative team. You’ll be surrounded in a culture that values innovation and works to always evolve to stay ahead of trends and client needs. We are intentional in seeking out team members who will challenge us. Our employees choose us for the career opportunities, commitment to philanthropy and desire to have a meaningful impact in the lives of our clients. You have career passions and goals. We have ambition and opportunity for you to grow your future in tech. Discover today: https://careers.northwesternmutual.com/

OtherRemoteTeam 5,001-10,000Since 1869H1B Sponsor

• Lead the People function’s AI platform strategy, including the architectural design of AI-enabled employee and leader experiences. • Own platform selection and evaluation for the People AI assistant ecosystem (orchestration layers or agent communication protocol, RAG, agent frameworks, workflow engines). • Define the modular AI architecture required for secure integrations with Workday, ServiceNow, intranet platforms, and other data sources. • Establish standards for model usage, retrieval patterns, data access, versioning, logging, and system reliability. • Serve as the People function’s primary partner with IT, Enterprise Architecture, Cybersecurity, Legal and Risk for all AI-related technology and governance decisions. • Negotiate and align on data access patterns, authorization models, and integration pathways. • Represent the People function in enterprise-level AI forums, steering groups, and architecture reviews. • Ensure the People function’s AI strategy aligns to enterprise roadmaps while enabling innovative pilot opportunities. • Maintain the People function’s AI governance framework, including compliance standards, auditability requirements, safety guardrails, and escalation pathways. • Define guardrails for automated actions, system boundaries, fail-safes, and human-in-the-loop patterns. • Partner with Legal and Cyber to assess and mitigate risk associated with new use cases and vendor capabilities. • Establish monitoring mechanisms for reliability, performance, and ethical deployment. • Lead technical evaluation of AI vendors and platforms; establish architecture requirements for onboarding and implementation. • Partner with Digital Experience and Enablement Architecture teams to ensure platforms adhere to design, workflow, and content standards. • Define integration expectations, data flows, and technical constraints. • Oversee technical specialists and AI engineering resources supporting the platform. • Partner with Experience Architecture on workflow logic, triage patterns, decision boundaries, and human-machine interaction standards. • Support Workday/VNDLY, ServiceNow, and digital platform teams in AI-enablement opportunities, while ensuring integrity of system-of-record logic. • Provide technical guidance and architectural oversight for AI-enabled processes and cross-system workflows. • Build a multi-year AI roadmap for the People Function, including platform evolution, capability expansion, and integration milestones. • Evaluate emerging technologies, enterprise AI investments, and new architectural patterns to inform strategic decision-making. • Ensure the People function’s AI ecosystem evolves toward greater automation, personalization, and employee experience quality.

Wisconsin
$154K - $286K / year
Job Closed
Creative Fabrica logo

AI Marketing Specialist, Agentic AI – Automation

Creative Fabrica

We offer premium content from leading designers through an affordable subscription.

Full TimeRemoteTeam 51-200H1B No Sponsor

• Architecting Agentic Workflows: Design, deploy, and iterate on multi-agent systems. • Email Operations at Scale: Build and manage agent-driven email frameworks. • Content Supply Chain: Develop AI-powered pipelines. • Orchestration & Governance: Define the 'human-in-the-loop' (HITL) protocols. • Tech Stack Integration: Develop integrations between AI agents and our core toolkit. • Continuous Optimization: Establish KPIs for AI agents and refine prompts.

Netherlands
Meliorate GmbH logo

Mandatory Internship – Artificial Intelligence (AI), Market Analysis, Applications, Implementation

Meliorate GmbH

We are convinced that technical assets are THE key to a sustainable society and environment.

InternshipRemoteTeam 11-50Since 2011H1B No Sponsor

• Analyze existing company and consulting processes and identify areas with potential for AI deployment • Conduct market and technology research on suitable AI tools • Evaluate the identified technologies with regard to practical applicability • Prepare a structured overview of your findings and recommendations to jointly decide which tools should be adopted • Implement the agreed solutions • Test and document the new solutions

Germany
Job Closed
Weekday (YC W21) logo

AI Red-Teamer – Adversarial AI Testing, English

Weekday (YC W21)

We are a Y-Combinator-backed startup building your AI-powered Recruiter Agent

OtherRemoteTeam 11-50Since 2021H1B No Sponsor

• Red-team AI models and agents by testing jailbreak attempts, prompt injections, misuse scenarios, and exploit strategies • Generate high-quality human evaluation data by annotating model failures, classifying vulnerabilities, and identifying systemic risks • Apply structured testing methodologies using taxonomies, benchmarks, and playbooks to ensure consistent evaluation • Document findings clearly and reproducibly, producing reports, datasets, and adversarial test cases that teams can act upon • Work across multiple projects, supporting different AI systems and evaluation objectives

United States
$50 - $111 / hour
Job Closed