Role Description This role at Arbitration Forums is as unique as it is rewarding because of the AF IPAAL Values (Integrity, Passion, Accountability, Achievement, Leadership) and TRI Model (Trust, Respect, Inclusion). The GenAI and Agentic AI Engineer is responsible for designing, developing, optimizing, and testing generative AI and agentic AI solutions to enable the production of accurate, relevant, and high-quality outputs across various business applications. The ideal candidate will have a strong background in AI, natural language processing, and RPA, and deep expertise in Large Language Models, RAG, prompt engineering, and Robotic Process Automation. The GenAI and Agentic AI Engineer will partner with stakeholders to drive business value to Arbitration Forums and our members through GenAI and Agentic AI solutions. This role excels at the implementation of the agentic AI development process, employing AI techniques to guide and enhance solutions, developing effective AI interactions and automations through proficient programming and testing. Qualifications - Bachelor’s or Master’s degree in Computer Science, Information Systems, Data Science, Linguistics, or a related field. - Minimum of 7 years of experience in NLP, GenAI, LLMs, RAG, data governance, data science, or a related role. - Deep understanding of the components and development process of an Agentic AI solution. Requirements - Advanced programming knowledge, including mastery of programming languages such as Python, and especially AI-centric libraries like TensorFlow, PyTorch, and Keras. - Working knowledge of Azure AI Foundry and ML Studio. - Working knowledge of low code solutions, as well as tool integration and multi-agent collaboration. - Experience mapping and defining business processes and/or workflows for automation purposes. - Cloud computing knowledge for deploying and managing AI applications on cloud platforms like AWS, Google Cloud, or Microsoft Azure. - Deep understanding of containerization technologies like Docker and orchestration tools like Kubernetes for scaling AI solutions. - Experience combining design patterns like Tool Use, Reflection, and Planning for robust and solid automation. - Expertise in generative models such as generative adversarial networks (GANs) and variational autoencoders (VAEs). - Experience in Natural Language Processing (NLP) for text generation projects. - Working knowledge of techniques for text parsing, sentiment analysis, and the use of transformers like GPT models. - Data management knowledge, including data pre-processing, augmentation, and generation of synthetic data. - Proficiency in prompt engineering for generative AI models (GPT-4, DALL-E, etc.) and experience creating and integrating RAG capabilities. - Strong understanding of natural language processing concepts and techniques. - Understanding of Agentic AI frameworks and tools, like CrewAI and Azure AI Foundry. - Working knowledge of cloud services (i.e., MS Azure, AWS, Google Cloud). - Experience with AI tools, such as MS Azure ML, Databricks AI, Snowflake CortexAI, Dataiku. - Strong knowledge of data governance, data security, and compliance practices. - Familiarity with data visualization and reporting tools (e.g., Webfocus, Power BI). - Proficiency in programming languages such as Python, R, or SQL. Benefits - Support model and solution observability efforts to ensure adherence to company policies and enforce governance standards. - Other duties as assigned by manager or project needs. Company Description This is a fully remote position requiring reliable high-speed internet access and a dedicated workspace. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

• Lead a Forward Deployed AI engineering team and deliver AI-powered workflow transformation across various business functions. • Partner closely with the AI Platform team to ensure workflows transition cleanly to long-term ownership. • Drive deployments from discovery through production including workflow design, agent build, and human-in-the-loop checkpoints. • Serve as the primary contact for function heads throughout AI initiatives. • Maintain a prioritized pipeline of AI and automation opportunities.

• Lead the technical design, architecture, and delivery of AI solutions, with a focus on AI agents, agentic workflows, automation, and AI-assisted business processes. • Own the end-to-end engineering lifecycle of AI products: discovery, prototyping, evaluation, production implementation, rollout, monitoring, and continuous improvement. • Lead and manage an AI engineering team, including technical direction, task breakdown, mentoring, code reviews, delivery planning, and engineering quality. • Design and implement solutions using AWS AI/ML services, including Amazon Bedrock, Amazon Bedrock AgentCore, Amazon SageMaker, and other AWS services for model hosting, orchestration, data processing, monitoring, and security. • Build and integrate AI applications using technologies such as Python (FastAPI/Flask/Django) or equivalent, along with relevant AI/ML frameworks. • Design agentic systems that can interact with APIs, internal platforms, business workflows, knowledge bases, and external tools in a safe, observable, and controlled way. • Define and implement best practices for LLM application development, including prompt engineering, RAG, tool use, function calling, memory, evaluation, guardrails, and hallucination mitigation. • Drive improvements in internal engineering practices around AI-assisted development, engineering productivity, AI efficiency, automation, and responsible use of AI tools across software delivery. • Work with stakeholders to identify high-value AI use cases, assess feasibility, define success metrics, and prioritize delivery. • Establish engineering standards for AI systems, including code quality, testing, observability, reliability, security, scalability, and maintainability. • Drive MLOps and LLMOps practices, including model lifecycle management, deployment pipelines, monitoring, evaluation, drift detection, and rollback strategies. • Collaborate with DevOps, cloud, security, and platform teams to ensure AI systems are production-ready, compliant, cost-efficient, and operationally stable. • Support rollout and adoption of AI solutions across the organization, including documentation, training, stakeholder communication, and production support. • Evaluate emerging AI technologies, frameworks, models, and vendors, and provide pragmatic recommendations on adoption. • Ensure AI solutions follow responsible AI principles, including data privacy, access control, auditability, fairness, explainability where applicable, and secure handling of sensitive data.

• Advanced Prompt Engineering: Designing complex, dynamic prompt templates with conditional logic and efficiently reusing information and context within prompts to maximize generation quality and reasoning. • Structured Outputs & Schemas: Implementing various response schemes (JSON mode, function calling, Zod/JSON schemas) to ensure AI outputs are predictable and ready for seamless integration into application logic. • Prompt Engineering & Evaluations: Building robust evaluation pipelines and using Langfuse to collect feedback and score the quality of responses in real time. • Tracing & Debugging: Performing deep debugging of complex LLM chains using Langfuse traces to identify bottlenecks and optimize for cost, latency, and context window usage. • AI A/B Testing: Running systematic experiments across different models via OpenRouter (e.g., comparing Claude 3.5 Sonnet vs. GPT-4o) and analyzing results based on quantitative metrics. • Data-Driven Decisions: Making deployment decisions for new prompts or models strictly based on quantitative benchmarks and trace data, rather than intuition. • Output Scoring & Analysis: Developing scoring systems to analyze the “Problem → Solution” chain and identify root causes of hallucinations or logic errors using Langfuse analytics. • Model Performance & Fine-Tuning: Regularly re-evaluating model performance as new architectures emerge and performing fine-tuning when necessary to meet specific domain requirements.