Title: AI Engineer III Location: 3750 Market Street, Philadelphia, PA, USA 118,160-147,700 per year Salary Full Time Job Description: POSITION OVERVIEW NBME is seeking a highly skilled AI Engineer to join our team and take ownership of building, maintaining, and scaling AI-driven systems. This role is critical for translating research innovations into production-ready pipelines and tools that support multiple programs, products, and services. The AI Engineer will work closely with research, IT, and product teams to operationalize AI solutions, ensuring they are reliable, secure, and optimized for performance. This role has been designated as primarily remote, meaning it requires little onsite attendance. While this role can be designated as remote, you are offered the flexibility to select if you prefer to work primarily remotely, hybrid, or onsite. We're open to considering candidates from the following states: PA, NJ, DE, MD, TX, GA, IA, IL, IN, KS, NC, SC, TN, VT, WV. Please note that onsite interviews and onboarding at our Philadelphia office may be required for this role. If applicable, advance notice will be provided to support planning.* At NBME, we continue to innovate and improve how we fulfill the evolving needs of the healthcare community. This commitment starts and ends with the people at NBME. By recruiting and empowering talented individuals from various disciplines and backgrounds, which includes professionals with diverse life experiences, abilities, and perspectives, NBME can take a well-informed, robust approach to advancing medical education and assessment for years to come. RESPONSIBILITIES: - Design, implement, and maintain production-grade AI and NLP pipelines, including model training, deployment, monitoring, and updates. - Collaborate with research teams on pre-production research and to operationalize prototypes, ensuring reproducibility, scalability, and robustness. - Build and maintain infrastructure for embeddings, large language models, and other AI assets. - Optimize AI systems for performance, reliability, and cost-efficiency. - Implement and maintain monitoring, logging, and alerting for AI systems in production. - Ensure compliance with data security, privacy, and organizational standards. - Support multiple product teams by providing AI engineering expertise, tools, and best practices. - Contribute to technical documentation, knowledge sharing, and internal training initiatives. QUALIFICATIONS: - Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field. - A minimum of 3 years of experience in AI/ML engineering, including model deployment, MLOps, system monitoring, and production pipelines. - Proficiency with Python and AI/ML libraries (e.g., PyTorch, TensorFlow, Hugging Face Transformers). - Experience with cloud platforms (AWS, GCP, Azure) and containerization (Docker, Kubernetes). - Familiarity with vector databases, embeddings, and large-scale NLP applications. - Strong problem-solving skills and ability to work collaboratively in a cross-functional team. - Excellent communication skills for technical and non-technical stakeholders. Preferred Qualifications: - Experience in educational assessment or healthcare-related AI applications. - Experience with Agentic LLM architectures for complex task completion. About NBME NBME is a not-for-profit organization that specializes in the creation of assessments and learning tools for physicians and health professionals. Our mission is to advance assessment of these professionals to achieve optimal care for all, supporting the development of a highly effective, diverse and compassionate health care workforce. NBME's Community Collaborations and Contributions NBME believes that a key path to meeting our mission is collaborating with and supporting our communities. Our Community Collaborations and Contributions programs provide resources and recognition to medical educators and researchers advancing the field of assessment throughout their careers. In addition, we invest in fee assistance, scholarships and pathway programs to aid learners and help support increased representation in health care. Headquartered in Philadelphia, NBME also gives to local organizations that advance health equity and access and positively impact social determinants of health. Compensation we are offering for this position is at 118,160 (min) -- $147,700 /year. Please note that the offered rate for this position typically aligns with the minimum to midrange of this grade, but it can vary based on the successful candidate's qualifications and experience, department budget, and an internal equity review. The NBME offers competitive salaries, excellent benefits, and a rewarding work environment. Excellent Benefits include: Healthcare, Dental, Prescription, and Vision plans; 401(k) w/match, Tuition Reimbursement Plan, Commuter Benefit: Public Transit or Parking options. Remote Friendly Workplace. NBME is an Equal Opportunity Employer. We will consider all qualified applicants for employment without regard to race; color; sex; gender identity; sexual orientation; religious practices and observances; national origin; pregnancy, childbirth, or related medical conditions; status as a protected veteran or spouse/family member of a protected veteran; or disability. #LI-Remote #Senior #Associate

Role Description We are looking for a Senior AI Engineer to join our AI Engineering team and help design, build, and roll out production-grade AI solutions, with a strong focus on AI engineering, AI agents, agentic workflows, machine learning, GenAI, and LLM-based applications. This is a senior individual contributor role within the AI Engineering team. The Senior AI Engineer will work closely with the AI Tech Lead, engineering teams, product stakeholders, data teams, cloud/platform teams, and security teams to deliver reliable AI capabilities into real business systems. The technology stack is diverse and can include Python (FastAPI/Flask/Django) or equivalent frameworks; React on the frontend side, and various ML/AI frameworks, APIs, cloud-native services, along with modern AI tooling. The role will have a strong focus on AWS, including Amazon Bedrock, Amazon Bedrock AgentCore, Amazon SageMaker, and other AWS AI/ML services. Responsibilities - Design, build, and maintain AI-powered applications, services, and integrations as part of the AI Engineering team. - Implement solutions focused on AI agents, agentic workflows, automation, LLM-based applications, and AI-assisted business processes. - Build and integrate AI applications using technologies such as Python (FastAPI/Flask/Django) or equivalent frameworks, React frontends, and relevant AI/ML frameworks. - Implement AI solutions using AWS AI/ML services, including Amazon Bedrock, Amazon Bedrock AgentCore, Amazon SageMaker, and other AWS services for model hosting, inference, orchestration, data processing, monitoring, and security. - Work closely with the AI Tech Lead to align on architecture, technology choices, engineering standards, AI patterns, and rollout approaches. - Provide technical input and guidance to other engineers on AI implementation patterns, code quality, testing, observability, and production readiness. - Develop and integrate AI agents that interact with internal APIs, business workflows, enterprise systems, knowledge bases, and external tools in a safe and controlled way. - Build and maintain RAG-based solutions, including document ingestion, chunking, embeddings, vector search, retrieval logic, reranking, and grounding techniques. - Support the development and deployment of machine learning models and AI solutions into production environments. - Contribute to ML pipelines and MLOps practices, including data preparation, model training, experiment tracking, model deployment, monitoring, evaluation, and lifecycle management. - Integrate LLMs through APIs. - Implement AI evaluation approaches for LLM outputs, RAG quality, agent behavior, model performance, hallucination detection, safety, and reliability. - Support prompt engineering, prompt versioning, function calling, tool use, memory patterns, guardrails, and LLM application testing. - Design and consume APIs and contribute to cloud-based, scalable backend architectures. - Collaborate with product managers, engineers, data scientists, DevOps, security, and business stakeholders to deliver practical AI solutions. - Write clean, maintainable, testable, and well-documented code. - Support production rollouts, troubleshooting, monitoring, optimization, and continuous improvement of AI systems. - Stay current with modern AI technologies, frameworks, models, and engineering practices, and bring practical recommendations to the team. Qualifications - Minimum 7-8 years of professional experience in software engineering, AI engineering, ML engineering, data science, or related technical roles. - At least 2-3 years of experience in AI development, ML engineering, or data science, with a demonstrated track record of deploying machine learning models and AI solutions in production environments. - Strong hands-on experience building production-grade AI, ML, and data-driven systems. - Practical experience with AI agents, agentic workflows, LLM-based applications, tool-calling architectures, workflow automation, and AI orchestration patterns. - Strong understanding of modern AI concepts, including deep learning, generative AI, LLMs, embeddings, RAG, LLM fine-tuning, and AI evaluation. - Strong Python development experience, including experience with Python (FastAPI/Flask/Django) or equivalent frameworks. - Some experience with React for building user-facing AI tools, internal applications, dashboards, or workflow interfaces. - Strong knowledge of AWS, including practical experience with cloud-native architectures, Amazon Bedrock, Amazon Bedrock AgentCore, Amazon SageMaker, and related AWS AI/ML services (the more, the better). - Experience with advanced LLM frameworks such as LangChain, LlamaIndex, Semantic Kernel, CrewAI, AutoGen, or similar agent/orchestration frameworks. - Experience with PyTorch or TensorFlow, and familiarity with Hugging Face Transformers. - Hands-on experience using LLMs via APIs, such as OpenAI, Anthropic, Gemini, or similar providers. - Experience with ML pipelines and MLOps, including data preparation, model training, model deployment, experiment tracking, model/version management, monitoring, evaluation, and production support. - Experience with AI evaluation frameworks, tools, and techniques for assessing LLM outputs, RAG performance, agent behavior, model quality, safety, reliability, and regression over time. - Knowledge or practical experience with RLHF - human-in-the-loop evaluation, preference data, reward modeling, or feedback-driven model improvement. - Experience with vector databases and retrieval/search technologies, such as Amazon OpenSearch, Pinecone, pgvector, or similar. - Experience building RAG systems, including document ingestion, chunking strategies, embeddings, retrieval evaluation, reranking, and grounding techniques. - Experience with model fine-tuning, embedding models, transformer architectures, open-source LLMs, and model benchmarking. - Knowledge of API design, microservices, event-driven systems, and cloud-based architectures. - Good understanding of security and governance requirements for AI systems, including access control, secrets management, data privacy, audit logging, and safe handling of sensitive data. - Experience working in cross-functional teams with engineers, product managers, data scientists, DevOps, security, and business stakeholders. - Strong problem-solving skills and ability to turn AI prototypes into reliable, maintainable production systems. - Strong communication skills and ability to explain technical decisions clearly to both technical and non-technical stakeholders. Considered as an Advantage - Experience with Amazon Bedrock Agents, Amazon Bedrock Knowledge Bases, Amazon Bedrock Guardrails, or similar managed AI capabilities. - Experience with containerization and orchestration, including Docker and EKS/ECS. - Experience with infrastructure as code using Terraform, AWS CDK, or CloudFormation. - Experience with data platforms, ETL/ELT pipelines, data lakes, feature stores, and real-time data processing. - Experience implementing responsible AI controls, AI governance frameworks, safety guardrails, and compliance processes. - Experience with observability for AI systems, including tracing, cost monitoring, prompt/model analytics, latency tracking, and quality dashboards. - Experience integrating AI systems with enterprise platforms, internal APIs, CRM/ERP systems, ticketing systems, knowledge bases, and workflow engines. - Contributions to open-source AI/ML projects, published technical content, conference talks, or patents in AI/ML-related areas. - AWS certifications, especially in architecture, machine learning, security, or DevOps. - Experience in fintech. Benefits - Fast-growing payment company; - Excellent working conditions, casual atmosphere, and state-of-the-art hardware; - Modern, challenging, constantly growing business; - Professional development – books, trainings, certifications, etc.; - Team buildings and fun activities; - 25 days paid holiday, 1 day for every 2 years with us; - Fully distributed and remote.

Role Description Dayforce is building an AI Developer Experience team within Engineering to scale high-quality, agentic development practices across the organization. This team transforms early-stage experimentation into durable, reusable engineering capabilities that development teams can adopt in real codebases under real delivery pressure. The Staff Developer, AI Experience is a senior hands-on engineering role and a key contributor to the broader AI Engineering strategy. This role focuses on building the systems, standards, workflows, and reusable development capabilities that enable engineers to work more effectively with AI-assisted and agentic coding tools. The role combines architectural thinking with hands-on implementation, helping define scalable engineering patterns while working directly within repositories and delivery environments to operationalize modern AI-driven development practices. What You’ll Get to Do - Own and evolve reusable skill design patterns and coding artifact standards that drive AI-assisted development effectiveness across Engineering. - Define how agent context is structured, scoped, and maintained across repositories, toolchains, and delivery pipelines. - Design reusable context patterns including prompts, retrieval strategies, memory/state patterns, and tool exposure configurations. - Establish practical guardrails and standards that reduce failure modes and support responsible AI-assisted development adoption. - Build and maintain reusable AI agent skills, workflows, templates, scaffolding, and implementation guides. - Define production-readiness standards for reusable skills including contracts, triggers, evaluation coverage, failure handling, and documentation. - Partner directly with development teams to operationalize AI-assisted workflows in repositories, testing practices, and engineering delivery processes. - Establish standards for emerging engineering artifacts such as AI-assisted specifications, implementation plans, and workflow patterns. - Define evaluation and adoption criteria for scalable AI engineering capabilities including reliability, engineering value, and maintainability. - Build data-driven visibility into AI-assisted development outcomes and ROI across Engineering. - Define scalable implementation standards that support consistent and lightweight AI engineering adoption. - Partner with internal platform teams and external vendors on tooling integration, feedback, and capability evolution. - Help establish and evolve the AI Engineering enablement operating model including priorities, success metrics, and organizational scaling strategies. Qualifications - Strong software engineering background with experience delivering and operating complex production systems. - Hands-on experience using agentic coding tools and AI-assisted development workflows on real engineering problems. - Experience implementing developer workflows across repositories, testing frameworks, CI/CD pipelines, and delivery processes. - Experience with context engineering including prompts, retrieval strategies, memory/state patterns, and LLM tooling configurations. - Experience designing reusable agent skills with defined contracts, evaluation coverage, and operational safeguards. - Demonstrated ability to define engineering standards, guardrails, and evaluation frameworks that improve consistency and quality across teams. - Experience partnering directly with development teams to implement shared engineering practices. - Strong technical judgment distinguishing scalable engineering practices from experimental concepts. - Experience working across organizational boundaries to improve engineering effectiveness and implementation consistency. - Excellent written communication skills with the ability to translate emerging technical practices into actionable engineering guidance. What Would Make You Stand Out - Experience building AI engineering enablement programs at scale. - Strong understanding of LLM evaluation methodologies and operational reliability patterns. - Experience designing enterprise AI developer platforms or reusable internal tooling ecosystems. - Demonstrated ability to influence engineering standards and adoption across large organizations. - Strong understanding of balancing experimentation, governance, scalability, and developer productivity in AI-assisted engineering environments. Benefits - Excellent time away from work programs. - Comprehensive wellness initiatives. - Recognition through competitive pay and benefits. - Opportunities for personal and professional growth. - Commitment to community impact, including volunteer days and charity initiatives.

• Design and implement core Agent OS platform services. • Write production code and review implementation details from other engineers. • Build reliable APIs, workflows, tools, and services for agent execution. • Inspect traces, debug failures, and improve production behavior. • Design evaluation scenarios and regression suites for agent workflows. • Work through real agent failure modes: stale context, wrong tool calls, missing permissions, unsafe actions, poor retrieval, latency spikes, and cost regressions. • Partner with domain teams to turn agent use cases into reusable platform patterns. • Help define platform contracts for tools, actions, approvals, context, memory, evidence, and evaluation. • Contribute to technical direction while staying grounded in what can ship quickly and safely. • Communicate clearly with engineers, product managers, architects, security partners, and engineering leadership.