The Single Source for Smarter Data™
Principal AI Engineer
Location
Pennsylvania
Posted
18 days ago
Salary
$180K - $200K / year
Seniority
Lead
Job Description
Principal AI Engineer
MRO
• Own the end-to-end technical vision for MRO Prodigy's AI layer — a production system that uses RAG, generative AI, and structured data reasoning to automate answers to Healthcare Registry questionnaires. • Define the AI roadmap for Prodigy, balancing near-term customer commitments against foundational capability investments that scale the product to enterprise maturity. • Evaluate and make build/buy/integrate decisions for AI capabilities — foundation model selection, embedding strategies, retrieval architectures, and orchestration frameworks — and own the consequences of those decisions. • Serve as the technical authority on all AI design decisions for Prodigy; produce architecture decision records, set standards, and ensure the architecture is defensible, auditable, and extensible. • Architect and evolve Prodigy's multi-modal retrieval pipeline, combining unstructured clinical document ingestion with structured EHR/FHIR data to surface accurate, citation-backed answers to registry questionnaire items. • Design and refine the answer generation layer — prompt engineering, context construction, grounding strategies, and output formatting — ensuring generated answers are clinically accurate and audit-ready. • Own the question routing and data source classification logic that maps registry questions to the right retrieval path, structured data field, or generation strategy. • Build and maintain the answer validation and confidence scoring framework, defining the statistics and quality thresholds that govern when answers are auto-accepted versus routed for human review. • Stay hands-on and close to the work: run direct ideation and feedback loops with Prodigy's end users (abstractors, registry, and quality teams) and with production analytics and monitoring systems — turning real usage signals into prioritized improvements that demonstrably move value, not just model metrics. • Evolve the feedback loop architecture that captures human corrections and routes them into continuous model improvement — ensuring Prodigy gets measurably better with every customer interaction. • Define the evals framework for Prodigy: how accuracy is measured, how regression is detected, and what signals trigger retraining or prompt revision. • Establish guardrails for hallucination detection and factual grounding specific to clinical registry use cases, where answer accuracy has direct downstream compliance implications. • Architect AI infrastructure across GCP (Vertex AI, BigQuery, Dataflow) and AWS (Bedrock), ensuring the pipeline is scalable, cost-efficient, and operationally observable. • Collaborate with data engineering to maintain high-quality, well-governed clinical and FHIR data inputs; define feature engineering and chunking strategies that optimize retrieval precision. • Define MLOps standards for Prodigy: model versioning, deployment gates, rollback procedures, drift monitoring, and audit trail requirements consistent with HIPAA compliance. • Act as the AI technical mentor for the Prodigy squad and adjacent engineering teams — guiding developers on RAG patterns, LLM integration, responsible AI practices, and clinical data handling. • Collaborate with Security and Compliance to ensure Prodigy's AI layer meets HIPAA requirements, including PHI handling in prompts, data residency, and model audit logging. • Foster AI literacy across the broader engineering organization, helping teams understand when and how to apply AI safely in a regulated healthcare context. • Partner with Product Management to translate registry workflow complexity and customer feedback into technically sound AI capability improvements.
Job Requirements
- Bachelor's in Computer Science, AI/ML, or related field; Master's or PhD preferred - or equivalent depth proven through shipped AI systems.
- Strong ML / data science / statistics theory foundation with the ability to read research, assess applicability, and execute.
- Hands-on LLM integration: prompt engineering, grounding, citation, hallucination mitigation, and output validation at clinical accuracy standards.
- Experience with LangChain, LlamaIndex, or equivalent orchestration frameworks.
- Built confidence scoring and auto-acceptance thresholds that govern when answers route to human review.
- Designed human-in-the-loop feedback systems that capture corrections and feed them back into model improvement.
- Production experience building RAG pipelines — document ingestion, chunking, embedding model selection, vector store management, and retrieval evaluation.
- Full ML lifecycle ownership in production: versioning, deployment gates, drift monitoring, rollback, and audit trails.
- Strong Python and software engineering fundamentals — CI/CD, testing, code review.
- Hands-on with vector databases (pgvector, Pinecone, Weaviate, or equivalent) and hybrid search.
- Built evals frameworks that measure accuracy, precision, recall, and F1 to inform product decisioning.
- Solid AWS and GCP experience: Bedrock, SageMaker, Vertex AI, BigQuery, Dataflow.
- Azure familiarity a plus.
- Experience building pipelines over mixed unstructured and structured data sources.
- Clinical NLP or healthcare AI experience — medical terminology, document structure, and regulated accuracy standards are not new territory.
- Knows how HIPAA applies to AI systems specifically: PHI in prompts and embeddings, data residency, audit logging, de-identification.
- Familiar with AI governance in practice: bias detection, explainability, responsible AI in compliance-sensitive contexts.
- Has owned AI technical vision before — not just contributed to it.
- Can write an ADR, set an engineering standard, and make it stick across teams.
- Communicates tradeoffs clearly to both engineers and non-technical stakeholders.
- Track record of mentoring engineers and raising AI maturity on a team.
Benefits
- Eligible employees may also receive an annual cash bonus
- comprehensive benefits offering, including medical, dental, vision, life insurance, and a 401(k) plan.
Related Guides
Related Job Pages
More AI Engineer Jobs
SEO, AI Search Lead
SoftrThe easiest no-code platform to build powerful portals and internal tools on top of your existing data.
• Own organic growth: Drive SEO and AEO/AIO performance across traffic, signups, and share of voice in AI answers you set the targets and you hit them. • Set and run the roadmap: From keyword and prompt research to publishing priorities, programmatic systems, and link strategy you decide what we ship and in what order. • Reposition Softr in LLMs: Move us off "no-code portal builder" and into accurate, current positioning across ChatGPT, Perplexity, Gemini, and Claude. Track it in Profound, close citation gaps, and engineer the content patterns LLMs actually quote. • Lead technical and programmatic SEO: Site health, indexation, Core Web Vitals, schema, and internal linking plus scoping programmatic page systems with engineering and managing quality at scale. • Build link, citation, and Reddit authority: Drive link and citation building across owned, earned, and partner channels, with a deliberate Reddit strategy that earns mentions LLMs and Google both trust. • Partner, report, and pitch new bets: Brief Content on structure and on-page, report monthly on SEO/AEO performance, flag risks early, and translate algorithm and LLM behavior changes into actions for the team.
Applied AI Engineer
BJAKBjak is a technology company focused on making financial services easy, fun and more rewarding for everyone
• Build AI-powered product features integrated into real insurance workflows. • Design and optimize LLM-based interactions across customer and internal systems. • Develop prompts, reasoning flows, and structured AI outputs for production use cases. • Evaluate model performance using real-world data, feedback loops, and experiments. • Improve reliability of AI outputs through guardrails, fallback logic, and validation layers. • Work closely with backend engineers to integrate AI capabilities into APIs and workflows. • Design and implement AI-driven automation features (chat, recommendations, decision support). • Analyze user behavior and model responses to continuously improve AI performance. • Collaborate with product, design, and engineering teams to ship end-to-end AI features. • Contribute to AI system design decisions focused on usability and real-world impact.
Applied AI Engineer
BJAKBjak is a technology company focused on making financial services easy, fun and more rewarding for everyone
• Build AI-powered product features integrated into real insurance workflows. • Design and optimize LLM-based interactions across customer and internal systems. • Develop prompts, reasoning flows, and structured AI outputs for production use cases. • Evaluate model performance using real-world data, feedback loops, and experiments. • Improve reliability of AI outputs through guardrails, fallback logic, and validation layers. • Work closely with backend engineers to integrate AI capabilities into APIs and workflows. • Design and implement AI-driven automation features (chat, recommendations, decision support). • Analyze user behavior and model responses to continuously improve AI performance. • Collaborate with product, design, and engineering teams to ship end-to-end AI features. • Contribute to AI system design decisions focused on usability and real-world impact.
Applied AI Engineer
BJAKBjak is a technology company focused on making financial services easy, fun and more rewarding for everyone
• Build AI-powered product features integrated into real insurance workflows. • Design and optimize LLM-based interactions across customer and internal systems. • Develop prompts, reasoning flows, and structured AI outputs for production use cases. • Evaluate model performance using real-world data, feedback loops, and experiments. • Improve reliability of AI outputs through guardrails, fallback logic, and validation layers. • Work closely with backend engineers to integrate AI capabilities into APIs and workflows. • Design and implement AI-driven automation features (chat, recommendations, decision support). • Analyze user behavior and model responses to continuously improve AI performance. • Collaborate with product, design, and engineering teams to ship end-to-end AI features. • Contribute to AI system design decisions focused on usability and real-world impact.
