LawPro.ai

• Continuous LLM Evaluation: Design and operate a systematic, ongoing process to evaluate new and emerging LLMs across accuracy, relevancy, speed, and cost — continuously benchmarking them against the specific tasks in our orchestration pipeline proactively optimizing outcomes. • Eval Framework Development: Build and maintain rigorous evaluation frameworks (Evals) to measure LLM output accuracy, relevance, faithfulness, and speed with a specific focus on reducing hallucinations in medical record summarization and legal document analysis. • Proactive Model Transition Planning: Monitor the LLM landscape across providers to identify deprecation timelines and suitable replacement models — and own the full execution of those transitions, including integrating new models into the production pipeline and maintaining necessary changes to account for model behavior. • AI Pipeline Optimization: Directly implement optimizations to LLM-based orchestration pipelines for document understanding, medical record summarization, case chronology generation, and drafting support — owning code changes, deployments, and production validation from start to finish. • Cross-Functional Collaboration: Partner with product and GTM stakeholders to communicate model evaluation findings — then lead the technical implementation yourself rather than delegating execution to a separate engineering team. • End-to-End Implementation Ownership: Take full responsibility for shipping model changes into production — writing the integration code, managing deployments, running validation tests, and ensuring a clean rollout. • Operational Monitoring: Implement monitoring and observability for model performance in production, benchmarking outputs and cost, detecting drift with ongoing and continuous reporting to management. • Documentation: Maintain thorough documentation of evaluation methodologies, model comparison results, transition decisions, and runbooks for the systems you own.