Driver

• Own the LLM evaluation strategy at Driver — from first principles to production infrastructure. • Define quality metrics and build evaluation datasets. • Establish what 'good' looks like for each content type across the pipeline. • Build and curate gold-standard evaluation datasets across languages and repo archetypes (monorepos, microservices, libraries, applications). • Design rubrics that capture accuracy, completeness, usefulness, and readability. • Build benchmarking and experimentation infrastructure. • Create automated evaluation pipelines that score output against reference datasets. • Instrument the content generation pipeline to support A/B comparisons — run the same codebase through two strategies and compare results. • Build tooling for LLM-as-judge evaluation and regression detection. • Integrate evaluation into CI so pipeline changes come with quality evidence. • Develop automated quality signals at scale. • Build quality checks that flag degraded output without requiring human review of every document. • Monitor content quality trends over time. • Design sampling strategies for human review that maximize signal with minimal annotation effort. • Quantify tradeoffs and inform decisions. • Run experiments on model selection, context strategies, and pipeline architecture changes. • Quantify cost/quality/latency tradeoffs. • Partner with the engineering team to turn evaluation insights into shipped improvements.

• Design, deploy, and maintain cloud infrastructure (primarily AWS) • Build and automate CI/CD pipelines and deployment systems • Write code (Python, Go, or similar) to improve tooling and workflows • Manage and optimize Linux-based systems in production • Configure and troubleshoot networking (DNS, firewalls, routing, etc.) • Implement monitoring, logging, and alerting systems • Improve developer experience and internal tooling • Leverage AI tools (Claude, ChatGPT, etc.) to accelerate operations and engineering workflows • Continuously refine infrastructure for performance, reliability, and cost-efficiency • Manage bi-weekly releases • Work with sales engineering to optimize customer onboarding and automation.

• Contribute to building an efficient and scalable backend data model. • Build and maintain critical backend integrations (e.g., VCS providers). • Build and maintain the backend web server. • Design, build, and maintain internal APIs for our web application. • Build and maintain backend APIs for our Model Context Protocol (MCP) products. • Bring strong architectural instincts to the team. • Embrace AI-assisted development. • Communicate effectively with team members and across key team interfaces.