Building foundational AI for speech transcription and understanding.
Senior Software Engineer – Model Evaluation, AI Systems
Location
United States
Posted
5 days ago
Salary
$180K - $240K / year
Seniority
Senior
Job Description
Senior Software Engineer – Model Evaluation, AI Systems
Deepgram
• Define and build evaluation methodologies for Deepgram's models, spanning speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems. • Design, build, and maintain automated evaluation pipelines across batch and streaming (e.g. WER, runaway/hallucination detection, latency and time-to-first-byte), with a focus on correctness, reproducibility, and ease of adoption. • Build scalable, reproducible evaluation infrastructure — harnesses, orchestration, and result-aggregation pipelines — running against production models and, where needed, large GPU clusters. • Translate Research benchmarks and expected model metrics into automated, enforceable pass/fail gates. • Build and operate canaries and continuous-monitoring systems that detect quality regressions in production before they reach customers. • Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure. • Work alongside Research, model training, inference, and product teams to provide trusted evaluation signals that inform release and optimization decisions. • Integrate evaluation and quality gates into CI/CD so quality is verified continuously, not manually. • Help raise the bar through code reviews, technical design discussions, and strong engineering and QA practices.
Job Requirements
- BS, MS, or PhD in Computer Science, AI, Applied Math, or a related field, or equivalent experience.
- 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems (senior candidates with significantly deeper experience welcome).
- Solid backend/scripting experience in a language such as Python, Rust, Go, or similar.
- Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
- Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results — able to distinguish real regressions from noise.
- Ability to take charge of ambiguous technical challenges and communicate effectively across research, engineering, and product teams.
Benefits
- Offers Equity
- Offers Bonus
- 10% Annual Bonus
Related Guides
Related Categories
Related Job Pages
More Systems Engineer Jobs
Senior EDA Tools and Systems Support Hardware Engineer
CiscoWe securely connect everything to make anything possible.
• Provide expert-level support for ECAD tools focusing on schematic capture and PCB layout workflows • Develop and maintain scripts to automate design tasks, improve data integrity, and streamline the design-to-manufacturing flow • Manage and maintain Linux-based EDA environments, provide secondary support for Windows-based design workstations • Manage EDA license servers, monitor usage, and troubleshoot checkout issues • Coordinate tool upgrades, patch rollouts, validation, and user communication and training • Create and maintain technical documentation, SOPs, internal best practices, support knowledge bases, and user guides
• Administer, configure, and maintain enterprise learning platforms, including courses, learning plans, users, enrollments, roles, and permissions • Support one or more learning systems as part of a broader learning technology ecosystem, ensuring consistent governance and user experience across platforms • Maintain data integrity, audit readiness, and compliance through regular reviews, controls, and documentation • Define and communicate platform-level standards, naming conventions, and structural guidance • Enable stakeholders through documentation, onboarding, coaching, and review processes that support scalable self-service • Support accurate, defensible learning data and reporting across learning platforms and connected systems • Leverage built-in automation and AI-enabled features within learning platforms and supporting tools to improve consistency, discoverability, and administrative efficiency
Lead, Embedded Software Engineer
Baker HughesWe take energy forward – making it safer, cleaner, and more efficient for people and the planet.
• Design high-reliability embedded software • Collaborate with cross-functional teams • Participate in the full software development lifecycle
• Gathering, analyzing, and documenting business and technical requirements • Supporting various technology initiatives • Working closely with stakeholders to translate complex business needs into actionable solutions • Ensuring alignment between business objectives and technical capabilities




