Deepgram logo
Deepgram

Building foundational AI for speech transcription and understanding.

Senior Software Engineer – Model Evaluation, AI Systems

Systems EngineerSystems EngineerFull TimeRemoteSeniorTeam 51-200Since 2015H1B SponsorCompany SiteLinkedIn

Location

United States

Posted

5 days ago

Salary

$180K - $240K / year

Seniority

Senior

Postgraduate Degree5 yrs expExperience acceptedEnglishPythonRustGo

Job Description

Senior Software Engineer – Model Evaluation, AI Systems

Deepgram

• Define and build evaluation methodologies for Deepgram's models, spanning speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems. • Design, build, and maintain automated evaluation pipelines across batch and streaming (e.g. WER, runaway/hallucination detection, latency and time-to-first-byte), with a focus on correctness, reproducibility, and ease of adoption. • Build scalable, reproducible evaluation infrastructure — harnesses, orchestration, and result-aggregation pipelines — running against production models and, where needed, large GPU clusters. • Translate Research benchmarks and expected model metrics into automated, enforceable pass/fail gates. • Build and operate canaries and continuous-monitoring systems that detect quality regressions in production before they reach customers. • Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure. • Work alongside Research, model training, inference, and product teams to provide trusted evaluation signals that inform release and optimization decisions. • Integrate evaluation and quality gates into CI/CD so quality is verified continuously, not manually. • Help raise the bar through code reviews, technical design discussions, and strong engineering and QA practices.

Job Requirements

  • BS, MS, or PhD in Computer Science, AI, Applied Math, or a related field, or equivalent experience.
  • 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems (senior candidates with significantly deeper experience welcome).
  • Solid backend/scripting experience in a language such as Python, Rust, Go, or similar.
  • Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
  • Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results — able to distinguish real regressions from noise.
  • Ability to take charge of ambiguous technical challenges and communicate effectively across research, engineering, and product teams.

Benefits

  • Offers Equity
  • Offers Bonus
  • 10% Annual Bonus

Related Categories

Related Job Pages

More Systems Engineer Jobs

Cisco logo

Senior EDA Tools and Systems Support Hardware Engineer

Cisco

We securely connect everything to make anything possible.

Full TimeRemoteTeam 10,001+Since 1984H1B Sponsor

• Provide expert-level support for ECAD tools focusing on schematic capture and PCB layout workflows • Develop and maintain scripts to automate design tasks, improve data integrity, and streamline the design-to-manufacturing flow • Manage and maintain Linux-based EDA environments, provide secondary support for Windows-based design workstations • Manage EDA license servers, monitor usage, and troubleshoot checkout issues • Coordinate tool upgrades, patch rollouts, validation, and user communication and training • Create and maintain technical documentation, SOPs, internal best practices, support knowledge bases, and user guides

California
$152.4K - $221.8K / year
Job Closed
Full TimeRemoteTeam 1,001-5,000

• Administer, configure, and maintain enterprise learning platforms, including courses, learning plans, users, enrollments, roles, and permissions • Support one or more learning systems as part of a broader learning technology ecosystem, ensuring consistent governance and user experience across platforms • Maintain data integrity, audit readiness, and compliance through regular reviews, controls, and documentation • Define and communicate platform-level standards, naming conventions, and structural guidance • Enable stakeholders through documentation, onboarding, coaching, and review processes that support scalable self-service • Support accurate, defensible learning data and reporting across learning platforms and connected systems • Leverage built-in automation and AI-enabled features within learning platforms and supporting tools to improve consistency, discoverability, and administrative efficiency

Costa Rica
Baker Hughes logo

Lead, Embedded Software Engineer

Baker Hughes

We take energy forward – making it safer, cleaner, and more efficient for people and the planet.

Full TimeRemoteTeam 10,001+Since 1907H1B Sponsor

• Design high-reliability embedded software • Collaborate with cross-functional teams • Participate in the full software development lifecycle

Vietnam
ContractRemoteTeam 11-50H1B No Sponsor

• Gathering, analyzing, and documenting business and technical requirements • Supporting various technology initiatives • Working closely with stakeholders to translate complex business needs into actionable solutions • Ensuring alignment between business objectives and technical capabilities

United States
$80 - $90 / hour