DigitalOcean logo
DigitalOcean

The cloud ☁️ of choice for developers, startups, and growing digital businesses around the world.

Senior Engineer, Inference Data Plane

Full-stack EngineerSoftware EngineerFull TimeRemoteSeniorTeam 1,001-5,000Since 2011H1B SponsorCompany SiteLinkedIn

Location

United States

Posted

2 days ago

Salary

$139.2K - $174K / year

Seniority

Senior

Job Description

Senior Engineer, Inference Data Plane

DigitalOcean

Role Description DigitalOcean is expanding its AI Infrastructure layer to support the next generation of AI-driven applications. We are seeking a Senior Engineer 2 to join our AI Inference Data Plane team. In this role, you will be a key technical leader responsible for designing, developing, and delivering high-scale, resilient data plane services that power our "Inference as a Service" offering. - Technical Leadership: Act as a technical leader on the team, driving the end-to-end design, development, and delivery of critical data plane components hosting large generative AI models. - System Design: Architect and refine system design proposals for our high-scale, multi-tenant AI inference cloud ecosystem, ensuring they meet rigorous availability and resiliency standards. - Performance Optimization: Implement and optimize distributed inference hosting using techniques like tensor/data parallelism, KV cache optimizations, and smart routing. - Collaboration: Work cross-functionally with Product Managers, customer-facing teams, and other engineering teams to align technical roadmaps with customer needs. - Distributed Serving at Scale: Build on Kubernetes-native distributed inference frameworks like llm-d (or alternatives such as NVIDIA Dynamo, Ray Serve) to deliver prefill/decode disaggregation, KV-cache-aware routing, tiered prefix caching, and wide expert parallelism for MoE models. - Flow Control & Load Balancing: Solve the distributed-systems problems unique to LLM serving — inference-aware load balancing on queue depth, cache locality, and predicted latency; flow control and fairness across tenants; autoscaling inference pools; and moving gigabytes of KV-cache between prefill and decode instances with negligible overhead. - Open Source Contributions: Contribute upstream to llm-d, vLLM, and the inference gateway ecosystem, and represent DigitalOcean in these communities. - Mentorship: Coach and mentor junior engineers, fostering a culture of technical excellence and continuous improvement. - Operational Excellence: Maintain and operate critical, high-scale services, utilizing observability tools and defining SLOs to ensure superior platform health. Qualifications - AI/ML Domain Knowledge: Hands-on experience hosting large language or multimodal models using inference engines like vLLM, SGLang, or TensorRT. - Inference Frameworks: Familiarity with distributed inference serving frameworks such as llm-d, NVIDIA Dynamo, or Ray Serve. - Inference Engine Depth: Hands-on experience with vLLM or alternatives (SGLang, TensorRT-LLM, TGI, Modular MAX), including internals like continuous batching, paged attention, and prefix caching. - Distributed Inference Fluency: Understanding of why cluster-scale serving is hard: KV-cache locality is partitioned across workers, naive round-robin routing destroys cache hit rates and tail latency, and disaggregated prefill/decode requires fast cross-pod KV transfer (e.g., NIXL). - Upstream Track Record: Merged contributions to vLLM, llm-d, SGLang, or similar projects strongly preferred. - Architecture Proficiency: Knowledge of common LLM architectures and optimization techniques (e.g., continuous batching, quantization). - Software Engineering: Expert-level proficiency in GoLang or Python and familiarity with gRPC. - Cloud Operations: Proven experience shipping customer-facing software products and running critical services in a high-scale environment similar to DigitalOcean. - Open Source Mindset: Experience integrating and building with open-source software. Requirements - Compensation Range: $139,200 - $174,000 - This is a remote role. - JR: 2026-7624 - #LI-Remote Benefits - We innovate with purpose. - We prioritize career development. - We care about your well-being. - We reward our employees.

Related Job Pages

More Full-stack Engineer Jobs

AssureSoft - Careers logo

Interface Software Engineer

AssureSoft - Careers

AssureSoft is a multinational software development and information technology company providing strategic consulting, technology services, and outsourcing business processes. We work to innovate and create quality software with motivated, passionate, and qualified teams that develop in an environment of professional, stable growth and continuous learning. Inclusive Opportunities for Every Talent. At AssureSoft, we believe that true innovation is born from diversity—of ideas, experiences, and perspectives. That’s why our hiring practices are inclusive and reflect a firm commitment to equity and equal opportunity. Here, every person—regardless of origin, gender, orientation, or beliefs—finds a space to grow, contribute, and be valued not only for their talent, but also for who they are.

Full TimeRemoteTeam 201-500

Role Description - Design, develop, document, test, modify, and deploy clinical laboratory interfaces. - Build, maintain, and support HL7 and JSON integrations. - Collaborate with third-party vendors to define interface functionality and system specifications. - Troubleshoot and resolve issues during development, testing, deployment, and production support. - Provide ongoing technical support for clients using clinical laboratory integrations. - Assist with feature design, requirements analysis, documentation, prototyping, and testing. - Deliver interface training to healthcare facilities and clinical laboratories. - Audit integrations, document findings, and provide quality assurance feedback. - Respond to support requests from clinical laboratory partners and healthcare facilities. - Deliver a consistent, high-quality support experience through thorough analysis and strong attention to detail. Qualifications - 1–2 years of experience building, testing, and deploying SaaS interfaces. - 1+ year of experience with HL7 or an HL7 certification. - 1+ year of knowledge or experience in clinical laboratory operations. - Experience with SFTP clients. - Experience with Zendesk, ServiceNow, or similar ticketing systems. - Proficiency in Microsoft Office Suite. - High attention to detail and the ability to work on repetitive, accuracy-sensitive tasks. - Ability to work independently with minimal supervision. - Strong organizational and prioritization skills. - Strong written and verbal communication skills, including the ability to train end users. - Advanced English proficiency. Requirements - Nice to have: Automation scripting. Benefits - Great Place To Work certification. - A company with more than 15 years of experience. - Work with world-class clients and long-term projects. - English scholarships for an external institute. - English classes with company teachers. - State-of-the-art tools and resources. - Certifications for your professional growth. - Recreation and leisure activities. - Compliance with the regulations and labor rights of your region. Company Description AssureSoft is a multinational software development and information technology company providing strategic consulting, technology services, and outsourcing business processes. We work to innovate and create quality software with motivated, passionate, and qualified teams that develop in an environment of professional, stable growth and continuous learning. Inclusive Opportunities for Every Talent. At AssureSoft, we believe that true innovation is born from diversity—of ideas, experiences, and perspectives. That’s why our hiring practices are inclusive and reflect a firm commitment to equity and equal opportunity. Here, every person—regardless of origin, gender, orientation, or beliefs—finds a space to grow, contribute, and be valued not only for their talent, but also for who they are.

Worldwide
helloo logo

Coordenador de Engenharia de Software

helloo

Especialista em comercialização de mídia digital em shoppings, aeroportos e elevadores residenciais pelo Brasil.

Full TimeRemoteTeam 201-500Since 2016H1B No Sponsor

• Liderar a estratégia, evolução técnica e operacional dos times de engenharia, garantindo entregas escaláveis, seguras e alinhadas aos objetivos do negócio. • Garantir previsibilidade, qualidade e eficiência das entregas, acompanhando roadmaps, indicadores e capacidade das squads. • Atuar na definição de soluções técnicas, arquitetura e evolução tecnológica dos produtos. • Apoiar iniciativas de modernização de sistemas, microsserviços, Cloud e práticas de DevOps. • Promover boas práticas de engenharia, governança, observabilidade, automação e melhoria contínua. • Atuar como elo entre Engenharia, Produto, Arquitetura e áreas de negócio, participando de decisões estratégicas. • Liderar aproximadamente 15 profissionais entre Desenvolvedores, Tech Leads, Analistas e Key Users. • Desenvolver equipes de alta performance por meio de feedbacks, coaching, mentoring e planos de desenvolvimento. • Liderar o discovery técnico das iniciativas.

Brazil
DLS Discovery, LLC logo

Director, Software Engineering

DLS Discovery, LLC

The DLS Discovery Legal team provides leading e-Discovery and legal support services.

Full TimeRemoteTeam 51-200Since 2018H1B No Sponsor

• Run the Nebula development lifecycle end to end: quarterly planning, sprint execution, release management, and post-release quality • Set and track engineering metrics such as cycle time, escaped defects, and availability • Surface risk early and remove roadblocks so teams can stay focused • Advance the platform architecture with your senior engineers • Sponsor modernization work in areas like service architecture, infrastructure automation, and observability • Partner with Product and applied research to take AI features such as ECAi, summarization, and sensitive-data detection from prototype to production quality • Hire, coach, and grow engineering managers and senior individual contributors • Act as the engineering voice for Nebula with executives, Product Marketing, and Client Services • Own headcount planning, third-party technology relationships, and the engineering budget for the group

United States
$190K - $225K / year
Stefanini LATAM logo

Desarrollador Fullstack Sr.

Stefanini LATAM

Co-creating solutions for a better future

Full TimeRemoteTeam 10,001+Since 1987H1B No Sponsor

• Desarrollo End-to-End: Diseñar y construir interfaces de usuario interactivas (Front-End) y robustas arquitecturas lógicas (Back-End). • Integración Core: Configurar, personalizar y conectar módulos de la plataforma INSIS con aplicaciones internas y de terceros. • Diseño de APIs: Crear y mantener servicios web (REST/SOAP) para la cotización, emisión de pólizas y gestión de siniestros en tiempo real. • Optimización de Datos: Diseñar y optimizar consultas complejas en bases de datos para asegurar el correcto flujo de la información de seguros. • Mentoría Técnica: Colaborar con arquitectos de software y guiar a desarrolladores junior en las mejores prácticas del ecosistema INSIS.

Panama