nDeavour Consulting logo
nDeavour Consulting

We are a staffing and IT recruitment company based in Sofia, Bulgaria.

Senior Data Engineer – Databricks, AWS

Cloud EngineerCloud EngineerFull TimeRemoteSeniorTeam 1-10Since 2019H1B No SponsorCompany SiteLinkedIn

Location

United States

Posted

3 days ago

Salary

0

Seniority

Senior

Bachelor Degree5 yrs expEnglishAirflowAWSCloudKafkaPySparkPythonSQLUnity

Job Description

Senior Data Engineer – Databricks, AWS

nDeavour Consulting

• Design and implement robust, scalable data ingestion and transformation pipelines using Databricks, PySpark, and distributed processing. Utilize Airflow (or similar) for reliable workflow orchestration. • Implement Delta Lake principles, focusing on CDC and schema evolution. Establish and integrate data quality frameworks (e.g., Great Expectations) within CI/CD pipelines for data integrity. • Develop and optimize complex SQL and Python scripts. Integrate diverse data sources (APIs, S3 files, etc.) and handle both structured and unstructured data. • Support the implementation of data governance and cataloguing solutions (e.g., Unity Catalog). Proactively investigate and improve inconsistent legacy datasets. • Guide and manage AI agents for code generation, ensuring quality and stability. Work pragmatically and collaboratively to drive technical solutions.

Job Requirements

  • 5+ years of professional experience in data engineering, focused on cloud and distributed processing environments.
  • Strong experience with Databricks, PySpark, distributed processing, and Delta Lake. Deep knowledge of CDC and schema evolution.
  • Expert SQL optimization and Python skills. Hands-on experience with Airflow (or similar).
  • Familiarity with relevant AWS components. Good understanding of CI/CD for data workflows and implementing data quality frameworks.
  • Knowledge of streaming (Kafka/Kinesis) and exposure to Unity Catalog or similar governance tools.
  • Proactive problem-solver; comfortable working with complex, inconsistent data without needing explicit specs.
  • Able to work independently in a fast-paced rebuild environment.
  • Collaborative, pragmatic, thrives on rapid iteration.
  • Skilled at directing AI agents to code while maintaining high quality standards.

Benefits

  • Remote Office – Flexible hybrid form of working
  • Parking Space – We provide free parking spots
  • Fun Office Space – We offer a game zone and a relaxation area
  • Health Insurance – Additional private health insurance, including a dental care plan
  • Personal Development – Company-sponsored training budget to further develop your skills
  • Employee Referral Program – Receive a bonus for referring a friend
  • Holidays – Enjoy an extra 5 days after your 1st and 5th year
  • Social Events – We love to celebrate our success together
  • Family Insurance – Add insurance to a family member
  • Offering sports cards – 100% sponsored by the company

Related Categories

Related Job Pages

More Cloud Engineer Jobs

Role Description Estamos em busca de um/uma Cloud FinOps Engineer com foco em IA/LLM para se juntar à nossa equipe. Quem ocupar essa posição trabalhará em estreita colaboração com as equipes de desenvolvimento, operações, dados/ML e finanças, garantindo a otimização de custos e a eficiência na utilização dos recursos em cloud — incluindo a nova frente de workloads de IA generativa e consumo de LLMs. Além das responsabilidades tradicionais de FinOps, esta posição terá um papel central na construção e operação da governança de custos da plataforma de IA: - Pilotar o gateway de LLM (LiteLLM) - Estruturar o modelo de chargeback baseado em tokens - Garantir visibilidade de custo por time, aplicação e caso de uso Sua rotina envolverá análise técnica, recomendação de soluções e busca contínua por oportunidades de otimização, tanto na infraestrutura cloud clássica quanto no consumo de modelos de IA. Como será seu dia a dia? - Analisar e otimizar a utilização dos recursos de cloud para garantir eficiência e redução de custos. - Colaborar com as equipes de desenvolvimento, operações e finanças para implementar práticas de FinOps e governança de custos. - Monitorar e gerenciar custos e orçamentos de projetos em cloud. - Implantar e gerir ferramentas de FinOps. - Garantir que as tags sejam devidamente aplicadas nos recursos da cloud. - Desenvolver e implementar políticas e estratégias de gerenciamento de custos de cloud. - Identificar oportunidades de adoção de novas tecnologias e práticas que melhorem a eficiência do uso da cloud. Frente de FinOps de IA/LLM (diferencial da vaga): - Operar e evoluir o gateway de LLM (LiteLLM): gestão de virtual keys, roteamento de modelos, rate limits e budgets por chave/time. - Estruturar e manter o modelo de chargeback/showback de consumo de IA (custo por token, por aplicação, por squad, por caso de uso). - Definir unit economics de IA: custo por requisição, por usuário e por feature. - Otimizar custo de inferência: seleção de modelo por tier de tarefa, caching de prompts, ajuste de context window. - Implementar guardrails e alertas de custo para workloads de IA. - Apoiar observabilidade de custo/uso integrando dados de ferramentas como Langfuse ao modelo de FinOps. Qualifications - Profundo conhecimento das principais plataformas cloud: AWS, Azure e GCP. - Experiência com Kubernetes e outras tecnologias de orquestração de contêineres. - Conhecimento em ferramentas de análise e gerenciamento de custos de cloud. - Conhecimento das arquiteturas de cloud e seus respectivos custos. - Familiaridade com linguagens de script, como Shell Scripting e Python. - Habilidade em automação de processos de infraestrutura (Infrastructure as Code). Requirements - Experiência com gateways de LLM (LiteLLM ou similar) e modelagem de chargeback por tokens. - Familiaridade com serviços de IA gerenciada (Amazon Bedrock, Vertex AI, Azure OpenAI) e seus modelos de precificação. - Noções de observabilidade de LLM (Langfuse ou equivalente). - Entendimento de unit economics aplicado a produtos de IA. Benefits - Auxílio Alimentação (VA/VR) no cartão Caju; - Auxílio Home-Office no cartão Caju; - Programa de Pontos Flexíveis em seu Caju; - Plano de Saúde Nacional Porto Seguro (Plano Prata com apartamento e sem coparticipação) com 50% da mensalidade custeada pela Builders; - Assistência Odontológica pela Porto Seguro (Plano Bronze) com 100% da mensalidade paga pela Builders; - Auxílio Creche para crianças de até 72 meses; - Auxílio à Saúde Mental - acesso ao aplicativo Conexa; - Auxílio à Saúde Física - acesso ao aplicativo Wellhub e/ou Totalpass; - Seguro de vida.

Brazil
Full TimeRemoteTeam 1,001-5,000Since 2006H1B Sponsor

• Designing, building, and owning our end-to-end observability strategy (metrics, logs, and distributed traces) for our distributed cloud infrastructure • Managing the Wrike product infrastructure • Implementing reliable solutions to ensure a product uptime SLA of 99.9% • Working with GCP, AWS and other cloud providers in the IaC paradigm • Introducing and supporting new infrastructure services • Actively participating in incident response and management, including on-call duties • Developing and maintaining professional connections within and outside of the team

Cyprus
Full TimeRemoteTeam 1,001-5,000Since 2006H1B Sponsor

• Designing, building, and owning our end-to-end observability strategy (metrics, logs, and distributed traces) for our distributed cloud infrastructure • Managing the Wrike product infrastructure • Implementing reliable solutions to ensure a product uptime SLA of 99.9% • Working with GCP, AWS and other cloud providers in the IaC paradigm • Introducing and supporting new infrastructure services • Actively participating in incident response and management, including on-call duties • Developing and maintaining professional connections within and outside of the team

Czechia
Dynatrace logo

Senior Java Software Engineer with cloud focus

Dynatrace

Dynatrace is a global application performance management software firm and a former member of Compuware. As an employer, the company is in support of helping it

Cloud Engineer3 days ago
Full TimeRemoteTeam 5,600Since 2005

Your role at DynatraceDynatrace, the world’s number one Software Intelligence monitoring platform, provides answers, not just data, about application performance and end-user experience. We’re looking for talented Software Engineers to make our product even better. Your role: - You will develop an application in Java/Spring Boot for managing multi-cloud native architecture. - You will provide an automated way of creating and configuring multi-cloud infrastructure. - You will be a part of the whole implementation process, starting with talks with Product Managers and Architects through design, coding and release preparation. - You will have one day per sprint devoted just to self-development and mentoring from best-in-class software engineers. - You will be working with highly skilled engineers who prioritize teamwork, utilizing collaborative practices such as pair programming to build software end-to-end, including debugging, testing, and fixing. - You will have a high degree of autonomy in solution design and implementation utilizing AI tools. - You will transition into a full-stack development role with end-to-end ownership, including deployment pipelines and deployment processes using modern CI/CD tools What will help you succeed - Solid 3+ years of experience with Java Professional experience in development and architectural design - Experience with AWS, Azure or GCP - Experience with test-driven development, clean code, design patterns and microservice architecture. - Experience in developing distributed and multi-tier applications (e.g. on Kubernetes) - Good command of English (min. B2) - Solid foundation in object-oriented programming, data structures, and algorithms - Familiarity with or curiosity about AI-assisted developer and security tooling, and a drive to embed AI-First practices into your team’s workflows Why you will love being a Dynatracer - Dynatrace is a leader in unified observability and security. - We provide a culture of excellence with competitive compensation packages designed to recognize and reward performance. - Our employees work with the largest cloud providers, including AWS, Microsoft, and Google Cloud, and other leading partners worldwide to create strategic alliances. - You'll get to work at the forefront of innovation with Dynatrace Intelligence—the industry's first agentic operations system. Bringing together deterministic and agentic AI, it helps teams understand what's happening, why it matters, and what to do next— automatically. - Over 50% of the Fortune 100 companies are current customers of Dynatrace. Compensation and Rewards - We offer only employment contracts (PL: Umowa o pracę / UoP) and a hybrid working setup with 2/3 days per week in the office. This is a permanent role and not a B2B contract. - We offer attractive compensation packages and stock purchase options with numerous benefits and advantages. - Base salary range — 16.833 - 21.000 PLN gross per month with higher pay based on experience and qualifications Equal Employment OpportunityDynatrace provides equal employment opportunities to all employees and applicants without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, veteran status, or any other protected characteristic. We actively foster an inclusive workplace that celebrates differences and promotes accessibility, collaboration, and growth for all.

Poland