Senior Data Engineer

Location

United States

Posted

97 days ago

Salary

$125.8K - $194.8K / year

Seniority

Senior

Job Description

Senior Data Engineer

Paul Hastings

Role Description We have an opening for a Senior Data Engineer. The Senior Data Engineer is responsible for designing, building and maintaining modern data pipelines and data integration solutions in the Azure Cloud. This role enables enterprise analytics by delivering clean, timely and well-structured data to platforms such as Power BI. The Senior Data Engineer collaborates closely with architects, analysts and business stakeholders to deliver scalable, secure and efficient data solutions using Azure Data Factory, Azure Data Lake Storage (ADLS), Databricks, and Azure DevOps. In this capacity, the Senior Data Engineer will: - Design, build and manage scalable data pipelines using Azure Data Factory (ADF), Azure Data Lake Storage (ADLS) and Azure Databricks; - Develop ELT frameworks for ingesting and transforming structured and semi-structured data from on-prem and cloud sources; - Implement robust DevOps pipelines for code versioning, testing, deployment and monitoring using Azure DevOps; - Ensure pipeline performance, reliability and observability using tools like Azure Monitor and Log Analytics; - Collaborate with BI teams to support Power BI datasets and ensure optimal data model performance; - Automate environment setup, configuration and permissions using infrastructure-as-code where possible; - Write modular, testable code in Python or PySpark and manage version control with Git; - Support production operations, root cause analysis and performance tuning; - Maintain technical documentation for pipelines, data mappings and deployment processes; - Work as part of an Agile delivery team, participating in sprints, retrospectives and code review; - Mentor junior data engineers and promote best practices in CI/CD, version control, and release management. Qualifications - Bachelor’s degree in Computer Science, Engineering or related field (required); - 7+ years of data engineering or BI development experience; - 3+ years building data solutions in Microsoft Azure Cloud; - 2+ years using Azure DevOps for automated builds, releases and pipeline orchestration; - Proven experience working with structured and semi-structured data (Parquet, JSON, Avro); - Experience in professional services, legal, or financial environments preferred. Requirements - Advanced proficiency in Azure Data Factory, Azure Data Lake Storage (Gen2) and Azure Databricks; - Experience building and managing CI/CD pipelines in Azure DevOps for data solutions; - Strong SQL and ELT experience; familiarity with schema design (e.g., star, snowflake, denormalized); - Hands-on expertise with PySpark and Databricks notebooks, including Delta Lake workflows; - Familiarity with Git, branching strategies and repository management within Azure Repos; - Understanding of data security and privacy best practices in Azure (RBAC, managed identities, encryption); - Experience with Power BI dataflows, datasets and refresh configuration (preferred); - Familiarity with data cataloging and governance tools (e.g., Microsoft Purview) a plus; - Experience with Agile tools such as Azure Boards or JIRA. Benefits - Medical, Dental, Vision, Life/AD&D, Long Term Care, and Short- and Long-Term Disability; - Flexible Spending Accounts and Health Savings Account; - Healthcare Concierge and Advocacy; - Lifestyle Spending Account; - Voluntary 401(k) Plan and Profit Sharing; - 10 Paid Holidays per year and a generous PTO Program; - Family Support including Paid Parental Leave, Fertility Benefits, Breast Milk Shipping, Back-up Child Care, Elder Care, and Tutoring; - Wellbeing programs (Employee Assistance Program, Relationship Support, Mental Health, Menopause and Midlife Health and Well-Being Events); - Retirement Plan Consulting; - Anniversary Bonus Program; - Professional Development Programs; - Transportation and Commuter Benefits; - International Travel Insurance; - Critical Illness, Hospital Indemnity and Accident Insurance; - Auto/Home/Pet Insurance; - Prepaid Legal Insurance; - Employee Discounts; - And More!

Related Categories

Related Job Pages

More Data Engineer Jobs

Spread Tecnologia logo

Senior Data Engineer

Spread Tecnologia

#PaixãoPorTransformação

Data Engineer97 days ago
Full TimeRemoteTeam 1,001-5,000Since 1983H1B No Sponsor

• Senior Data Engineer • Experience with Databricks • Experience with Python/Spark • Experience with Unity Catalog • CI/CD pipeline deployments • Experience working in Scrum

Brazil
Job Closed
Simples Dental logo

Support Assistant – Data Migration

Simples Dental

Simplificando a Gestão Odontológica

Data Engineer97 days ago
Full TimeRemoteTeam 51-200Since 2012H1B No Sponsor

• Contact clients directly immediately after the sale to welcome them and begin follow-up for the migration process; • Act as the frontline for starting migrations, recording and routing complaints, and coordinating with clients in complex cases; • Make calls to explain the process, resolve support questions, respond to emails, and ensure clarity in communications; • Proactively follow up with clients to request pending information essential for migration (e.g., sending backups, registering professionals, validation responses); • Receive and handle requests from the Sales, Support, or Customer Success (CS) teams related to migration status; • Actively research how to request backups from competing systems to guide clients and, subsequently, perform these requests independently; • Collect received backups and maintain strict organization of our migrations drive; • Perform basic data imports or imports that already have a fully established process; • Assist with migration tests to pre-validate data before final delivery to the client; • Continuously update the migration status control spreadsheet, ensuring the entire team has visibility into the progress of each clinic.

Brazil
Job Closed
Full TimeRemoteTeam 5,001-10,000H1B No Sponsor

Role Description Colaborar com membros da equipe para coletar requisitos de negócio e projetar modelos de dados. - Construir confiança em todas as interações e fornecer os dados mais completos, confiáveis e precisos para a empresa. - Projetar, desenvolver e estender código ETL e pipelines de dados para expandir o Data Lake Empresarial. - Criar e manter documentação de arquitetura e sistemas. - Manter conjuntos de dados principais ou produtos de dados atualizados no Catálogo de Dados para apoiar análises de Self-Service e Fonte Única da Verdade. - Desenvolver código que atenda aos nossos padrões internos de estilo, manutenibilidade e melhores práticas para um ambiente de dados de alta escala. - Manter e defender esses padrões através de revisão de código. - Aprovar mudanças em modelos de dados como Revisor da Equipe de Dados e ser responsável por conjuntos de dados específicos e esquemas de modelos de dados. - Compartilhar expertise em modelagem e transformação de dados para todas as equipes do iFood através de revisões de código, pareamento e treinamento para ajudar a entregar designs e consultas otimizados e escaláveis de conjuntos de dados ou produtos de dados no Databricks. Qualifications - Disposição para aprender coisas novas todos os dias. - Habilidade com SQL, Python e Spark. - Excelente comunicação: Alcançar consenso regularmente entre equipes técnicas e de negócio. - Capacidade demonstrada de comunicar de forma clara e concisa atividades complexas de negócio, requisitos técnicos e recomendações. - 2+ anos na área de Dados como analista, engenheiro ou equivalente. - 2+ anos de experiência projetando, implementando, operando e estendendo modelos de dados empresariais (dimensionais ou não). - 2+ anos trabalhando com Banco de Dados ou Data Warehouse de grande escala, preferencialmente em ambiente de nuvem. - Inglês técnico. - Desejável: Experiência com Databricks, Airflow e AWS. Requirements - Experiência prática em inteligência artificial. - Habilidade para gerar indicadores mensuráveis e identificar mudanças necessárias. - Capacidade de se adaptar rapidamente a novas tecnologias e desafios. - Interesse em promover a cultura de colaboração e troca de conhecimentos. Benefits - Buscamos uma pessoa apaixonada por tecnologia e dados, que esteja sempre em busca de novos aprendizados e que goste de desafios.

Brazil
Job Closed
Plus Power logo

Principal Data Architect – Battery Storage

Plus Power

Plus Power develops battery energy storage systems that enable a more efficient and reliable electrical grid.

Data Engineer97 days ago
Full TimeRemoteTeam 51-200H1B Sponsor

• Define and evolve data architecture standards for analytics and reporting, including data modeling, naming conventions, schema design, and documentation practices across the organization • Own the data catalog and metadata strategy, partnering with stakeholders to define, name, and organize data assets across multiple domains and source systems • Collaborate closely with Principal Data Engineering leadership and application engineering teams to align on ELT patterns, Snowflake usage, schema evolution, and analytical data modeling practices • Contribute hands‑on through SQL and Python, developing reference data models, prototypes, templates, and example implementations that demonstrate architectural intent • Support and enable data analysts by establishing consistent data usage, modeling standards, and shared definitions across a wide range of technical skill levels • Partner with application engineers on schema design to support rapid application development and reliable integration between operational and analytical data systems • Support PostgreSQL (including AWS Aurora) and Snowflake data modeling and analytical access patterns in collaboration with platform and database stakeholders • Establish and promote data governance practices covering data quality, ownership, lifecycle management, and schema change management • Drive incremental delivery of data architecture improvements, aligning short‑term progress with a clear long‑term architectural vision • Design high-ingestion pipelines (using tools like InfluxDB, Timescale, or Snowflake) capable of handling millions of data points per second from globally distributed battery sites • Ensure data can be seamlessly ingested from various industrial protocols such as Modbus, CAN bus, or DNP3, and translated into standardized cloud formats • Ensure data architectures comply with grid-specific regulations (like NERC CIP) and mandate on-site data storage for grid resilience • Help set the vision, roadmap and communicate the enterprise data strategy for the company

United States
$170K / year
Job Closed