Improving people’s lives by harnessing the healthcare data explosion through an intelligent data integration platform.
Principal Data Engineer
Location
United States
Posted
2 days ago
Salary
0
Seniority
Lead
Job Description
Principal Data Engineer
Abacus Insights
• Architect Enterprise‑Scale Data Solutions: Design, build, and evolve high‑volume batch and real‑time data pipelines using PySpark, SparkSQL, Databricks Workflows, and distributed processing frameworks. • Own Platform‑Level Integrations: Develop end‑to‑end ingestion and transformation frameworks integrating Databricks, Snowflake, AWS services (such as S3, SQS, Lambda), and external data provider APIs, with a strong focus on data quality, lineage, and schema evolution. • Lead Technical Design for Clients: Serve as the technical lead for complex client implementations, defining highly available, fault‑tolerant architectures across multi‑account cloud environments. • Translate Business Needs into Architecture: Convert complex business and regulatory requirements into scalable technical designs, detailed specifications, and reusable engineering patterns. • Set Engineering Standards: Establish and champion best practices across CI/CD, code quality, testing, orchestration, monitoring, logging, and observability for data platforms. • Ensure Security & Compliance: Design and implement security‑first data solutions, including RBAC, encryption, PHI handling, auditability, and alignment with HIPAA and SOC 2 requirements. • Optimize Performance & Cost: Profile and tune compute workloads, cluster configurations, partitioning strategies, indexing, and caching across Databricks and Snowflake environments. • Provide Technical Mentorship: Mentor senior and junior engineers, conduct design and code reviews, and raise the overall technical bar across teams. • Produce Technical Artifacts: Create clear documentation, including architecture diagrams, runbooks, and operational standards that support scalable delivery.
Job Requirements
- 7+ years of hands‑on experience designing and operating large‑scale, distributed data systems in cloud‑based environments.
- Expert‑level proficiency in Python, SQL, and PySpark, including performance‑optimized distributed transformations.
- Proven experience building and operating production‑grade ETL/ELT pipelines using Databricks, Airflow, or similar orchestration frameworks.
- Strong working knowledge of AWS‑based data services (e.g., S3, SQS, Lambda, IAM) or equivalent cloud technologies.
- Experience working with dbt, Delta Lake, Kafka, or event‑driven architectures in modern data platforms.
- Hands‑on experience with Snowflake or other cloud data warehouses, including schema design and performance optimization.
- Demonstrated ability to design scalable, resilient systems requiring specialized knowledge of distributed computing and cloud‑scale data processing.
- Working experience with healthcare data domains such as claims, eligibility, provider, or clinical datasets.
- Ability to clearly communicate complex technical concepts to both technical and non‑technical partners.
- Bachelors or Masters degree in Computer Science, Engineering, Data Science, or a related field.
Benefits
- Unlimited paid time off – recharge when you need it
- Work from anywhere – flexibility to fit your life
- Comprehensive health coverage – multiple plan options to choose from
- Equity for every employee – share in our success
- Growth-focused environment – your development matters here
- Home office setup allowance – one-time support to get you started
- Monthly cell phone allowance – stay connected with ease
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Role Description A Deal Group é um ecossistema de tecnologia orientado a entregas end-to-end, que integra soluções especializadas em desenvolvimento digital (Deal Tech), marketing de experiência (Alot), cloud e Devops (O2B), inteligência de dados e IA (Sysvision) e soluções financeiras (baasic). Através de um modelo colaborativo e versátil, o grupo combina essas competências para resolver desafios complexos, oferecendo desde serviços pontuais até transformações digitais completas e escaláveis, garantindo que a expertise técnica de cada unidade de negócio culmine em uma execução única e estratégica para o negócio dos nossos clientes. - Desenhar e evoluir arquiteturas de dados: Definir padrões, boas práticas e estratégias que garantam eficiência e escalabilidade, incluindo FinOps, governança e integração entre plataforma de IA (ML/GenAI). - Viabilizar iniciativas de dados: Atuar desde a concepção dos projetos, interagindo com áreas de negócio para entender necessidades e traduzir requisitos funcionais em soluções técnicas robustas. - Criar soluções para ingestão e tratamento de dados: Desenvolver pipelines resilientes e escaláveis, aplicando conceitos de DataOps para garantir qualidade, automação e monitoramento contínuo. Qualifications - Experiência sólida em engenharia de dados em ambientes de grande escala. - Proficiência em arquitetura e modelagem de dados analíticos (Data Lake, Data Warehouse, Lakehouse, Data Mesh). - Domínio de tecnologias de Big Data e processamento distribuído (Spark). - Conhecimento avançado em SQL e otimização de consultas complexas. - Programação avançada em Python e pySpark. - Experiência com serviços de dados em nuvem AWS (Glue, EMR, Lake Formation, Redshift, S3, Step Functions, etc) e práticas de DataOps e FinOps. - Experiência com pipelines ETL/ELT (dbt, Glue) e orquestração (Airflow, Step Functions). - Familiaridade com microsserviços, APIs, containers e mensageria (Kafka, Kinesis). - Conhecimento em governança, segurança e compliance em ambientes críticos. - Boas práticas de engenharia de software (versionamento, CI/CD, testes). - Capacidade de traduzir conceitos de negócio em métricas quantitativas (KPIs/OKRs). - Experiência em mentoria técnica e liderança de times. Requirements - Atuação prévia em ambientes de IA (preparação de dados para ML/GenAI). Benefits - Trilha de carreira; - Convênio Farmácia; - Totalpass; - Assistência Médica; - Incentivo a certificação e qualificação; - Cultura Deal.
Senior AI Engineer (Payments Architecture & Strategy)
ZensarAt Zensar, we’re “experience-led everything”. We are committed to conceptualizing, designing, engineering, marketing, and managing digital solutions and experiences for over 130 leading enterprises. We are a company driven by a bold purpose: Together, we shape experiences for better futures. Whether for our clients, our people, or the world around us, this belief powers everything we do. At the heart of our culture is ONE with Client - a set of four core values that reflect who we are and how we work: One Zensar, Nurturing, Empowering, and Client Focus. Part of the $4.8 billion RPG Group, we’re a community of 10,000+ innovators across 30+ global locations, including Milpitas, Seattle, Princeton, Cape Town, London, Zurich, Singapore, and Mexico City. We believe the best work happens when individuality is celebrated, growth is encouraged, and well-being is prioritized. We are an equal employment opportunity (EEO) and affirmative action employer, committed to creating an inclusive workplace. All qualified applicants will be considered without regard to race, creed, color, ancestry, religion, sex, national origin, citizenship, age, sexual orientation, gender identity, disability, marital status, family medical leave status, or protected veteran status.
Role Description We are building a next-generation AI Payments Engineering Team to design and deliver an AI-native Payments Operating System. This team will combine AI engineering, payments domain expertise, and production automation to transform high-effort, high-risk financial operations into scalable, intelligent, and auditable systems. The team will focus on automating critical Payments processes such as: - Least cost routing - Reconciliations - Audit - Incident automation The mission is to deliver efficiency with control integrity—not just automation speed. Key Responsibilities - Assess and map end-to-end Payments operations, including: - Reconciliations - Reporting & controls - Audit - Identify operational bottlenecks, capability gaps, and integration failures. - Define a Payments automation roadmap based on operational impact, regulatory risk, and scalability. - Design the AI-native Payments Operating System, including: - Agentic system design - Data pipelines - Integration with financial systems - Control and reporting frameworks - Design and implement an agentic Payments layer that: - Operates alongside the current stack - Automates workflows today - Enables gradual migration to a future-state architecture - Evaluate and simplify the existing Payments technology stack. - Represent Payments in the AI Center of Excellence, ensuring alignment across teams. - Classify all automation by risk tier before development. - Ensure each solution includes: - Named Process Owner - Documented data flows - Access controls - Full audit logging - Design automation that preserves control integrity in regulated environments. - Develop reusable frameworks, templates, and workflows. - Create documentation and standards to scale adoption. - Build dashboards and metrics to measure: - Efficiency gains - Cost reduction - Risk outcomes - ROI - Train Payments teams on AI-enabled workflows and tools. - Work closely with: - Payments Backend teams - Infrastructure & platform teams - Compliance & audit teams - AI Center of Excellence - Translate business and finance requirements into technical solutions and roadmaps. - Communicate effectively with technical and non-technical stakeholders. Qualifications - Hands-on experience deploying production-grade AI/agentic systems. - Strong understanding of: - LLM orchestration - Multi-agent systems - AI failure modes in high-stakes environments - Proficiency in: - Spring AI/Python-based orchestration - Workflow engines (n8n or equivalent) - AI APIs (Anthropic/Claude or similar) - Ability to deliver across the lifecycle: Discovery → Design → Build → Deploy → Optimize. - Strong architectural thinking for scalable, future-ready platforms. Ways of Working - Operates effectively in fast-paced, ambiguous environments. - Strong ownership mindset with a bias for execution. - Ability to bridge business and engineering domains.
Data Engineer
Municipal Credit UnionOur members are New York City's everyday heroes. We are proud to be their credit union.
• Integrate data sets from various source systems into unified formats • Perform ELT (Extract, Load, Transform) operations • Cleanse and transform data by applying business rules • Conduct data cleansing and validation • Monitor and troubleshoot data issues • Create and maintain conceptual, logical, and physical data models • Build and maintain data pipelines • Load processed data into analytical data stores • Monitor the performance and quality of data transfers • Integrate test suites into CI/CD pipelines • Optimize data flows for efficiency, scalability, and reliability • Collaborate with team members to gather requirements
• Design, implement, and continuously improve systems for data ingestion, processing, storage, and sharing within a Data Mesh platform across cloud and legacy DWH environments • Build and optimize AWS cloud and on-premises (Oracle DB) data architectures for performance, scalability, and reliability • Develop and maintain ETL / ELT pipelines using modern tools and frameworks to support regulatory, operational, and analytical needs across banking areas • Support and modernize legacy data systems, ensuring smooth migration toward cloud platform architecture • Uphold high standards of data quality, security, availability, and performance • Collaborate with analysts, software engineers, and business stakeholders to understand data needs and deliver solutions • Perform code reviews, troubleshoot software, and resolve defects • Implement monitoring and alerting for data workflows • Participate in the EOD process — including a one-week ROTA rotation with availability until 23:00–24:00 for incident troubleshooting and resolution • Work as part of a cross-functional Agile team in a Pan-Baltic international environment
