Job Closed

This listing is no longer active.

Visa logo
Visa

Based in Foster City, California, Visa is a global payments technology organization. Visa was founded in 1958, coinciding with Bank of America’s launch of the

Staff Data Engineer

Location

Brazil

Posted

52 days ago

Salary

0

Seniority

Lead

Job Description

Staff Data Engineer

Visa

• Design and implement data ingestion and transformation pipelines (batch and near-real-time) using PySpark/SparkSQL on Databricks. • Own data pipelines end-to-end in production: freshness, correctness, availability, and SLA adherence. • Build and maintain Delta Lake tables following medallion architecture patterns (bronze/silver/gold). • Design and optimize Airflow DAGs (MWAA) for complex orchestration scenarios. • Implement and maintain data quality frameworks (Great Expectations or equivalent) as integrated pipeline gates. • Write advanced SQL for data modeling, transformation, and performance optimization. • Conduct thorough code reviews. • Mentor Analyst-level engineers through pairing and design guidance. • Investigate, diagnose, and resolve data quality incidents and pipeline failures independently. • Collaborate with Analytics, BI, and Product teams to design consumer-friendly datasets. • Contribute to CI/CD, testing standards, and data governance practices.

Job Requirements

  • Apache Spark (PySpark, SparkSQL) — production experience
  • Databricks (jobs, workflows, cluster management, tuning)
  • Delta Lake (ACID tables, OPTIMIZE, VACUUM, schema evolution, MERGE)
  • Advanced SQL (window functions, CTEs, query optimization)
  • Apache Airflow / MWAA (DAG design, retries, backfills, SLAs)
  • Amazon S3 data lake design (partitioning, layout, lifecycle)
  • Data quality frameworks (Great Expectations or equivalent)
  • Data modeling (dimensional / Kimball, medallion layers)
  • Git/GitHub, CI/CD for data pipelines
  • Terraform
  • Python for automation and data processing
  • English level B2
  • Be based in Brazil
  • Desirable qualifications: CDC patterns (DMS, incremental processing, MERGE upserts) Streaming ingestion (Structured Streaming, Auto Loader) AWS Glue Catalog / Unity Catalog Metadata management (OpenMetadata) BI integration (Superset, dashboarding)

Benefits

  • Employees can work remotely

Related Categories

Related Job Pages

More Data Engineer Jobs

Duke Health logo

Oncology Data Specialist - Remote

Duke Health

Duke is an Equal Opportunity Employer committed to providing employment opportunity without regard to an individual's age, color, disability, gender, gender expression, gender identity, genetic information, national origin, race, religion, sex (including pregnancy and pregnancy related conditions), sexual orientation or military status. Duke aspires to create a community built on collaboration, innovation, creativity, and belonging. Our collective success depends on the robust exchange of ideas—an exchange that is best when the rich diversity of our perspectives, backgrounds, and experiences flourishes. To achieve this exchange, it is essential that all members of the community feel secure and welcome, that the contributions of all individuals are respected, and that all voices are heard. All members of our community have a responsibility to uphold these values. Essential Physical Job Functions: Certain jobs at Duke University and Duke University Health System may include essential job functions that require specific physical and/or mental abilities. Additional information and provision for requests for reasonable accommodation will be provided by each hiring department.

Data Engineer52 days ago
Full TimeRemoteTeam 10,001

At Duke Health, we're driven by a commitment to compassionate care that changes the lives of patients, their loved ones, and the greater community. No matter where your talents lie, join us and discover how we can advance health together. About Duke University Hospital Pursue your passion for caring with Duke University Hospital in Durham, North Carolina, which is consistently ranked among the best in the United States. The largest of the four Duke Healthhospitals with 1062 patient beds, it features comprehensive diagnostic and therapeutic facilities, including a regional emergency/trauma center, an endo-surgery center, and more. Commitment Bonus of $7,500 (2 equal installments over 12 months - 6 month increment) Location: Duke Hospital, Duke Cancer Center Institute (Remote) Must reside in one of the following states: · Indiana, Michigan, Maine, Ohio, and New Hampshire, Arizona, Hawaii, Illinois, Montana, Colorado, Massachusetts, North Carolina, New Jersey, Pennsylvania, California, Florida, Georgia, Maryland, New York, South Carolina, Tennessee, Texas, Virginia, Washington, DC Work Hours: Standard: Monday-Friday Duties and Responsibilities of this Level All hospitals in North Carolina are legally required to provide their cancer data to the State Department of Health's Central Cancer Registry monthly. In addition, the Duke Cancer Institute is an accredited NCI- Designated Comprehensive Cancer Center Program established by the American College of Surgeons. Therefore, the Duke Cancer Institute is required to adhere to Program Standards which include uniform coding as specified in the Facility Oncology Registry Data Standards. The Oncology Data Specialists, Certified abstract a wide range of medical data from the EMR and code it in compliance with data standard sets established by the American College of Surgeons and the NC Central Cancer Registry. This includes: - Medical history physical findings, screening information, occupation and any history of a previous cancer. - Diagnostic finding types, dates, and results of procedures used to diagnose cancer, cancer identification primary site, cell type, and extent of disease. - Cancer treatment surgery, radiation therapy, chemotherapy, hormone, or immunotherapy. - Patient identification age, gender, race/ethnicity, birthplace and residence. - Outcomes of annual follow-up information regarding patient status, recurrence and treatment. - Oncology Data Specialists, Certified must maintain a thorough understanding of anatomy and physiology, medical terminology, disease processes, surgical and radiation techniques. - They must participate in continuing education programs to learn about new chemo and hormone drugs, to effectively apply ICD-1O, FORDS, AJCC, and SEER coding guidelines to inpatient and outpatient diagnoses and procedures. All coding is correlated from supporting clinical documentation in the EMR. Oncology Data Specialists, Certified also participate in the development of coding policies and procedures as identified as well as independently research and solve complex coding problems and assist with special projects as required. - Oncology Data Specialists, Certified ensure adherence to data management protocols as set forth in state and national requirements, departmental standards and guidelines set forth by the DCI Cancer Committee. Perform other related duties incidental to the work described herein. Required Qualifications at this Level Education Preferred: Associate degree in Cancer Registry Management or Health Information Management. Experience Preferred: 2 years as an Oncology Data Specialist or Cancer/Tumor Registrar in a hospital or central cancer registry. Degrees, Licensure, and/or Certification Oncology Data Specialist certification required. Knowledge, Skills, and Abilities Proficient in Facility Oncology Registry Data Standards (FORDS), ICD-1O Coding, AJCC TNM, Collaborative Staging and Surveillance, Epidemiology, End Results (SEER) staging, multiple primary and histology coding. Excellent organizational skills with good follow through, investigative/analytic skills with detail orientation. Ability to achieve productivity goals under pressure. Good verbal and written communication skills. Working knowledge of Oncology Registry software, Windows-based software and EPIC EMR. Duke is an Equal Opportunity Employer committed to providing employment opportunity without regard to an individual's age, color, disability, gender, gender expression, gender identity, genetic information, national origin, race, religion, sex (including pregnancy and pregnancy related conditions), sexual orientation or military status. Duke aspires to create a community built on collaboration, innovation, creativity, and belonging. Our collective success depends on the robust exchange of ideas—an exchange that is best when the rich diversity of our perspectives, backgrounds, and experiences flourishes. To achieve this exchange, it is essential that all members of the community feel secure and welcome, that the contributions of all individuals are respected, and that all voices are heard. All members of our community have a responsibility to uphold these values. Essential Physical Job Functions: Certain jobs at Duke University and Duke University Health System may include essential job functions that require specific physical and/or mental abilities. Additional information and provision for requests for reasonable accommodation will be provided by each hiring department.

United States
Salla E-Commerce Platform logo

Senior Data Engineer

Salla E-Commerce Platform

مستقبل التجارة الإلكترونية، ابدأ تجارتك بسهولة 🛒

Data Engineer52 days ago
Full TimeRemoteTeam 51-200H1B No Sponsor

We are looking for a Senior Data Engineer to own, build, and scale the data infrastructure that powers Salla’s e-commerce ecosystem. In this role, you will be a core member of the data team, responsible for ensuring our data analytics platform is high-performing, engineering-grade, and capable of handling massive datasets, including real-time event tracking and product analytics. You will be responsible for the end-to-end lifecycle of our data pipelines, from ingestion (Data Lakes, Production Databases, APIs) to transformation (dbt, ClickHouse Warehouse) and consumption (reverse ETL, secure APIs). This is a hands-on role for a technical leader who prioritizes data quality, CI/CD excellence, and infrastructure optimization. You will partner closely with application engineers, data modelers, and analysts to build a world-class Medallion architecture. This role is ideal for someone who thrives on technical challenges, values architectural integrity, and wants to build the backbone of data-driven decision-making at the top e-commerce enabler in Saudi Arabia. Responsibilities - Pipeline Engineering: Design, build, and maintain scalable ETL/ELT pipelines from diverse sources including Data Lakes, Production ClickHouse instances, flat files, and various APIs. - Infrastructure & Orchestration: Configure and optimize our Data Warehouse infrastructure (ClickHouse) and orchestration layers (Mage.ai). - Engineering Excellence: Implement and manage "engineering-grade" CI/CD workflows, conduct rigorous PR reviews, and ensure robust dependency management across the stack. - Data Modeling & Architecture: Implement Medallion architecture (Bronze/Silver/Gold) and maintain high-performance data models using dbt. - Quality & Observability: Build automated data quality monitoring and alerting; proactively escalate upstream data issues to engineering teams and keep stakeholders informed of pipeline health. - Advanced Data Flows: Develop reverse ETL (rETL) pipelines and expose secure data APIs to enable seamless data consumption across the organization. - Strategic Integration: Manage event streaming and real-time data ingestion (Kafka, CDC) to support high-volume product analytics and tracking. Nice to Have - Proficiency in Arabic. - Based in Saudi (Jeddah, Makkah). - Direct experience with Kafka or CDC (Change Data Capture) integrations. - Experience in the e-commerce domain. - API development for data distribution.

Saudi Arabia
Job Closed

Data Engineer

United States Cold Storage - USCS

United States Cold Storage (USCS) is a leading provider of temperature-controlled warehousing and logistics services nationwide, committed to ensuring the safe

Data Engineer52 days ago

Title: Data Engineer Locations: Camden, New Jersey, 08103, United States Philadelphia, PA USA Job Category: IT Requisition Number: DATAE003972 - Full-Time - Hybrid Job Description: Who We Are US Cold owns and operates one of the most complex temperature-controlled logistics networks in North America. Every day, our systems coordinate the storage and movement of food at national scale across a network of state-of-the-art distribution centers, including multiple highly automated warehouse facilities. We continue to advance our core warehouse and logistics platforms. Our current focus is on modular, event-driven, API-first and cloud architectures. We continue to enhance reliability and accelerate engineering productivity by strengthening our SRE and AI practices. This is a large investment in innovation to continue to drive operational excellence at our facilities. If you want to build durable systems that operate in the physical world at scale, this is that opportunity. The Role: We are hiring a Data Engineer to design, build, and operate scalable, reliable data pipelines and platforms that power reporting, analytics, and machine learning across the organization. This role is suited for someone with a solid foundation in modern data engineering practices who enjoys solving complex data problems and working in production environments. You will develop and maintain data systems that ensure data is accurate, timely, and trusted. You will work across the full data lifecycle—from ingestion and transformation to validation, deployment, and monitoring. This role offers the opportunity to build a strong medallion data foundation—delivering reliable analytics‑ and AI‑driven systems where data quality, uptime, and operational impact truly matter. What You Will Own · Design, build, and operation of batch and streaming data pipelines. · Ingestion and processing of structured and semi-structured data from sources such as databases, APIs, event streams, logs, and files. · Data quality, reliability, and observability across data pipelines and platforms. · Development of clean, maintainable SQL and Python code for data transformations. · Delivery of high-quality, well-modeled datasets that support analytics and data science teams. · Application of best practices in data modeling, testing, version control, and documentation. · Leading and participating in code reviews and technical design discussions to ensure scalable and reliable solutions. · Staying current with modern data technologies and AI-enabled data workflows and applying them where they add value. Technical Environment · Multi-facility, high-availability Warehouse management and logistics systems · Migration toward cloud-native, event-driven architectures · Azure cloud-native services · ADF, Python and Azure functions to build data movement pipelines · Snowflake for real-time data warehousing · CI/CD-driven delivery model · Power BI for data visualization · Application of low-code platforms for process automations · AI mindset with application of data science models for warehouse optimization, routing, consolidation, and capacity planning What We’re Looking For · Professional experience building and supporting data solutions in production environments · Strong fundamentals in SQL (joins, aggregations, window functions) and Python or a similar programming language · Solid understanding of core data engineering concepts, including data pipelines and ETL/ELT patterns using tools such as ADF or Informatica · Experience with data modeling concepts, such as fact and dimension tables · Understanding of batch and streaming data processing paradigms · Familiarity with cloud data ecosystems (AWS, Azure, or GCP), with hands‑on or practical experience · Strong problem‑solving skills, curiosity, and a continuous‑learning mindset Education: Bachelor’s or Master’s degree in Computer Science, Data Engineering, Data Science, Engineering, Mathematics, or a closely related discipline. Helpful Experience · Hands‑on experience (academic or professional) with: o Snowflake, BigQuery, Redshift, or similar data warehouses o ETL tools such as ADF, Informatica o Dbt or transformation‑as‑code frameworks · Exposure to: o Machine learning pipelines or AI‑driven data products o Feature engineering or data preparation for ML models o Prompt engineering, LLM pipelines, or AI/analytics integration concepts · Understanding of data governance, privacy, or security fundamentals · Contributions to open‑source projects, hackathons, or strong personal projects · Knowledge of data visualization tools, such as Tableau or Power BI. Why this role is different You’ll help power live warehouses—moving freight and enabling real, day‑to‑day decisions that keep operations running. · You will help modernize the data and decision‑support backbone of a company that moves food at a national scale. · You will build durable, production‑grade data foundations—trusted datasets, clear definitions, and reliable data pipelines that operate under real‑world constraints, not slideware or one‑off analyses. The Kind of Problems You’ll Solve · Translating complex warehouse, logistics, and operational workflows into scalable data pipelines, models, and curated datasets that support analytics and AI use cases · Ensuring data used in daily decision‑making is accurate, timely, and resilient in environments where uptime and data quality matter · Modernizing data ingestion, transformation, and modeling without disrupting live warehouse and transportation operations You work closely with the business and technical teams to build production‑grade data systems that perform reliably in daily operations—not just in reports or prototypes. Compensation & Structure · Location: Hybrid – Greater Philadelphia · Reports to: IT – Data Engineering · Travel: Less than 5% · Salary Range: $80,000–$130,000 Operational Context This role is primarily technical and office-based, with occasional interaction in operational environments depending on system needs. Benefits Include Medical, Dental, Vision, Prescription, Legal Insurance, Pet Discount, Critical Illness, Accident Insurance, Hospital Indemnity, Long Term Care + Permanent Life Insurance, Identity Theft Protection, Short Term Disability Insurance, Long Term Disability Insurance, Additional disability benefit, Basic Life Insurance, Accidental Death and Dismemberment Insurance, Voluntary Life Insurance, Voluntary Spouse Life Insurance, Voluntary Child Life Insurance, Loan Solution, Health Flexible Spending Account, Dependent Flexible Spending Account, Commuter accounts, Telemedicine, Virtual Primary Care, Prescription Savings Plan, Prescription Specialty Copay Assistance Program, Weight Management Program, Chronic Condition Management, Care Navigator Program, 24/7 Nurse Line, Expert Medical Opinion, Precious Additions Maternity Program, Health Advocacy, Employee Assistance Program, Digital Cognitive Behavioral Therapy, Digital Physical Therapy, Behavioral and Mental Health Platforms, Auto and home discount program, Secure Travel Protection, Discount Programs, 401(k) plan, Education Assistance, Paid Time Off. Physical & Operational Context May require physical effort associated with using the computer to access information, or occasional standing, walking, lifting needed to carry out everyday activities. Effective communication, vision, and hearing are essential for safety and productivity. Operate scanners, tablets, radios, phones, computers, and other essential equipment as required. Additional work hours may be requested by management to help manage employee production, projects, and/or special events. Engage in frequent personal interaction and communication. Attend in-person meetings and/or training on a regular basis. Possess strong arithmetic and reading skills. Follow verbal instructions, written instructions, and company policies. Work independently and coordinate with others. Fast-paced environment, managing stress and meeting productivity standards. Additional Information Job functions may vary based on the area of operation. This description outlines the most common tasks required for the job. Reasonable accommodation may be provided to enable individuals with disabilities to perform essential duties. This job description may not encompass all tasks necessary to complete the role. Qualifications Education Preferred High School or better in Other.

Pennsylvania + 1 moreAll locations: Pennsylvania | New Jersey
$80K - $130K / year
Hy-Vee, Inc. logo

Senior Software Engineer, Data

Hy-Vee, Inc.

Making lives easier, healthier, happier

Data Engineer52 days ago
Full TimeHybridTeam 10,001+H1B No Sponsor

Design and maintain data integration pipelines, assemble complex data sets, implement process improvements, and build infrastructure for optimal data extraction and transformation to support analytics and business performance insights.

Iowa