Diaconia logo
Diaconia

Creative IT Solutions | An Officially Great Place To Work!

Senior AWS Data Engineer

Data EngineerData EngineerOtherRemoteSeniorTeam 51-200Since 2020H1B No SponsorCompany SiteLinkedIn

Location

United States

Posted

100 days ago

Salary

$145K - $160K / year

Seniority

Senior

Bachelor Degree8 yrs expEnglishAWSETLJenkinsApache SparkSQLTerraform

Job Description

Senior AWS Data Engineer

Diaconia

• Build and maintain scalable data pipelines in AWS to support ingestion, transformation, and enrichment of structured and semi-structured data • Design and implement Delta Lake tables optimized for ACID compliance, partition pruning, schema enforcement, and query performance across large datasets • Develop ETL and ELT workflows that integrate multiple source systems into a centralized, query-optimized data warehouse architecture • Leverage AWS tools to implement business rules, dimensional joins, and aggregation logic aligned to warehouse modeling best practices • Collaborate with data architects and engineers to implement cloud-native data solutions on AWS using S3, Glue, RDS, and IAM for secure, scalable storage and access control • Optimize pipeline performance through intelligent partitioning, caching, broadcast joins, and adaptive query tuning • Deploy and version data engineering assets using Git-integrated development workflows and automate deployment with CI/CD tools such as GitLab or Jenkins • Monitor pipeline health, job execution, and cluster utilization using AWS CloudWatch, identifying bottlenecks and optimizing cost-performance tradeoffs • Conduct technical discovery and mapping of legacy source systems, identifying required transformations and designing end-to-end data flows • Implement governance practices including metadata tagging, data quality validation, audit logging, and lineage tracking using platform-native features and custom logic • Support ad hoc data access requests, develop reusable data assets, and maintain shared notebooks that meet operational reporting and analytics needs across teams

Job Requirements

  • This role requires knowledge and/or experience with Spark, Delta Lake, and distributed data pipelines
  • The ideal candidate brings both engineering and strategic insight into enterprise data modernization
  • 8+ years of experience in data engineering and Agile analytics
  • 5 years of experience building scalable ETL and ELT workflows for reporting and analytics
  • 3+ years of experience building enterprise data engineering solutions in the cloud, with preferred experience with cloud native technologies from AWS
  • Hands-on experience in the following: Glue / Spark SQL–based data transformations
  • S3 partitioning strategy
  • Step Functions–based orchestration
  • Infrastructure as Code (Terraform and/or CloudFormation)
  • Deployment automation for data pipelines
  • AWS services integration
  • Experience with data quality, validation frameworks, and storage optimization strategies
  • BA or BS degree
  • Excellent communication and organizational skills with the ability to manage multiple priorities
  • U.S. Citizenship is required by the Federal Client
  • Must have or able to obtain DoD Public Trust Clearance

Related Categories

Related Job Pages

More Data Engineer Jobs

Superlanet logo

Senior Data Engineer – Databricks, Provider, Patient & Enterprise Data

Superlanet

Advisory, Staffing, and Multi-State Employer of Record Solutions for Clinicians, by Clinicians.

Data Engineer100 days ago
OtherRemoteTeam 51-200Since 2017H1B No Sponsor

• Design and build Databricks pipelines for provider, patient, and enterprise data. • Support migration of existing Health Catalyst workloads. • Bridge Snowflake and Databricks environments. • Work closely with clinical, operational, and analytics stakeholders. • Ensure data quality, lineage, and performance. • Participate in backlog prioritization and execution.

California
$70 - $85 / hour
Job Closed
Jobgether logo

Data Engineer Lead

Jobgether

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best! Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time. #LI-CL1 We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.

Data Engineer100 days ago
OtherRemoteH1B No Sponsor

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more. Role Description This role involves serving as a senior technical leader responsible for shaping and managing the modern data ecosystem, directly impacting healthcare by delivering high-quality data products. - Design and evolve a scalable, secure, cloud-native lakehouse platform leveraging Databricks and Microsoft Fabric. - Define modeling patterns and governance frameworks across the data lifecycle. - Lead design reviews and guide teams in adopting scalable architectures. - Drive long-term platform strategy and evaluate emerging technologies. - Design and implement batch and streaming data pipelines for healthcare data sources. - Develop frameworks for ingestion, quality, lineage, metadata, and observability. - Produce clean, analytics-ready datasets for BI and machine learning. - Implement HIPAA-aligned access patterns and secure handling of PHI. - Architect Databricks workloads for reliability and efficiency. - Integrate Databricks and Microsoft Fabric with Azure services and enterprise systems. Qualifications - Bachelor's degree in Computer Science, Engineering, or a related field. - 7+ years of experience in data engineering. - 2+ years operating in a senior or staff level engineering role. - Proficiency with Databricks, Spark, Delta Lake, dbt, and Python. - Experience designing large-scale cloud data platforms (Azure preferred). - Familiarity with Data Engineering, Data Factory, and Lakehouse. - Advanced expertise in data platform architecture. - Ability to create modular, extensible frameworks for enterprise solutions. - Strong command of distributed data processing techniques. - Experience in HIPAA regulated environments and handling PHI. - Fluency in healthcare data compliance standards. - Technical leadership and mentorship capabilities. - Excellent communication skills with technical and clinical stakeholders. Benefits - 100% remote work (ET or CT work hours preferred). - Collaborative and inclusive work environment. - Opportunities for professional growth and development. - Mission-driven culture focused on improving healthcare. - Diversity, equity, and inclusion in recruitment practices. - Access to cutting-edge technology and resources.

United States
Job Closed
Jobgether logo

Staff Data Engineer

Jobgether

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best! Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time. #LI-CL1 We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.

Data Engineer100 days ago
OtherRemoteH1B No Sponsor

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more. Role Description This role involves serving as a senior technical leader responsible for shaping and managing the modern data ecosystem. Your contributions will directly impact our mission to enhance healthcare by delivering high-quality data products that support clinical and operational outcomes. - Design and evolve a scalable, secure, cloud-native lakehouse platform leveraging Databricks and Microsoft Fabric. - Define modeling patterns and governance frameworks across the data lifecycle. - Lead design reviews and guide teams in adopting scalable architectures. - Drive long-term platform strategy and evaluate emerging technologies. - Design and implement batch and streaming data pipelines for healthcare data sources. - Develop frameworks for ingestion, quality, lineage, metadata, and observability. - Produce clean, analytics-ready datasets for BI and machine learning. - Implement HIPAA-aligned access patterns and secure handling of PHI. - Architect Databricks workloads for reliability and efficiency. - Integrate Databricks and Microsoft Fabric with Azure services and enterprise systems. Qualifications - Bachelor's degree in Computer Science, Engineering, or a related field. - 7+ years of experience in data engineering. - 2+ years operating in a senior or staff level engineering role. - Proficiency with Databricks, Spark, Delta Lake, dbt, and Python. - Experience designing large-scale cloud data platforms (Azure preferred). - Familiarity with Data Engineering, Data Factory, and Lakehouse. - Advanced expertise in data platform architecture. - Ability to create modular, extensible frameworks for enterprise solutions. - Strong command of distributed data processing techniques. - Experience in HIPAA regulated environments and handling PHI. - Fluency in healthcare data compliance standards. - Technical leadership and mentorship capabilities. - Excellent communication skills with technical and clinical stakeholders. Benefits - 100% remote work (ET or CT work hours preferred). - Collaborative and inclusive work environment. - Opportunities for professional growth and development. - Mission-driven culture focused on improving healthcare. - Diversity, equity, and inclusion in recruitment practices. - Access to cutting-edge technology and resources. Company Description

United States
Job Closed
easysoftgroup logo

Data Migration Expert

easysoftgroup

If you are interested, please send your CV!

Data Engineer100 days ago

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more. Role Description As a Data Migration Expert, you will be responsible for planning, executing, validating, and troubleshooting complex data migrations into Shopify. You will work closely with developers, solution architects, project managers, and client technical teams to ensure seamless migrations with minimal business disruption. - Handle migrations of products, collections, customers, orders, gift cards, discounts, redirects, pages, and other critical commerce data, often from legacy or highly customized systems. Qualifications - Expert-level knowledge of Shopify, including: - Shopify Admin & data structures - Shopify Plus features - Metafields, APIs, and limitations - Proven experience handling complex ecommerce data migrations - Strong problem-solving skills for non-standard and edge-case scenarios - Excellent understanding of ecommerce data relationships (products, variants, inventory, orders, customers) - High attention to detail and data accuracy - Comfortable working with large datasets - Ability to explain technical concepts clearly to non-technical stakeholders Requirements - Lead and execute end-to-end data migrations into Shopify Plus - Design migration strategies for complex enterprise data models - Migrate and validate: - Products (including variants, metafields, media, pricing rules) - Collections (manual & automated) - Customers & customer data - Orders (historical and live cutovers) - Gift cards, discounts, and promotions - Pages, blogs, redirects, navigation, and related content - Work with Matrixify and other migration tools - Build, clean, transform, and validate large datasets (CSV, Excel, JSON, API-based exports) - Identify data gaps, inconsistencies, and risks early and propose solutions - Collaborate with: - Front-end developers (themes, custom storefronts) - Back-end developers (custom apps, APIs) - Middleware providers - ERP and PIM systems (e.g., NetSuite, SAP, Akeneo, etc.) - Support POS data considerations (customers, products, inventory, historical data) - Perform rigorous QA, reconciliation, and post-migration validation - Support cutover planning, dry runs, and go-live execution - Document migration processes, mappings, and edge cases - Provide technical guidance to internal teams and clients during migration phases Benefits - Hands-on experience with Matrixify - Experience migrating from: - Legacy ecommerce platforms - Headless or custom commerce solutions - Enterprise platforms (Magento, Salesforce Commerce Cloud, custom builds) - Experience working with: - ERP systems (NetSuite, SAP, Dynamics, etc.) - PIM systems - Middleware / iPaaS solutions - Knowledge of Shopify POS and retail data considerations - Basic scripting or technical skills (e.g., APIs, JSON, data transformation logic) - Familiarity with Shopify APIs, webhooks, or custom apps

United States + 171 moreAll locations: United States | Canada | Brazil | Colombia | Argentina | Chile | Venezuela | Bolivia | Ecuador | French Guiana | Guyana | Paraguay | Peru | Suriname | Uruguay | Mexico | Costa Rica | El Salvador | Guatemala | Honduras | Nicaragua | Panama | Dominican Republic | Puerto Rico | Bahamas | Guadeloupe | Haiti | Jamaica | Martinique | Montserrat | United Kingdom | Germany | France | Estonia | Portugal | Hungary | Poland | Ukraine | Romania | Bulgaria | Czechia | Slovakia | Belarus | Moldova | Sweden | Greece | Belgium | Italy | Ireland | Switzerland | Netherlands | Finland | Malta | Denmark | Lithuania | Croatia | Spain | Austria | Bosnia And Herzegovina | Iceland | Luxembourg | North Macedonia | Montenegro | Norway | Serbia | Slovenia | Albania | Cyprus | Latvia | Monaco | South Africa | Egypt | Algeria | Angola | Benin | Botswana | Burkina Faso | Burundi | Cameroon | Cabo Verde | Central African Republic | Chad | Congo | Côte D'ivoire | Democratic Republic of the Congo | Equatorial Guinea | Eritrea | Ethiopia | Gabon | Gambia | Ghana | Guinea | Guinea-bissau | Kenya | Lesotho | Liberia | Libya | Madagascar | Malawi | Mali | Mauritania | Mauritius | Mayotte | Morocco | Mozambique | Namibia | Niger | Nigeria | Réunion | Rwanda | Senegal | Seychelles | Sierra Leone | Somalia | Sudan | Eswatini | Tanzania | Togo | Tunisia | Uganda | Zambia | Zimbabwe | Georgia | Turkey | Israel | United Arab Emirates | Armenia | Azerbaijan | Bahrain | Iraq | Jordan | Kuwait | Lebanon | Oman | Qatar | Saudi Arabia | Palestine | Yemen | India | Japan | Philippines | Pakistan | Thailand | Singapore | Vietnam | Taiwan | Indonesia | Cambodia | Laos | Malaysia | Myanmar | South Korea | China | Afghanistan | Bangladesh | Bhutan | Kazakhstan | Kyrgyzstan | Maldives | Mongolia | Nepal | Sri Lanka | Tajikistan | Turkmenistan | Uzbekistan | Australia | Papua New Guinea | Kiribati | Palau | French Polynesia | Tuvalu | New Zealand
Job Closed