Job Closed

This listing is no longer active.

Senior Data Engineer, AWS

Data EngineerData EngineerFull TimeRemoteSeniorTeam 10,001+H1B SponsorCompany SiteLinkedIn

Location

Brazil

Posted

44 days ago

Salary

0

Seniority

Senior

Job Description

Senior Data Engineer, AWS

Compass

• Develop and maintain ETL/ELT data pipelines in high-volume, complex environments using Apache Spark (Scala and Python) and AWS; • Contribute to the evolution and expansion of the Analytics Platform, focusing on Big Data architectures and technologies; • Gather, analyze, and translate business requirements into technical data solutions; • Design and maintain analytical data models, metrics, and KPIs; • Integrate multiple data sources, including streams (Apache Kafka / MSK) and SQL and NoSQL databases; • Develop, operate, and evolve data pipelines in production environments; • Ensure scalability, performance, reliability, data quality, and governance; • Prepare, cleanse, and make data available for analytical and operational consumption; • Automate the creation, configuration, and monitoring of data pipelines and environments; • Manage SLAs, handle failure remediation, and resolve incidents in data flows; • Work collaboratively in agile squads, supporting technical decisions and promoting data engineering best practices; • Provide technical mentorship and share knowledge within the team.

Job Requirements

  • Strong experience with Amazon Web Services, particularly serverless architectures, Data Lake, and event-driven integration;
  • Hands-on experience with storage, orchestration, messaging, streaming, data cataloging, and NoSQL databases within the AWS ecosystem;
  • Experience building and maintaining batch and streaming pipelines, including orchestration and failure handling;
  • Proficiency in Python (primary development language);
  • Experience with PySpark for data processing;
  • Advanced SQL (daily use in Athena);
  • Experience with MySQL (modeling, performance, and troubleshooting);
  • Organizing Data Lake into layered structure: raw / trusted / refined;
  • Knowledge of Parquet and JSON formats, partitioning, and read optimization;
  • Experience integrating data via RESTful APIs;
  • Experience in critical environments, with focus on SLAs, reliability, and observability;
  • Ownership mindset, proactive attitude, and active participation in architectural decisions.

Related Categories

Related Job Pages

More Data Engineer Jobs

HighlightTA logo

Data Migration Consultant

HighlightTA

Your on-demand talent team.

Data Engineer44 days ago
ContractRemoteTeam 1-10Since 2024H1B No Sponsor

• Maintain a full portfolio of new CRM clients, serving as their data point of contact throughout implementation. • Conduct data imports by preparing complex data import rules for software engineers. • Perform data validation checks throughout the process to ensure data integrity and accuracy, including pre-migration audits and post-migration reconciliation. • Examine source Excel data to identify its structure, quality, and potential issues like inconsistencies, missing values, or non-standard entries. • Document the entire migration process, including data maps, transformation logic, and any issues encountered, and communicate with business stakeholders and development teams to ensure all requirements are met. • Address and resolve any issues that arise during or after the migration, such as data import errors or data discrepancies. • Manage all projects efficiently by keeping relevant internal systems accurate and up to date, including recording progress in ChurnZero, ZenDesk, JIRA, Salesforce, and other company systems. • Collaborate effectively with internal cross-functional teams (e.g., Sales, Implementation, Support) to ensure alignment on client strategy, project status, and a cohesive, unified service experience throughout the migration lifecycle.

United States
$30 - $35 / hour
Sequoia Connect logo

Senior Solution Data Engineer

Sequoia Connect

Our core expertise lies in connecting Top Technologists with Top Companies through unparalleled IT headhunting solutions

Data Engineer44 days ago
Full TimeRemoteTeam 11-50H1B No Sponsor

Role Description We are currently searching for a Solution Data Engineer Sr : - Full management of the Databricks environment, including configuration and workspace administration. - Implement data governance using Unity Catalog and manage structured storage with Delta Lake tables. - Write and optimize data processing logic using Python, SQL, and PySpark. - Manage data architecture and performance within Snowflake. - Handle Adobe Data Feeds and Adobe Analytics, ensuring correct ingestion and transformation of marketing data. - Oversee data warehousing operations in Amazon Redshift. - Schedule and monitor complex data workflows using Airflow. - Own end-to-end maintenance of data pipelines, ensuring reliability from ingestion to delivery across multiple cloud platforms. - Manage code lifecycle and collaborative development using Git. Qualifications - Advanced proficiency in the Databricks Ecosystem (Workspace admin, Delta Lake, Unity Catalog). - Expertise in Data Warehousing & Integration, specifically with Snowflake. - Strong development skills in Python, SQL, and PySpark. - Hands-on experience with Adobe Data Ingestion (Data Feeds/Analytics). - Solid experience in version control using Git. Requirements - Experience with Amazon Redshift for data warehousing operations. - Proficiency in orchestration tools, specifically Airflow. - Familiarity with Google Cloud Platform (BigQuery) operations. - Knowledge of AWS Data Services such as Athena and AWS Glue. Benefits - Fully remote position.

Worldwide
Anteriad logo

Data Engineer

Anteriad

Anteriad is an advertising services company that specializes in providing full-funnel business-to-business (B2B) marketing solutions to help its clients drive g

Data Engineer44 days ago

• Build and maintain data ingestion pipelines using Azure Data Factory (ADF) to move data into the corporate Azure Data Lake and related platforms. • Develop and enhance data transformation logic in Databricks notebooks, primarily leveraging SQL-based transformations in existing SQL scripts and stored procedures. • Support a medallion data architecture (Bronze, Silver, Gold) with Silver and Gold datasets stored primarily in Parquet format. • Assist in migrating SQL Server–based processes (scripts, stored procedures, SSIS logic) running on Azure VMs into Azure-native pipelines and Databricks transformations. • Continuously update and enhance existing database solutions and systems based on new and evolving requirements from marketing campaign activation and analytics teams, ensuring data remains accurate, timely, and fit for purpose. • Manage multiple concurrent initiatives, including modernization efforts and ongoing enhancement requests, while maintaining platform stability and data quality. • Configure Azure Data Factory pipelines to orchestrate Databricks notebooks and Fabric-related processes as applicable. • Use Azure DevOps for source control, versioning, and deployment of Azure Data Factory, Databricks, and Fabric assets. • Test, monitor, and troubleshoot pipelines to ensure data reliability, performance, and quality. • Collaborate with senior engineers, project managers, campaign services, analytics, and platform teams to deliver ongoing data solutions. Participate in the adoption and use of Microsoft Fabric, understanding how Fabric components (e.g., OneLake, Lakehouse, Warehousing) integrate with existing Azure and Databricks workflows. • Maintain technical documentation for data pipelines, transformations, and datasets

United States
Job Closed
Thermo Fisher Scientific logo

Manager, Platform Data Architecture

Thermo Fisher Scientific

Thermo Fisher Scientific is a global biotechnology product development company whose mission is to make the world healthier, cleaner, and safer. Thermo Fisher Scientific leads a gl

Data Engineer44 days ago

Title: Manager, Platform Data Architecture Location: Remote North Carolina United States of America Full time Remote At Thermo Fisher’s PPD clinical research business (CRG), we’re using digital innovation, data science, and AI to reimagine how life-changing therapies reach patients. Our teams combine deep scientific expertise with advanced analytics, automation, and digital platforms to make research smarter, faster, and more connected. We know that innovation happens when diverse minds meet. Our Digital Science, Data, and AI professionals collaborate closely with scientists, clinicians, and operational experts to solve real-world challenges in clinical research. Alongside our partnership with Open AI, you can be part of the collaboration that will help to improve the speed and success of drug development, enabling customers to get medicines to patients faster and more cost effectively. You’ll join a culture that values experimentation, learning, and collaboration — where your ideas can help shape how we deliver life-saving solutions and improve global health outcomes. Manager, Platform Data Architecture The Manager, Platform Data Architecture serves as a central point of accountability for data architecture, data modeling and cross-platform alignment for Clinical Analytics Data Modeling, including ensuring data integrity, traceability, and audit readiness. This is a hands-on role that leads a small team of developers, ensuring alignment with business objectives and enterprise data strategy. The ideal candidate brings deep technical expertise in modern data ecosystems along with strong leadership and cross-functional collaboration skills. This is a home-based position for candidates on the East Coast US, EST time zone. Key Responsibilities - Manage and mentor a team of data developers, fostering technical excellence and professional growth - Define and enforce data architecture standards, best practices, and governance frameworks - Partner with business stakeholders, IT leadership, and product teams to translate business requirements into scalable data solutions - Oversee data integration strategies across internal and external systems (ETL/ELT, APIs, streaming) - Ensure data quality, security, privacy, and compliance with regulatory standards (e.g., GDPR, HIPAA as applicable) - Support analytics, AI/ML, and reporting initiatives by enabling accessible and well-structured data assets - Develop and maintain data engineering architecture documentation, including design specifications, data flow diagrams, and interface guidelines. - Monitor and troubleshoot data engineering processes, identifying and resolving issues to ensure reliable and efficient data flow. - Drive continuous improvement in data engineering practices, adopting new technologies and methodologies to enhance the engineering process. - Provide regular updates and reports to senior management on data engineering projects, status, risks, and opportunities. - Foster a collaborative and innovative team culture, encouraging professional growth and development. Education and Experience: - Bachelor’s degree in Computer Science, Information Systems, Engineering, or related field (Advanced degree preferred) - Previous data architecture or data engineering experience that provides the knowledge, skills, and abilities to perform the job (comparable to 8+ years’ experience). - 2+ years of experience managing technical teams - In some cases an equivalency, consisting of appropriate education, training, and/or directly related experience will be considered sufficient for an individual to meet the requirements of the role. Knowledge, Skills and Abilities: - 5+ years of hands-on experience with DBT/Python, Talend and Snowflake/Databricks in production environment - Solid SQL and understanding of database technologies like Snowflake, Databricks - Excellent problem-solving skills and the ability to think strategically. - Strong experience with cloud platforms (AWS, Azure, or GCP) and modern data stack technologies - Expertise in data modeling, database design (SQL/NoSQL), and data integration frameworks - Experience with big data technologies (e.g., Spark, Hadoop, Kafka) - Solid understanding of data governance, data quality, and security best practices - Experience in regulated industries (e.g., healthcare, life sciences, finance) (preferred) - Familiarity with data mesh or data fabric architectures (preferred) - Certifications in cloud platforms or data architecture (preferred) - Demonstrated leadership abilities with a focus on team development - Proven track record in process improvement and project management - Strong analytical and problem-solving skills - Excellent communication abilities across all organizational levels - Ability to manage multiple priority projects simultaneously At Thermo Fisher Scientific, we are committed to fostering a healthy and harmonious workplace for our employees. We understand the importance of creating an environment that allows individuals to excel. Please see below for the required qualifications for this position, which also includes the possibility of equivalent experience: - Able to communicate, receive, and understand information and ideas with diverse groups of people in a comprehensible and reasonable manner. - Able to work upright and stationary for typical working hours. - Ability to use and learn standard office equipment and technology with proficiency. - Able to perform successfully under pressure while prioritizing and handling multiple projects or activities. - May require as-needed travel (0-20%). Location: Remote East Coast USA (EST time zone). Relocation assistance is NOT provided. *Must be legally authorized to work in the United States without sponsorship. *Must be able to pass a comprehensive background check, which includes a drug screening. The annual salary range estimated for this position in North Carolina is $110,000 - $150,000 USD. This position may also be eligible to receive a variable annual bonus based on company, team, and/or individual performance results in accordance with company policy.

Maine + 13 moreAll locations: Maine | New Hampshire | Massachusetts | Rhode Island | Connecticut | New York | New Jersey | Delaware | Maryland | Virginia | North Carolina | South Carolina | Georgia | Florida
$110K - $150K / year