Job Closed
This listing is no longer active.
GCP Data Engineer
Location
Portugal
Posted
74 days ago
Salary
0
Seniority
Senior
Job Description
GCP Data Engineer
Devoteam
• Design and develop robust data pipelines and ETL/ELT processes using Google Cloud Platform services, including BigQuery, Dataflow, and Pub/Sub • Build and optimize cloud-based data warehouses and data lakes, ensuring data quality, security, and accessibility • Write clean, efficient, and well-documented code in Python or Java to transform and process large-scale datasets • Collaborate with data analysts, scientists, and business stakeholders to understand requirements and translate them into technical solutions • Implement data modeling best practices and maintain comprehensive data documentation and metadata management • Monitor and troubleshoot data pipeline performance, identifying bottlenecks and implementing optimization strategies • Participate in code reviews and contribute to continuous improvement of data engineering practices and standards • Maintain version control using Git and implement CI/CD practices for data infrastructure deployments • Support data governance initiatives and ensure compliance with data quality and security standards • Communicate effectively with distributed teams and provide technical guidance on data engineering solutions
Job Requirements
- Proven experience as a Data Engineer with hands-on expertise in Google Cloud Platform (GCP)
- Strong proficiency in SQL and data modeling for complex analytical requirements
- Solid programming skills in Python or Java
- Demonstrated experience designing and implementing ETL/ELT pipelines
- Experience with BigQuery or similar cloud data warehousing solutions
- Knowledge of distributed computing concepts and data architecture patterns
- Familiarity with version control systems (Git) and collaborative development practices
- Strong analytical and problem-solving skills with attention to detail
- Ability to work effectively in a nearshore, distributed team environment
Benefits
- Flexible work arrangements
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Role Description Our client is looking for a Data Engineer to join their Data Platform team, focusing on building scalable data pipelines and enabling analytics across the organization. In this role, you will work with modern data stack tools like Databricks, AWS, Airflow, Airbyte, and dbt to design and maintain data workflows that support reporting, analytics, and data-driven decisions. - Design and build scalable ETL/ELT pipelines using both batch and streaming approaches. - Develop ingestion workflows from multiple sources such as databases, APIs, and event streams. - Implement ingestion strategies including full load, incremental load, and CDC. - Orchestrate data workflows using Apache Airflow. - Manage data connectors using Airbyte. - Work with Databricks Lakehouse to build and optimize data processing pipelines. - Write and optimize complex SQL queries for analytics and transformation. - Build modular and testable data models using dbt (staging → intermediate → marts). - Maintain data quality, observability, and reliability across the platform. - Work with AWS services such as S3, Lambda, EC2, IAM. - Containerize data services using Docker and Kubernetes (EKS) when needed. - Document pipelines, data models, and data dictionaries for long-term maintainability. Qualifications - At least 5 years of experience in Data Engineering. - Strong understanding of data architectures such as Data Lake, Data Warehouse, and Lakehouse. - Hands-on experience with ETL/ELT pipelines, including batch and streaming processing. - Familiar with ingestion patterns: full load, incremental, CDC, event-driven. - Experience working with Databricks (Delta Live Tables, Jobs, Notebooks). - Strong skills in PySpark or Spark SQL for large-scale data processing. - Solid understanding of Delta Lake (ACID, time travel, schema evolution). - Experience with Apache Airflow (DAGs, scheduling, monitoring). - Experience with Airbyte or similar ingestion tools. - Strong SQL skills (CTEs, joins, window functions, query optimization). - Experience with dbt for transformation, testing, and documentation. - Hands-on experience with AWS (S3, Lambda, IAM, etc.). Requirements - Experience with Docker, Kubernetes (EKS). - Experience running Airflow or Airbyte on Kubernetes. - Familiar with data quality tools such as Great Expectations or Soda. - Experience with Terraform or Infrastructure as Code. - Exposure to data governance or catalog tools (e.g., Databricks Catalog). - Experience with CI/CD pipelines (e.g., GitHub Actions). - Strong Python skills for automation and pipeline scripting. Benefits - Attractive salary range and we are open to negotiate if you're a strong fit. - Hybrid/Remote-friendly culture, work where you grow best. - Flexible hours, async teamwork (we respect your focus time). - Work equipment support. - Allowance for Certification & Skill Development. - Year-end bonus & performance-based rewards. - 15 paid leaves a year. - Career growth with personal coaching sessions. - Open, collaborative team culture - no micromanagement, only trust. - Tools & AI-powered workflows that make remote work easier.
• Create and maintain optimal data pipeline architecture. • Assemble large, complex data sets that meet functional / non-functional business requirements. • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc. • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and Azure ‘big data’ technologies. • Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics. • Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs. • Strong communication needed. • Keep our data separated and secure across national boundaries through multiple data centers and Azure regions. • Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader. • Work with data and analytics experts to strive for greater functionality in our data systems. • Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement. • Strong analytic skills related to working with unstructured datasets. • Build processes supporting data transformation, data structures, metadata, dependency and workload management. • A successful history of manipulating, processing and extracting value from large disconnected datasets.
Role Description AdvanSix is seeking a Big Data Engineer to build and operate our enterprise Unified Data Layer (UDL) - spanning IT and OT - to deliver trustworthy, performant data products that power Finance, Operations, Supply Chain & Logistics, HSE, Commercial, and corporate analytics. You’ll engineer batch/CDC/streaming pipelines, model curated/semantic layers, and harden run-state with testing, CI/CD, security, and observability. You’ll partner closely with the data team and larger IT organization. Mission - Design and deliver scalable, secure data pipelines and data models that safely connect operational systems to analytics. - Ensure trusted and well‑governed data. - Enable repeatable delivery of BI, ML, AI, and automation solutions. Data Engineering & Modeling - Build ingestion pipelines (batch, CDC, streaming) from S/4HANA/DataSphere, PHD/historian, LIMS, TMS, HSE, and other sources into landing → curated → semantic layers. - Implement data contracts, schema/versioning, SCD handling, partitioning, and performance tuning (file formats, clustering, caching). - Develop dimensional/semantic models that back certified Power BI datasets and APIs for apps/agents. OT/IT Integration & Safety - Integrate OT data via OPC UA/MQTT, broker/DMZ patterns, read-only historian feeds, and event/batch frames—no control-net reads. - Collaborate with plant controls on change control, signal quality, and downtime windows. Quality, Security & Observability - Embed data quality rules, unit/integration tests, and validation checks (freshness, completeness, drift/PSI). - Instrument lineage and end-to-end monitoring; build alerting and on-call runbooks to minimize MTTR. - Enforce RBAC, secrets management, PII/HSE classifications, and retention aligned to Governance/MDM policies. CI/CD, Cost & Reliability - Automate build/test/deploy with Git-based CI/CD (environments, approvals, blue/green). - Track and optimize cost/performance (cluster sizing, autoscaling, cache strategy); contribute to FinOps reviews. Collaboration & Documentation - Partner with Reporting & BI on semantic model contracts, RLS, and performance SLAs; avoid direct system scraping. - Produce “readme” docs, data dictionaries, runbooks, and post-incident reviews; support knowledge transfer with vendors. Qualifications - Minimum 5 years' in data engineering building production pipelines at scale (batch/CDC/streaming). - Hands-on with Azure data stack: Databricks or Fabric/Synapse, ADF/Pipelines, ADLS/OneLake, Azure SQL/SQL MI, Key Vault. - Strong SQL and Python/PySpark; comfort with Spark Structured Streaming and performance tuning. - Experience implementing tests/observability (freshness, schema, expectations), and Git-based CI/CD. - Familiarity with SAP S/4HANA structures and SAP DataSphere semantic modeling. - OT concepts: historians (PHD/PI), OPC UA/MQTT, event/batch frames, ISA-95/99 basics. - Understanding of Power BI consumption (semantic models, RLS) and APIs for downstream AI/ML apps/agents. Preferred Qualifications - Time-series/data-quality tooling (e.g., Great Expectations or equivalent patterns), feature/metric stores. - MDM concepts (keys, survivorship), lineage/catalog tooling. - TMS/WMS, LIMS, Historian, HSE domain exposure; Lean/Six Sigma mindset; FinOps awareness. Benefits - We provide benefits that are industry competitive and focused on employee well-being. - Total Rewards program includes a competitive compensation, health, dental, vision & wellness programs, paid vacation, 401K with company matching, health savings programs, disability & life insurance, employee assistance program. - Tuition reimbursement for continued education, certifications, training, and development. - Work within a fast paced and innovative company, meeting passionate colleagues and partners with diverse backgrounds and experiences.
Role Description The Data Architect defines and governs the target-state architecture for AdvanSix’s Unified Data Layer, ensuring that data across SAP, OT, lab, logistics, HSE, finance, and commercial domains is structured, integrated, and modeled for scale, trust, and reuse. This role owns the architectural blueprints, canonical data models, semantic design standards, integration patterns, and data product design principles that guide engineering delivery across the enterprise. The Data Architect partners closely with the AI, Automation, and Data team to ensure that what gets built is coherent, secure, and business-ready. Core Responsibilities: - Enterprise Data Architecture: - Define and maintain the target-state architecture for the Unified Data Layer, including landing, curated, and semantic layers, domain boundaries, and consumption patterns for BI, APIs, AI, and automation. - Develop architecture principles for modularity, interoperability, reusability, and governance across business and operational data domains. - Serve as the design authority for major data platform and integration decisions. - Canonical Data Modeling: - Design enterprise and domain-level canonical models for key entities such as material, product, asset, equipment, work order, batch, vendor, customer, cost center, logistics shipment, quality event, and energy usage. - Define standards for dimensional modeling, semantic modeling, event models, and reference data structures that support Power BI, AI, and operational decision-making. - Ensure alignment of curated and semantic data products with master data strategy and KPI canon. - Integration and Data Product Design: - Define architectural patterns for ingesting and harmonizing data from SAP S4 HANA, SAP DataSphere, historians, LIMS, TMS, HSE systems, and other enterprise platforms into the Unified Data Layer. - Establish standards for data contracts, schema evolution, surrogate keys, versioning, partitioning, and semantic publication. - Ensure that certified datasets and APIs are designed for downstream use by Reporting and BI, machine learning, Power Platform, and Copilot agents. - Architecture Governance and Design Review: - Lead design reviews for new data products, major source integrations, and semantic layer changes. - Approve architectural patterns and ensure alignment with security, lineage, quality, retention, and access standards. - Partner with Data Strategy and AI Governance to operationalize architecture guardrails into delivery standards. - Collaboration with Engineering and OT: - Work with the OT Data Integration Lead to ensure plant data models, asset hierarchies, event frames, and OT schemas fit the enterprise architecture without taking over OT connectivity operations. - Work with Big Data Engineers to translate architecture standards into implementable pipeline, storage, and semantic design patterns. - Partner with the Manager, Unified Data Platform and AI Engineering on platform roadmap, architecture decisions, and technical debt prioritization. - Documentation and Enablement: - Produce architecture diagrams, conceptual and logical data models, reference patterns, and design standards. - Create reusable templates and guidance for engineers, analysts, and vendors. - Help stakeholders understand how data should be structured and consumed across the enterprise. Qualifications - Minimum 8 years' of experience in enterprise data architecture, data modeling, and large-scale data platform design. - Proven experience designing lakehouse or enterprise data platform architectures across multiple domains. - Strong expertise in conceptual, logical, and physical data modeling, dimensional modeling, semantic layer design, and canonical data structures. - Strong understanding of Azure-based data architectures, including Databricks or Fabric and Synapse, ADLS or OneLake, Data Factory or pipelines, and integration with Power BI. - Working knowledge of SAP S4 HANA data structures and SAP DataSphere semantic concepts. - Familiarity with OT and industrial data patterns, including historians, event frames, asset hierarchies, time-series data, and ISA-based structures. - Strong knowledge of data contracts, schema versioning, lineage, master data alignment, and architecture governance. - Excellent communication skills with the ability to translate architecture into clear implementation guidance. Preferred Qualifications - Experience in manufacturing, chemicals, process industries, or industrial operations. - Familiarity with TMS, LIMS, HSE, maintenance, and finance data domains. - Experience with lineage and catalog tooling, master data design, and KPI canon alignment. - Exposure to API-enabled data product design for AI agents and automation. - Lean or Six Sigma mindset and comfort working in cross-functional transformation programs. Benefits - We provide benefits that are industry competitive and focused on employee well-being. - Total Rewards program includes a competitive compensation, health, dental, vision & wellness programs, paid vacation, 401K with company matching, health savings programs, disability & life insurance, employee assistance program. - Tuition reimbursement for continued education, certifications, training, and development. - Work within a fast paced and innovative company, meeting passionate colleagues and partners with diverse backgrounds and experiences.

