Job Closed

This listing is no longer active.

Leega logo
Leega

Inteligência, Inovação e Tecnologia.

Mid-level Data Engineer – GCP, DBT

Data EngineerData EngineerFull TimeRemoteSeniorTeam 201-500Since 2010H1B No SponsorCompany SiteLinkedIn

Location

Brazil

Posted

73 days ago

Salary

0

Seniority

Senior

Job Description

Mid-level Data Engineer – GCP, DBT

Leega

• Analysis and Planning of Loads/Pipelines: • Assess the data warehouse architecture and requirements. • Map data, transformations, and processes across GCP services (Cloud Storage, BigQuery, Dataproc). • Define the data migration strategy (full load, incremental, CDC). • Develop a data architecture plan on GCP. • Data Design and Modeling on GCP: • Design table schemas in BigQuery, considering performance, cost, and scalability. • Define partitioning and clustering strategies for BigQuery. • Model data zones in Cloud Storage (Bronze, Silver, Gold). • Development of ELT/ETL Pipelines: • Create data transformation routines using Dataproc (Spark) or Dataflow to load data into BigQuery. • Translate business logic and existing transformations into GCP. • Implement data validation and quality assurance mechanisms. • Performance and Cost Optimization: • Optimize BigQuery queries to reduce costs and improve performance. • Tune and optimize Spark jobs on Dataproc. • Monitor and optimize GCP resource usage to control costs. • Data Security and Governance: • Implement and ensure data security in transit and at rest. • Define and apply IAM policies to control access to data and resources. • Ensure compliance with data governance policies. • Monitoring and Support: • Troubleshoot performance and functionality issues in data pipelines and GCP resources. • Documentation: • Document the architecture, data pipelines, data models, and operational procedures. • Communication: • Communicate effectively with team members, stakeholders, and other areas of the company. • Ensure clear communication between architecture definitions and software components, and support the evolution and quality of the team's developments. • Jira / Agile Methodologies: • Be familiar with agile methodologies, their ceremonies, and be proficient with the Jira tool.

Job Requirements

  • Proven experience with DBT of at least 3 years.
  • Mastery of:
  • models (staging, intermediate, marts)
  • ref() and source()
  • macros (Jinja)
  • seeds and snapshots
  • tests (not null, unique, custom)
  • Layered organization:
  • Staging → Transform → Mart (Data Warehouse)
  • Google Cloud Platform (GCP):
  • BigQuery: Deep expertise in data modeling, query optimization, partitioning, clustering, data loading (streaming and batch), security, and data governance.
  • Cloud Storage: Experience managing buckets, storage classes, lifecycle policies, access control (IAM), and data security.
  • Dataproc: Skills in provisioning, configuring, and managing Spark/Hadoop clusters, optimizing jobs, and integrating with other GCP services.
  • Dataflow/Composer/DBT: Knowledge of orchestration and data-processing tools for ELT/ETL pipelines.
  • Cloud IAM (Identity and Access Management): Implementing security policies and fine-grained access control.
  • VPC, Networking and Security: Understanding networks, subnets, firewall rules, and cloud security best practices.
  • Programming Languages:
  • Python and PySpark: Essential for automation scripts, building data pipelines, and integrating with GCP APIs.
  • SQL (advanced): For BigQuery, DBT, and data transformations.
  • Shell Scripting: For task automation.
  • Version Control:
  • Git/GitHub/Bitbucket.

Benefits

  • 🏥 Porto Seguro Health Insurance
  • 🦷 Porto Seguro Dental Insurance
  • 💰 Profit Sharing (PLR)
  • 👶 Childcare Allowance
  • 🍽️ Alelo Food and Meal Vouchers
  • 💻 Home Office Allowance
  • 📚 Partnerships with Educational Institutions
  • 🚀 Support for Certifications, including Cloud
  • 🎁 Livelo Points
  • 🏋️‍♂️ TotalPass
  • 🧘‍♂️ Mindself

Related Categories

Related Job Pages

More Data Engineer Jobs

Role Description The Postdoctoral Fellow supports scientific aspects of research and evaluation projects including: - Proposal development and research design - Defining target population - Identifying data needed - Measurement tool development - Data collection - Data analysis - Deliverable development The fellowship is anticipated to be a two-year period, with the initial appointment lasting one year and possible renewal for a second year based on performance. This position will have the opportunity to support the development of journal articles and the possibility of applying for funding. Qualifications - Doctoral degree in nutrition, public health or similar health-related field - Skilled in program evaluation; experience evaluating nutrition-related programs - Strong interpersonal skills; experience collaborating and communicating with clients and funders - Proficient in quantitative and qualitative research methods and data analysis - Strong written and verbal communication skills; history of scientific presentations and peer-reviewed publications - Proficient in data collection and analysis tools such as SurveyMonkey, Qualtrics, NVivo, SPSS, and SAS - Proficient in Microsoft Office (Word, Excel, PowerPoint, and Outlook) and virtual meeting platforms (e.g., Zoom) Requirements - Maintain a productive, comfortable, and distraction-free remote work environment - Comply with the CNHI work schedule - Follow care and security standards for CNHI-issued equipment - Supply your own Internet connection, allowing for seamless communication for video conferencing Physical Requirements The physical demands described are representative of those that must be met by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. Listed below are the physical requirements required while performing the duties of this job: - Regularly required to stand, sit, talk, hear, use hands and fingers to operate a computer and telephone keyboard, and reach, stoop and/or kneel to install computer equipment for periods of time throughout the day - Specific vision ability to complete close vision requirements due to computer work - Ability to complete light to moderate lifting Benefits - Salary: $74,000-$79,000, commensurate with experience - Competitive benefits package provided Job Type - This is a full-time, remote position Travel - This position may include travel for client visits, conference attendance, other promotional or networking activities, and the annual all-staff retreat How to Apply Please apply through the CNHI career page: www.centerfornutrition.org/careers . Include a statement of interest and resume/CV. Applications will be reviewed on a rolling basis. Please mention any foreign language skills in your materials.

United States
$74K - $79K / year
Diversified Services Network, Inc. logo

Data Engineer

Diversified Services Network, Inc.

Please follow the link to our website for a list of job openings in Engineering, IT, Project Management, and more! https://www.dsnworldwide.com Salary expectations: 100,000-107,000 per annual.

Data Engineer73 days ago

Diversified Services Network, Inc. (DSN) is seeking a full-time Data Engineer to join our team in 100% Remote or open to all CAT hubs (Chicago, Dallas, Peoria, Champaign, Westminster! We offer full benefits, PTO, 401k, and more! If you're looking to grow your technical career within an extremely reputable, stable Fortune 500 company - let's talk!   JOB RESPONSIBILITIES:                  ·         Perform all necessary data related tasks which could include data design, data quality, data triage, data governance or data architecture – SQL, Snowflake, AWS ·         Develop a break/fix solution and address the root cause in the data pipeline implementation/code – Python, AWS ·         Develop scripts and automation tools to better detect and correct data issues. ·         Develop monitoring and alerting capabilities to proactively detect data issues ·         Works directly on complex application/technical problem identification and resolution, including responding to off-shift and weekend support calls. ·         This position will contribute significantly to the efficiency and effectiveness of our data operations, enabling the team to deliver high-quality data solutions and support to various business units. ·         Identify, investigate, and obtain resolution commitments for platform and data issues to maintain and improve the quality and performance of assigned digital product data. ·         Issue Identification: Reports in all forms from customers, dealers, industry representatives, and subsidiaries. ·         Issue Investigation: Statistical analysis, data triage, and infrastructure problem-solving. ·         Issue Resolution: Identify root causes, create SageMaker scripts to fix data, and perform break/fix tasks on data pipeline code. ·         Develop scripts and automation tools to better detect and correct data issues. ·         Develop monitoring and alerting capabilities to proactively detect data issues. ·         Work directly on complex application and technical problem identification and resolution, including responding to off-shift and weekend support calls. ·         Communicate with end users and internal customers to help direct the development, debugging, and testing of application software for accuracy, integrity, interoperability, and completeness.

Colorado
Job Closed
Diversified Services Network, Inc. logo

Data Engineer

Diversified Services Network, Inc.

Please follow the link to our website for a list of job openings in Engineering, IT, Project Management, and more! https://www.dsnworldwide.com Salary expectations: 100,000-107,000 per annual.

Data Engineer73 days ago

Diversified Services Network, Inc. (DSN) is seeking a full-time Data Engineer to join our team in 100% Remote or open to all CAT hubs (Chicago, Dallas, Peoria, Champaign, Westminster! We offer full benefits, PTO, 401k, and more! If you're looking to grow your technical career within an extremely reputable, stable Fortune 500 company - let's talk!   JOB RESPONSIBILITIES:                  ·         Perform all necessary data related tasks which could include data design, data quality, data triage, data governance or data architecture – SQL, Snowflake, AWS ·         Develop a break/fix solution and address the root cause in the data pipeline implementation/code – Python, AWS ·         Develop scripts and automation tools to better detect and correct data issues. ·         Develop monitoring and alerting capabilities to proactively detect data issues ·         Works directly on complex application/technical problem identification and resolution, including responding to off-shift and weekend support calls. ·         This position will contribute significantly to the efficiency and effectiveness of our data operations, enabling the team to deliver high-quality data solutions and support to various business units. ·         Identify, investigate, and obtain resolution commitments for platform and data issues to maintain and improve the quality and performance of assigned digital product data. ·         Issue Identification: Reports in all forms from customers, dealers, industry representatives, and subsidiaries. ·         Issue Investigation: Statistical analysis, data triage, and infrastructure problem-solving. ·         Issue Resolution: Identify root causes, create SageMaker scripts to fix data, and perform break/fix tasks on data pipeline code. ·         Develop scripts and automation tools to better detect and correct data issues. ·         Develop monitoring and alerting capabilities to proactively detect data issues. ·         Work directly on complex application and technical problem identification and resolution, including responding to off-shift and weekend support calls. ·         Communicate with end users and internal customers to help direct the development, debugging, and testing of application software for accuracy, integrity, interoperability, and completeness.

Illinois
Job Closed
Diversified Services Network, Inc. logo

Data Engineer

Diversified Services Network, Inc.

Please follow the link to our website for a list of job openings in Engineering, IT, Project Management, and more! https://www.dsnworldwide.com Salary expectations: 100,000-107,000 per annual.

Data Engineer73 days ago

Diversified Services Network, Inc. (DSN) is seeking a full-time Data Engineer to join our team in 100% Remote or open to all CAT hubs (Chicago, Dallas, Peoria, Champaign, Westminster! We offer full benefits, PTO, 401k, and more! If you're looking to grow your technical career within an extremely reputable, stable Fortune 500 company - let's talk!   JOB RESPONSIBILITIES:                  ·         Perform all necessary data related tasks which could include data design, data quality, data triage, data governance or data architecture – SQL, Snowflake, AWS ·         Develop a break/fix solution and address the root cause in the data pipeline implementation/code – Python, AWS ·         Develop scripts and automation tools to better detect and correct data issues. ·         Develop monitoring and alerting capabilities to proactively detect data issues ·         Works directly on complex application/technical problem identification and resolution, including responding to off-shift and weekend support calls. ·         This position will contribute significantly to the efficiency and effectiveness of our data operations, enabling the team to deliver high-quality data solutions and support to various business units. ·         Identify, investigate, and obtain resolution commitments for platform and data issues to maintain and improve the quality and performance of assigned digital product data. ·         Issue Identification: Reports in all forms from customers, dealers, industry representatives, and subsidiaries. ·         Issue Investigation: Statistical analysis, data triage, and infrastructure problem-solving. ·         Issue Resolution: Identify root causes, create SageMaker scripts to fix data, and perform break/fix tasks on data pipeline code. ·         Develop scripts and automation tools to better detect and correct data issues. ·         Develop monitoring and alerting capabilities to proactively detect data issues. ·         Work directly on complex application and technical problem identification and resolution, including responding to off-shift and weekend support calls. ·         Communicate with end users and internal customers to help direct the development, debugging, and testing of application software for accuracy, integrity, interoperability, and completeness.

Illinois
Job Closed