O hub de inovação do Grupo Mercantil.
Mid-level Data Engineer
Location
Brazil
Posted
121 days ago
Salary
0
Seniority
Senior
Job Description
Mid-level Data Engineer
Domo Inovação
• Actively support the definition of requirements for the IT/Data infrastructure area, contributing technical analyses of proposed solutions and alternatives and evaluating impacts on performance, cost, scalability, and alignment with the existing architecture; • Work with a moderate critical perspective, suggesting improvements and identifying technical risks in line with guidelines established by the senior team and corporate architecture; • Participate in the design and implementation of medium-complexity data architectures, following standards and guidelines defined by corporate architecture; • Collaborate with both technical and non-technical teams, translating complex concepts into accessible language to facilitate alignment and decision-making; • Actively contribute to project success by anticipating needs, supporting other areas, and participating in the resolution of technical disagreements with maturity and a results-oriented focus; • Collaborate in defining and extracting data from different sources for ingestion into Big Data platforms, ensuring adherence to established standards; • Support the modeling and organization of data structures to ensure they are appropriate, accessible, and available for consumption, ensuring alignment between operational data and the analytical models required; • Actively contribute to projects related to data and system integration, implementing medium-complexity solutions and ensuring compliance with defined architectural standards; • Support the technical design of solutions, participate in the evolution of the existing architecture, and work on the maintenance and continuous improvement of the systems under your responsibility.
Job Requirements
- Bachelor's degree in Computer Science, Information Systems, Systems Analysis and Development, or related fields;
- Practical experience developing and maintaining data pipelines;
- Strong knowledge of Python for ETL, data manipulation, and APIs (Pandas, NumPy, PySpark);
- Practical experience with Spark and/or Kafka;
- Knowledge of batch and streaming processing;
- Knowledge of Data Lake, Lakehouse, and Data Warehouse concepts;
- Proficiency in SQL and experience with relational databases;
- Basic knowledge of NoSQL databases;
- Familiarity with DataOps, versioning, and pipeline monitoring;
- Practical application of agile methodologies (Scrum, Kanban).
Benefits
- Health insurance and Dental plan
- Wellhub (Gympass)
- Commuter allowance (Vale Transporte)
- Meal and food vouchers
- Domo School (learning platform) and partnerships with educational institutions
- Private pension plan
- Life insurance
- Day off
- Extended maternity and paternity leave.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Responsible for integrating, analyzing raw data, developing, and maintaining datasets and improving data quality and efficiency. • Develops and maintains scalable data pipelines, integration tools and builds out new API integrations to support continuing increases in data usage, volume, and complexity. • Building analytical tools to utilize the data pipeline, providing actionable insight into key business performance metrics including operational efficiency and customer acquisition. • Designs data integrations, data quality framework and integrate it with monitoring services. • Strong hands on capabilities across integration toolsets including ADF, Replication, CDC etc. • Integrate data from different SOR’s, combine raw information and create resulting models ready for consumption. • Interpret trends and patterns. • Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues. • Identifying, designing, and implementing internal process improvements including re-designing infrastructure for greater scalability, DB, SQL, and data pipelines performance tuning, optimizing data delivery, and automating manual processes. • Build solution prototypes as needed and write algorithms against data. • Implements processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it. • Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility, and fostering data-driven decision making across the organization. • Works closely with a team of frontend and backend engineers, product managers, and analysts.
Data Engineer
LyftLyft, established in 2012 by Logan Green and John Zimmer, is a transportation network company offering a mobile application that promotes ride-sharing by connec
• Owner of the core data pipeline, responsible for scaling up data processing flow to meet the rapid data growth at Lyft • Evolve data model and data schema based on business and engineering needs • Implement systems tracking data quality and consistency • Develop tools supporting self-service data pipeline management (ETL) • SQL and MapReduce job tuning to improve data processing performance • Write well-crafted, well-tested, readable, maintainable code • Participate in code reviews to ensure code quality and distribute knowledge • Collaborate cross-functionally with product, engineering, data science, and marketing teams to understand business problems and align on prioritization and solutions
• Lead and manage data engineering projects from conception to deployment, ensuring alignment with stakeholder needs. • Design, build, and maintain scalable data pipelines to support data ingestion, transformation, and storage. • Collaborate with data scientists and analysts to ensure that data needs are met and provide technical guidance on best practices. • Implement and maintain data quality and integrity measures. • Utilize cloud-based data platforms and technologies, particularly Azure, to enhance data accessibility and usability. • Mentor and guide junior team members in data engineering concepts and practices. • Identify opportunities for process improvements and implement solutions that enhance efficiency, scalability, and performance.
• Apply an in-depth understanding of data structures and information content. • Select, deploy, and manage the systems and infrastructure required for a data processing pipeline in support of the project requirements. • Investigate, create, and maintain data flows, data content, data element definitions with a goal of enterprise Master data integration. • Determine technical breath in data profiling from different sources and determine whether and how data can support business and data requirements of its intended use. • Develop and maintain common business definitions and metadata criteria for consistent metrics reporting across the enterprise. • Design the architecture for new data and analytics platform to support analytics and data science and machine learning. • Design the data models and data movement processes that support analytics and data science. • Recommend and implement patterns and best practices for data engineering. • Ensure quality processes are built into the design of the platform. • Understand the architectural difference between solution approaches and communicate the advantages/disadvantages of your recommendation to both technical and non-technical audiences. • Design and develop analytics and interactive visualizations that create business insights and clearly communicate data and trends. • Develop complex SQL queries to obtain data from our source systems. • Perform data validation and quality assurance to ensure data integrity and accuracy. • Collaborate with IT and business partners to identify data sources and align data domains to authoritative sources of data. • Enforce tactical enforcement of Data Governance policies and rules. • Research new technologies while keeping up-to-date with technological developments in relevant areas of Data Governance, Master Data, and Data Quality.



