Data Engineer, Azure/Databricks – Senior
Location
Brazil
Posted
13 days ago
Salary
0
Seniority
Senior
Job Description
Data Engineer, Azure/Databricks – Senior
Compass
• Work on requirements gathering, backlog refinement, support for information architecture, technical documentation and collaboration with technical and business areas; • Development and maintenance of data solutions applied to the Climate context, involving integration, organization, processing and provisioning of information from various corporate and industrial sources; • Preparation and maintenance of technical documentation in Markdown, Word or SharePoint; • Development of data pipelines, support for information architecture, integration with SAP systems, APIs and databases, as well as structuring information for analysis, traceability and decision support; • Work in a multidisciplinary environment, with constant interaction between technical and business areas, participating in understanding requirements, refining needs, building solutions, producing technical documentation and monitoring deliveries throughout the development, evolution and maintenance lifecycle; • Integration and processing of data from REST APIs, RDBMS/NoSQL databases, SAP ERP/SAP Analytics environments and relational stores such as SQL Server, Oracle or PostgreSQL, using tools like SSMS, DBeaver, Postman and Insomnia;
Job Requirements
- Strong experience in Python, SQL, PySpark and Databricks, plus advanced knowledge of Azure Data Factory for development, orchestration and maintenance of data pipelines;
- Experience handling RDBMS and NoSQL databases, using tools like SSMS and DBeaver, as well as experience with traditional relational databases such as SQL Server, Oracle or PostgreSQL;
- Knowledge of SAP ERP and SAP Analytics environments, including Datasphere, BW and BO;
- Experience integrating data via REST APIs, using tools like Postman or Insomnia for connectivity testing;
- Experience with code versioning using Git, in environments such as GitHub and Azure DevOps;
- Familiarity with tracking and collaboration tools such as Jira, ServiceNow SPM, Teams and Miro;
- Experience with version control in Git/GitHub/Azure DevOps;
- Desirable experience with streaming, data modeling, data quality and validation, automated testing, Cognite Data Fusion, Dataiku, sensors, time series and IoT;
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• A specialist who will lead the creation and management of the data pipelines that feed the project's analytical models. • Responsible for ensuring the quality, integrity, and availability of data for training and inference of Machine Learning models in production. • Expected to mentor other team members and define data engineering and MLOps strategies.
IT Data Platform Engineer 2
EvergenEvergen is a global industry-leading contract development and manufacturing organization (CDMO) in regenerative medicine. As the only regenerative medicine company that offers a differentiated portfolio of allograft and xenograft biomaterials at scale, Evergen is headquartered in Alachua, FL, and has manufacturing facilities in West Lafayette, IN., Eden Prairie and Glencoe, MN., Neunkirchen, DE., Glasgow, UK., and Marton, NZ.
Role Description We are looking for a hands-on Data Platform Engineer to own, operate, and evolve our modern cloud data stack. You will be the primary technical owner of our data infrastructure — responsible for keeping data flowing reliably from source systems into Snowflake and ensuring clean, trusted data reaches our business teams through Power BI. This is a high-impact role on a small, focused team where your work will be directly visible to the business. What You Will Work On - DATA INGESTION - Own and manage Fivetran connectors across all source systems including NetSuite, HubSpot, ADP, SQL Server, SAP HANA, and SharePoint. - Configure and monitor sync schedules, column exclusions, and incremental load strategies to control cost and reliability. - Troubleshoot connector failures and proactively manage schema drift from upstream sources. - DATA TRANSFORMATION - Maintain and extend our dbt project across three layers: staging (L1), core dimensions and facts (L2), and business-ready marts (L3). - Write and optimize SQL models using incremental merge strategies and watermark patterns. - Author and maintain dbt tests, model documentation, and source freshness checks to ensure data quality. - Support the buildout of our finance EDW including GL activity, planning data from Workday Adaptive, and NetSuite financials. - ORCHESTRATION & PIPELINE OPERATIONS - Manage end-to-end pipeline scheduling and monitoring, ensuring daily refreshes complete reliably before business hours. - Maintain the integration between Fivetran, dbt, and Power BI dataset refresh triggers. - Build and maintain alerting so pipeline failures are caught and communicated before the business is impacted. - DATA WAREHOUSE & GOVERNANCE - Manage Snowflake environments including databases, schemas, roles, warehouses, and cost controls. - Implement and maintain access controls and role-based permissions across the data platform. - Contribute to data catalog and lineage documentation to support a growing team and reduce knowledge concentration risk. - COLLABORATION - Partner with Finance, Sales, and Operations teams to understand reporting requirements and translate them into reliable data models. - Support and mentor the junior member of the data team as they develop their skills. - Work closely with the incoming NetSuite implementation team to ensure clean data integration into the warehouse. Qualifications - 3 to 5 years of experience in data engineering, analytics engineering, or a closely related role. - Hands-on experience with dbt (Core or Cloud) including incremental models, tests, macros, and documentation. - Proficiency with Snowflake including schema design, query optimization, warehouses, and role-based access control. - Experience with a managed ingestion tool such as Fivetran and dlt including connector configuration and monitoring. - Strong SQL skills with the ability to write and debug complex analytical queries. - Familiarity with ELT pipeline patterns and medallion-style data warehouse architecture. - Experience troubleshooting pipeline failures independently and communicating issues clearly to non-technical stakeholders. - Comfort working autonomously in a small team environment with limited oversight. Requirements - Experience connecting to on-premises source systems (SQL Server, SAP HANA, Oracle) via ODBC or CDC tooling. - Familiarity with ERP financial data in NetSuite, SAP, or similar, particularly GL structures and chart of accounts. - Exposure to Power BI including dataset refresh management and understanding of how semantic models consume warehouse data. - Experience with Git-based workflows and basic CI/CD practices for data projects. - Prior involvement in an EDW build or dimensional modeling project (star schema, slowly changing dimensions). What Success Looks Like - In your first 30 days: - You have completed a full walkthrough of the existing stack with our departing team member. - You can independently run, monitor, and troubleshoot the daily pipeline end to end. - You have documented any gaps or risks you have identified in the current setup. - In your first 90 days: - All Fivetran connectors are live and the dlt migration is complete. - dbt is running in dbt Cloud or Snowflake Workspaces with jobs, alerting, and documentation in place. - The Finance team has improved confidence in the GL data flowing through the warehouse. - In your first year: - The data platform is running reliably with minimal intervention and strong business trust. - The NetSuite data integration is live and the finance EDW is serving reporting needs. - You have grown the junior team member's capability and reduced single-person dependency on yourself. Our Stack - Ingestion: Fivetran, dlt - Warehouse: Snowflake - Transformation: dbt (Cloud or Snowflake-native) - Reporting: Power BI - Sources: NetSuite, HubSpot, ADP, SQL Server, SAP HANA, Workday Adaptive, SharePoint, Oracle DB Company Description Evergen is a global industry-leading contract development and manufacturing organization (CDMO) in regenerative medicine. As the only regenerative medicine company that offers a differentiated portfolio of allograft and xenograft biomaterials at scale, Evergen is headquartered in Alachua, FL, and has manufacturing facilities in West Lafayette, IN., Eden Prairie and Glencoe, MN., Neunkirchen, DE., Glasgow, UK., and Marton, NZ.
Role Description We are looking for individuals to record natural conversations using provided scripts. You will work with a partner and play a specific role (such as Customer or Support Agent). The goal is to create clear, natural-sounding audio recordings by following simple instructions. This project involves recording natural conversations with a partner using provided scripts. You will play a role (such as a Customer or Support Agent) and speak in a clear, natural way. The conversations are based on customer support situations, where one person helps solve a problem. These may include: - Verifying account details - Solving basic technical issues - Explaining simple policies Each assignment takes approximately 30 to 90 minutes and must be completed in one continuous session. You must record in a quiet, noise-free environment. Qualifications - Fluent in Brazilian Portuguese and English - Good communication skills in the required language(s) - Ability to follow instructions - Attention to detail - Willingness to work with a partner - Ability to meet deadlines Requirements - Speak naturally (like a real conversation, not robotic) - Follow all instructions carefully - Ensure clear audio with no background noise - Be available to re-record if needed - Computer or laptop with internet access - Working microphone - Basic knowledge of: - Google Sheets - Google Drive - Zencastr - Ability to download, rename, and upload files Payment - You will be paid for up to 1.5 hours per assignment, based on approved work. - Payment depends on audio quality and following instructions. - Work that does not meet quality standards may be rejected or require re-recording. - Payment is processed after successful quality review. - $4.00 USD per hour. Agreement & Acknowledgement Selected participants will be required to sign an agreement and acknowledgement form confirming: - Understanding of the task and instructions - Acceptance of quality and payment terms - Consent to use of recorded data for the project
Senior Cloud Data Engineer, AWS, GCP, Databricks
Future ProcessingGreat software... because we put people first
• odpowiedzialność za całość rozwiązań współtworzonych wraz z zespołem • tworzenie lub modyfikowanie rozwiązań do przetwarzania danych w chmurze • tworzenie i modyfikowanie dokumentacji • analizowanie i optymalizowanie rozwiązań w zakresie działającego lub projektowanego systemu • analizowanie wymagań klienta pod kątem dostarczenia optymalnego rozwiązania jego potrzeby biznesowej • analizowanie potencjalnych zagrożeń • dostosowywanie rozwiązań względem wymagań biznesowych • testowanie rozwiązań.



