Data Engineer, Databricks
Location
Brazil
Posted
3 days ago
Salary
0
Seniority
Senior
Job Description
Data Engineer, Databricks
Compass
• Collaborate with data engineering teams to understand requirements and deliver efficient solutions; • Develop and maintain Azure, Databricks (PySpark), Python, Data Lake, and SQL technologies; • Use Python ElementTree for efficient XML data handling and integration of heterogeneous data; • Develop Python scripts using Pandas for manipulation and analysis of structured data; • Ensure availability, stability and continuous evolution of client systems, ensuring high performance; • Work on security and compliance for business requirements through proactive management, process automation, and continuous improvement of support services; • Maintain data quality and integrity by implementing testing and monitoring practices; • Keep comprehensive technical documentation for processes and solutions implemented; • Support corrective actions, assist with production deliveries and monitor change requests (CHGs);
Job Requirements
- Experience with Databricks, PySpark and Python: foundational knowledge of these technologies;
- Basic knowledge of SQL and regular expressions (Regex);
- Development experience with SQL data modeling and basic distributed computing;
- Availability for on-call duty outside business hours and on weekends;
- Familiarity with text manipulation in programming and data analysis, particularly regular expressions (Regex);
- Experience with Data Lake;
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Senior AWS Data Engineer / Lead Data Architect
Naveera Technology LLCEngineering Production-Ready Data, AI & Cloud Platforms - Scalable, Secure, and Built for Enterprise Growth.
• Define and implement end-to-end Data Lakehouse solutions on AWS. • Lead the automation of cloud infrastructure using Terraform. • Orchestrate large-scale performance tuning initiatives. • Establish automated Data Quality gates using AWS Glue Data Quality. • Design complex, event-driven workflows using Step Functions and Airflow. • Serve as the primary technical liaison between Data Science, BI teams, and Business Stakeholders.
• Lead a distributed engineering team across platform/product engineering, connectors, QA, DevOps/infra, AI implementation, data governance, and support. • Own delivery, quality, release discipline, and execution of the technical roadmap. • Install engineering discipline where it's thin: automated testing, QA, CI/CD, release governance, and versioning. • Drive repeatable, perimeter-safe deployments, including containerization, infrastructure-as-code, secure deployment, and SOC 2 readiness. • Partner with the Chief Architect on the connector framework, canonical metadata model, architecture decisions, and product IP. • Build the team: assess current talent, retain the strong, hire the gaps, and align the team plan to the roadmap. • Push practical AI use across coding, review, testing, ops, and engineering productivity.
• Own the end-to-end product design vision for a data-dense enterprise platform. • Translate complex lineage, migration impact, governance, and blast-radius workflows into clear, navigable product experiences. • Partner closely with Product and Engineering to define what gets built, for whom, and why. • Design for two audiences at once: technical users who need depth and senior stakeholders who need confidence in decisions. • Build and maintain the design system, set the quality bar, and create repeatable standards for future product surfaces. • Run user discovery and research; turn what you learn into product direction, design decisions, and prioritization input. • Use AI across the design workflow, including research synthesis, ideation, prototyping, iteration, and productivity.
• Define the multi-year vision for the Data Engineering Practice, ensuring our technical capabilities are ahead of the curve for enterprise demand for Data & AI transformation. • Own the full P&L, including pipeline, pricing, delivery margin, and revenue growth, while reporting directly to executive leadership with clear commercial accountability. • Build and sustain senior relationships with Google Cloud partner teams and client executives to generate meaningful deal flow and expand strategic accounts. • Lead the practice's position on ethical AI and data democratisation while making principled, evidence-based bets on where advanced analytics and AI are creating real enterprise value. • Drive investment in reusable IP, data accelerators, and delivery assets that improve consistency, reduce time-to-value, and protect margins at scale. • Serve as a strategic advisor to client C-suite executives, helping them define data strategy, navigate AI adoption, and build the organisational capabilities to sustain it. • Set the standard for engineering and analytical excellence across the practice by hiring well, developing talent deliberately, and building teams that clients trust and return to.



