Helping Visionaries Change the World
Middle Data Engineer (Azure Databricks)
Location
Poland
Posted
78 days ago
Salary
0
Seniority
Mid Level
Job Description
Middle Data Engineer (Azure Databricks)
Miratech
Company Description Miratech helps visionaries change the world. We are a global IT services and consulting company that brings together enterprise and start-up innovation. Today, we support digital transformation for some of the world's largest enterprises. By partnering with both large and small players, we stay at the leading edge of technology, remain nimble even as a global leader, and create technology that helps our clients further enhance their business. We are a values-driven organization and our culture of Relentless Performance has enabled over 99% of Miratech's engagements to succeed by meeting or exceeding our scope, schedule, and/or budget objectives since our inception in 1989. Miratech has coverage across 5 continents and operates in over 25 countries around the world. Miratech retains nearly 1000 full-time professionals, and our annual growth rate exceeds 25%. Job Description We are looking for a Middle Data Engineer specialized in Azure Databricks to join our data platform team. The candidate will design and develop modern data pipelines and Lakehouse architectures, leveraging Azure Databricks, Spark, and Azure Data Factory, while integrating with existing SQL Server-based data warehouse environments, also evolving our data platform towards scalable, cloud-based data architectures, enabling advanced analytics and business intelligence. Key responsibilities: - Design, develop, and maintain data pipelines using Azure Databricks - Build and optimize data transformations using PySpark and SQL in Databricks - Implement and maintain Lakehouse architectures using Delta Lake - Develop ETL/ELT pipelines orchestrated through Azure Data Factory - Integrate data from multiple sources into the data platform and analytical layers - Design and maintain data models and data warehouse structures for analytics - Ensure data quality, scalability, and performance of large-scale data processing pipelines - Collaborate with BI teams to support Power BI and reporting platforms - Support and evolve existing SQL Server data platforms and ETL solutions (SSIS) when required - Contribute to the design of modern cloud-based data architectures Qualifications - 3+ years of experience in Data Engineering or Data Warehouse development - Experience with Azure Databricks - Experience developing data pipelines using PySpark and Spark SQL - Solid understanding of distributed data processing and big data concepts - Experience working with Delta Lake and Lakehouse architectures - Strong SQL skills and experience with SQL Server relational databases - Experience building data pipelines using Azure Data Factory - Experience handling large datasets and performance optimization Nice to have - Experience with Spark optimization techniques (partitioning, caching, cluster tuning) - Experience with structured streaming in Databricks - Knowledge of CI/CD pipelines for data platforms (Azure Devops) - Familiarity with Power BI - Experience in migrating from traditional ETL process to cloud architectures Soft Skills - Strong analytical and problem-solving skills. - Ability to work in collaborative environments and to adapt - Committed to continuous learning and professional development, with a keen focus on advancing cloud computing expertise - Team player - Good communication skills with technical and non-technical roles - Is proactive rather than reactive, actively identifying improvements and proposing solutions - Is comfortable working in dynamic environments where priorities and technologies evolve Additional Information We offer: - Culture of Relentless Performance: join an unstoppable technology development team with a 99% project success rate and more than 30% year-over-year revenue growth. - Competitive Pay and Benefits: enjoy a comprehensive compensation and benefits package, including health insurance, and a relocation program. - Work From Anywhere Culture: make the most of the flexibility that comes with remote work. - Growth Mindset: reap the benefits of a range of professional development opportunities, including certification programs, mentorship and talent investment programs, internal mobility and internship opportunities. - Global Impact: collaborate on impactful projects for top global clients and shape the future of industries. - Welcoming Multicultural Environment: be a part of a dynamic, global team and thrive in an inclusive and supportive work environment with open communication and regular team-building company social events. - Social Sustainability Values: join our sustainable business practices focused on five pillars, including IT education, community empowerment, fair operating practices, environmental sustainability, and gender equality. * Miratech is an equal opportunity employer and does not discriminate against any employee or applicant for employment on the basis of race, color, religion, sex, national origin, age, disability, veteran status, sexual orientation, gender identity, or any other protected status under applicable law.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Big Data Engineer
ForhyreWe are a leading healthcare staffing company dedicated to providing high-quality nursing talent to healthcare facilities nationwide. Our mission is to connect healthcare organizations with exceptional nurses, ensuring the best patient care possible.
Role Description Looking for an experienced Senior Big Data Developer who will be responsible for: - Technical leadership in driving solutions and hands-on contributions - Building new cloud-based ingestion, transformation, and data movement applications - Migrating/modernizing legacy data platforms - Contributing and assisting in translating requirements to high-level and low-level solution designs and working program/code - Interacting with business/IT stakeholders and other involved teams to understand requirements, identify dependencies, and suggest solutions - Performing hands-on work to deliver on commitments and coordinating with team members both onsite and offshore Qualifications - 8 - 10 years of experience - Good working knowledge and strong concepts on Spark framework - Comfortably work with one or more of the scripting languages listed in order of preference: Scala, Python, Unix shell scripting - Experience/exposure in AWS Services (EC2, Lambda, S3, RDS) and related cloud technologies - Good understanding of DATA space - Data Integration, Building Data Warehouse solutions Requirements - Primary / Essential Skills: SPARK with Scala or Python - Secondary / Optional Skills: AWS, UNIX & SQL
• Build & Maintain Pipelines: Design, build, and operate end-to-end data solutions , including both real-time and batch data pipelines. Ensure data flows correctly across architectures, from raw ingestion to modeling-ready layers. • Cross-Cloud Engineering: Leverage expertise across multiple cloud environments to build robust data ecosystems, whether utilizing AWS (S3 and core cloud services) , GCP (BigQuery and Google Cloud Storage) , or Azure (Databricks). • Automate & Orchestrate: Use deep knowledge of Apache Airflow for scheduling, orchestrating, and monitoring workflows. Build software services and automate processes using strong Python development skills. • Advanced Data Modeling: Apply the correct methodologies (Kimball, Inmon, Data Vault) to design Star Schemas, Semantic Models, and highly optimized platforms.
• Be a data visionary, anticipating future data needs • Influence AI model training through high-quality data • Own data end-to-end: sourcing, structuring, scaling • Source and curate multimodal data (text, video, images) • Master video data challenges for ML training • Optimize labeling and automation workflows • Unlock value from internal platform data • Balance speed with precision
Senior Data Quality Engineer (m/f/d)
TecAllianceTecAlliance provides digital solutions to the automotive market, specializing in spare parts and repair services. Founded in 1994, TecAlliance was created to enhance connectivity a
We are TecAlliance — founded by more than 30 automotive aftermarket companies as a neutral Data‑as‑a‑Service market connector, serving as the neutral data terrain that interconnects them. With over 1,000 employees worldwide and customers in 140 countries, we enable the market — we never compete with it. 📍In this position, you will become part of our Information Management Digitalization Tools Team The team acts as the internal IT service provider for the TecAlliance Information Management Digitalization Tools. They are a diverse, global team of 40+ professionals across Europe and Vietnam. They work in small, agile teams, continuously improving how they deliver value. Their mission: support the IM Data Factory by delivering robust IT solutions, optimizing processes, and reducing manual effort in the critical process of capturing and standardizing vehicle technical data. The global data platform, built on AWS technologies like Glue and DataZone, designed to process hundreds of data flows that are critical to our products and customers. Now they're looking for Senior Data Quality Engineer (m/f/d). In this role, you will close the critical gap between Data Engineering and QA by driving an overarching QA strategy and implementing automated test data and quality management practices that enable reliable, scalable, and production-ready data pipelines across all projects. You’ll help shape how we validate data flows end-to-end - and thereby enabling a faster, safer delivery. Your responsibilities will include - Designing and implementing an automated testing framework for complex data processing pipelines. - Developing and maintaining a strategy for managing test data across various data flows and environments. - Preparing test environments by provisioning test data and setting the system to the correct state for execution. - Executing data flows and validate the results against expected outcomes. - Collaborating with data engineers to integrate testing into existing pipelines and workflows. - Applying tools to automate checks for data quality and consistency. Your Profile - Degree in Computer Science, Software Engineering, or a related field is highly welcomed. - Significant recent experience in testing data-driven systems, ideally in roles focused on data quality, data engineering or data platform testing. - Solid experience with testing ETL and data processing pipelines, including validation of complex data transformations. - Deep understanding of test data management in complex, dynamic environments, including strategies for provisioning, maintaining and automating test data across multiple flows. - Familiarity with AWS services such as Glue / Glue Data Quality, S3, Athena and DataZone. - Ability to orchestrate multi-step test processes across systems and environments. - Hands-on experience with data quality tools (e.g.: AWS Glue Data Quality / Great Expectations). - Fluency in English (C1), German, Dutch or Spanish is a plus. If you feel you can fulfil the demands of this role but do not meet all of the criteria above, we still encourage you to apply. Applications will be reviewed on their merit, please elaborate on why you feel you would be a good candidate for this position in your application. We kindly ask you to send in an English CV only. Thank you. Benefits We offer a wide range of benefits. As our positions are advertised in various locations the benefits may vary. Therefore, below you will only find those that apply to all locations—for details, please speak to the recruiter, who will be happy to answer your questions. Contracts, Salary, and non-monetary benefits 💯 Remote work: we work 80-100% remotely; however, we have regular on-site team events. 🚀 Structured Onboarding: you receive an individual onboarding plan, have multiple onboarding days in the beginning, and a wealth of e-learning, training, and documentation besides your team at your disposal. 👩💻 Set-up: for this we will provide you with a tailored tech set-up (Dell devices, the standard remote package includes a notebook, 2monitors, headset, mobile phone, mouse + keyboard, docking station). ⚖️ Balance job & life: Flexible working hours: you decide where and when you work. Culture ❤️🔥 Kununu Top Company since 2022: we're proud to state that our score currently sits at 4,2/5 score with an 85%+ recommendation rate. 🧬 We value ownership, cooperation, entrepreneurial thinking & and self-reflection in order to communicate effectively as ONE team. 🎟️ Team culture is Key! We have fun at work and beyond: There is always something to celebrate and regular team events. 🦄 Come as you are - do you prefer T-shirts over a shirt? Great. ✨ Your contribution matters: shape our value-driven culture and agile transformation together with your colleagues – we're curious, want to go one step further, build further trust together, and join forces to tackle our challenges as a team. * Please note - that TecAlliance can only consider candidates for employment who are legally authorized to work in a country listed in the job posting, where we have an established legal entity and payroll system. Unfortunately, we are currently unable to hire candidates who require relocation, visa support, or are located outside of that country. Of course you can apply, if you hold a work permit and are willing to / or already moved to the country. Thank you for your understanding. - that it is not possible to work outside of the country you are applying for. Meaning that, if you apply e.g. for Germany, you must work from within Germany. It is not possible to work from abroad. You can work at any location within the borders of the listed country, unless the job posting specifies a certain City. - 📕 Contract title: your contract title is “Senior Data Quality Engineer”. Please only apply with your CV (in English only) via the system. Applications outside of our applicant tracking system cannot be considered due to GDPR.


