Building digital businesses, together.
Lead Data Engineer, Azure
Location
Guatemala
Posted
6 days ago
Salary
0
Seniority
Senior
Job Description
Lead Data Engineer, Azure
3Pillar Global
• Understand client requirements and lead the creation of robust data platforms leveraging Data Lakehouse architectures, serving as the primary subject matter expert for the Azure data ecosystem (Microsoft Fabric, Azure SQL, and Power BI). • Design and implement scalable data models, ETL/ELT pipelines, and analytics layers using modern cloud ecosystems. • Architect Microsoft Fabric reporting solutions, establishing medallion layering, managing capacity planning, ensuring freshness governance, and configuring CDC/mirroring from Azure SQL into Fabric. • Enforce strict data-layer segregation ensuring PII is excluded by construction, maintaining multi-tenant isolation within the analytics layer, and strictly adhering to 10+ year regulated data retention policies. • Lead complex migration efforts off legacy systems (e.g., AirTable, Smartsheet) to Microsoft Fabric, effectively managing mid-program form and rule drift, and establishing reliable, per-client export channels. • Lead end-to-end analytics projects — from requirement gathering to deployment. • Mentor junior team members, perform code reviews, and ensure adherence to quality standards.**Work closely with cross-functional teams (engineering, product, business) to align data strategy with business goals. • Develop and optimize complex SQL queries, Power BI semantic models, dashboards, and reports (transitioning from legacy LookML/BigQuery paradigms). • Troubleshoot performance bottlenecks, data inconsistencies, and integration challenges. • Stay up to date with advancements in the analytics and cloud data ecosystem. • Drive innovation in reporting and analytics capabilities.**Recommend and implement new tools, frameworks, and practices to enhance the data platform. • Define best practices, coding standards, and governance for analytics projects. • Monitor data systems performance and implement optimization strategies. • Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures. • Build and oversee AI-ready data pipelines, focused on feeding scrubbed, segmented data to assistive AI (embeddings, vector stores, and feature stores) and integrating GenAI into analytics and data products. • Drive adoption of AI-assisted data engineering across the team: copilots for SQL/transformation/pipeline code, plus automated testing and documentation. • Define governance and quality standards for data consumed by LLMs and agents.
Job Requirements
- Demonstrated expertise with a minimum of 7 - 10 years of relevant experience in data engineering and business intelligence domain.
- Strong primary expertise in the **Azure Data Stack, specifically Azure SQL, Microsoft Fabric**, and Power BI (semantic layer design, robust dashboard development), with a minimum of two years of hands-on experience in these specific tools.
- Advanced SQL skills and a solid understanding of modern data architecture, including **Data Lakehouse** concepts, medallion architecture, star/snowflake schemas, and dimensional modeling.
- Proficiency in Python** for data engineering, ETL/ELT workflows, and automation tasks.
- Experience designing and maintaining enterprise-grade **data pipelines** — specifically in Azure environments.
- Proven track record of **leading analytics projects** and delivering scalable data solutions.
- Ability to function as a **player-coach** — leading while also contributing hands-on.
- Solid understanding of **data governance, security** (especially around PII and multi-tenant architectures), and **performance optimization** practices.
- Strong communication skills and ability to translate complex technical concepts for business stakeholders.
Benefits
- Medical Insurance benefits as per company policy.
- Life Insurance as per company policy
- 15 days of paid vacation, sick leave and paid holidays as per local law
- Paternity and maternity leave as per as per local law
- Marriage, bereavement and graduation leaves as per company policy
- Sick leave and paid holidays as per local law
- Christmas and Middle year bonuses as per local law
- Monthly productivity bonus as per local law
- Discounts in local shops
- Direct deposit of payroll.
- Paid professional certifications
Related Guides
Related Categories
Related Job Pages
More Cloud Engineer Jobs
Security Engineer, Cloud
VercelVercel offers an industry-leading cloud technology platform used by developers and designers to make the web faster and more personalized. In past hiring, the c
• Design and implement scalable security controls across our cloud-native platform. • Harden infrastructure components using infrastructure-as-code, policy enforcement, and service isolation. • Build secure by default infrastructure and code CI/CD pipelines. • Collaborate with platform and infrastructure teams to integrate security best practices into architecture and workflows. • Stay ahead of cloud security trends and adopt cutting-edge technologies to enhance platform resilience. • Conduct threat modeling, risk analysis, and mitigation planning for critical systems. • Drive improvements in monitoring, detection, and incident response at the platform level. • Build, deploy and maintain relevant tooling.
• Work hands-on in the development and evolution of applications • Contribute to full-stack solutions with a primary focus on backend • Provide technical leadership for key initiatives • Influence the team through code reviews, mentoring and technical guidance • Design scalable solutions, balancing decisions across application, architecture and efficient use of cloud resources • Contribute to the development and evolution of services, integrations, automations and components required to support products and platforms • Work on resolving complex problems • Collaborate with Product and Engineering to define solutions • Contribute to the continuous improvement of engineering practices • Support decisions related to infrastructure, scalability and efficient use of cloud resources
Role Description At 1KOMMA5°, we connect and manage tens of thousands of distributed energy devices at scale. As a (Senior) IoT Cloud Engineer, you will own the cloud side of our IoT Platform that ingests, routes, and observes every message flowing between our gateway fleet and our cloud services. You will work at the intersection of cloud infrastructure, messaging systems, data and IoT engineering, making sure our device-to-cloud highway is reliable, secure, observable, and cost-efficient as the fleet grows. We are a Platform team: your work directly enables product teams across Heartbeat AI to ship features faster, while giving the business a clear, data-driven view of fleet health and performance. - Take ownership of our systems at the intersection of Cloud Services and residential IoT Gateways - Operate and evolve our Kubernetes-based IoT Cloud Platform, including the deployment, scaling, and hardening of our connectivity and fleet services - Develop and run our Eclipse Hono stack as the messaging backbone between gateways and the cloud - Design, build and operate features around fleet management enabling the lifecycle of gateways from production line to field - Integrate tightly with Mender to operate safe OTA rollouts across the fleet - Contribute to load and resilience tests to prove the platform scales and to find limits before our customers do - Build and own the observability pipelines for the IoT Platform and turn metrics into automated monitors, alerts, and self-healing mechanisms - Build data pipelines and BI models to turn raw data into fleet insights - Work collaboratively with product managers, data engineers, IoT engineers, as well as cloud engineers to understand problems, support discoveries, and create solutions enabling a scalable virtual power plant - Foster a culture of experiment-driven innovation within the team Technologies you will work with include: - Cloud & Infra: Google Cloud Platform, GKE (Kubernetes), Cloud Run, Terraform/IaC - Messaging & IoT: Eclipse Hono, MQTT, Mender - Data & BI: PostgreSQL, BigQuery, SQLMesh, Looker - Observability: Datadog, OpenTelemetry - Languages: TypeScript (Nest.js), Java, Go - DevOps: CI/CD pipelines (GitHub Actions) Qualifications - Bachelor's or Master's degree in Computer Science, Software Engineering, or related field - 5+ years of experience building and operating cloud / distributed backend systems in production - Strong hands-on experience with Kubernetes (ideally GKE) and modern cloud-native operations - Solid understanding of MQTT and device-to-cloud messaging patterns at scale, including back-pressure, QoS, and fan-out - Experience designing and running observability pipelines (logs, metrics, traces) for large-scale distributed systems, and turning them into actionable monitors - Comfortable working with data and BI tooling — writing SQL, modeling data, and building dashboards that drive decisions - Experience with CI/CD, IaC, and platform engineering practices - A strong reliability and security mindset - Ability to guide architectural decisions and navigate across embedded, hardware, data, and product teams - Agile, product-oriented mindset and a strong team player with a bias for action and ownership - Strong troubleshooting and analytical mindset - Clear and effective communicator in English (German is a plus) Requirements - Production experience with Eclipse Hono or a comparable IoT connectivity platform - Familiarity with Mender or other large-scale OTA / fleet management systems - Experience with SQLMesh and/or Looker (LookML) specifically - Familiarity with Google Cloud Platform as the primary cloud - Experience running load tests for messaging infrastructure or large-scale distributed systems - Passion for climate tech and distributed energy systems Benefits - You are part of an international, dynamic, and highly motivated team of people who have proven to make things happen - With your work, you accelerate the "energy transition" and hence have a direct impact on our climate - Work with and learn from other super-smart colleagues - You will enjoy direct contact with core decision-makers - You will enjoy the best chances of entering full-time in one of Europe’s most thriving scaleups - You work remotely (Germany-wide), with offices in Hamburg, Berlin or Munich - Create a healthy balance alongside your work and enjoy all the benefits of the EGYM Wellpass - Benefits and discounts are yours with Futurebens - Whether city bike or e-bike - be flexible with our job bike leasing and do something good for the environment at the same time
• Troubleshoot reported incidents and problems. • Provide support for any problems associated with the platform infrastructure for OnPrem, IaaS, PaaS, and SaaS. • Develop documentation and mentor Jr. team members in process development, documentation standards, and remediation processes. • Design technology solutions that meet technical requirements as defined by architectural documentation both on on-premise and cloud-based IaaS, PaaS, and SaaS. • Document and maintain an inventory of all technology platforms. • Install, configure, and maintain server software including configurations such as IP, SAN, OS, Policies, etc . • Monitor, administer and support all infrastructure technology platforms. • Monitor and maintain on-premise and cloud-based backup and recovery service as documented by Hanger policies and Standard Operating Procedures. • Develop and implement automated backup strategies for both on-premises and cloud infrastructure. • Automate the failover and fallback process to ensure reliability and availability using Azure Backup and Azure Site Recovery. • Works on complex issues where analysis of situations or data requires an in-depth and expert evaluation of variable factors including external service providers, administrative controls, architectural principals, and business objectives. • Provide subject matter expertise in the implementation and development of on-premise and cloud-based technology services across all technology delivery methods including IaaS, PaaS, and SaaS.




