Job Closed
This listing is no longer active.
Affordable energy and water for everyone.
Senior Data Engineer – Integration Hub, Data Pipelines
Location
India
Posted
125 days ago
Salary
0
Seniority
Senior
Job Description
Senior Data Engineer – Integration Hub, Data Pipelines
Cuculus GmbH
• Design, build, and maintain robust ETL/ELT data pipelines for batch and streaming workloads. • Implement data ingestion and transformation workflows using Apache Airflow, Apache NiFi, Apache Spark, and Kafka. • Integrate data from multiple sources including REST APIs, files, relational databases, message queues, and external SaaS platforms. • Optimize pipelines for performance, scalability, reliability, and cost efficiency. • Develop and operate a centralized data integration hub that supports multiple upstream and downstream systems. • Build reusable, modular integration components and frameworks. • Ensure high availability, fault tolerance, and observability of data workflows. • Design and manage data warehouses, data lakes, and operational data stores using PostgreSQL and related technologies. • Implement appropriate data modeling strategies for analytical and operational use cases. • Manage schema evolution, metadata, and versioning. • Implement data validation, monitoring, and reconciliation mechanisms to ensure data accuracy and completeness. • Enforce data security best practices, access controls, and compliance with internal governance policies. • Establish logging, alerting, and auditability across pipelines. • Automate data workflows, deployments, and operational processes to support scale and reliability. • Monitor pipelines proactively and troubleshoot production issues. • Improve CI/CD practices for data engineering workflows. • Work closely with data scientists, analysts, backend engineers, and business stakeholders to understand data requirements. • Translate business needs into technical data solutions.
Job Requirements
- 5+ years of hands-on experience as a Data Engineer or in a similar role.
- Proven experience as an individual contributor on at least three end-to-end data engineering projects, from design to production.
- Strong hands-on experience with: Apache Airflow / Dagster, Apache NiFi, Apache Spark, Apache Kafka, PostgreSQL.
- Extensive experience integrating data from APIs, files, databases, and third-party systems.
- Strong SQL skills and experience with data modeling.
- Solid programming experience in Python and/or Java/Scala.
- Experience with Linux environments and version control systems (Git).
- Strong problem-solving, debugging, and performance-tuning skills.
Benefits
- Professional development opportunities
- Flexible working arrangements
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Design and develop conceptual, logical, and physical data models for various domains • Lead the development of data modeling standards, best practices, and guidelines • Develop end-to-end solution architectures for data-driven and AI-focused applications • Mentor and guide junior data modelers • Design data models to support enterprise and operational reporting • Collaborate with data scientists to develop features for machine learning models • Optimize data models for performance and scalability • Ensure data models comply with data governance policies and standards • Identify and define data quality checks and validation processes • Implement data quality monitoring and reporting mechanisms
• Análisis, diseño y desarrollo de soluciones de ingeniería de datos en entornos cloud. • Construcción y mantenimiento de pipelines de datos. • Tratamiento, integración y análisis de grandes volúmenes de información. • Optimización de procesos de datos en plataformas Microsoft Azure. • Data engineering y data analysis.
Senior Azure Data Engineer
SmartbridgeSimplifying business transformation through thought leadership and innovation. Bring your digital agenda to reality.
• Define the target-state Azure data architecture (ingestion, orchestration, storage zones, serving patterns), security/networking boundaries, cost/perf tradeoffs, and promotion strategy (Dev→Test→Prod). • Implement robust ELT/ETL with ADF/Synapse Pipelines (parameters, reusable templates, CI/CD). • Hands-on in T-SQL and Python/PySpark for transformations, utilities, and tests. • Physical/semantic modeling, partitioning, columnstore strategies, statistics management, query plan analysis, index design, concurrency & transaction isolation, workload management. • SLA/SLO definitions, Azure Monitor / Log Analytics / App Insights dashboards and alerts; error handling, retries/backoff, idempotency, CDC and schema drift strategies. • RBAC, Key Vault, managed identities, private endpoints/VNet, data masking patterns; document data contracts and access patterns. • Code reviews, PR discipline, mentoring, and crisp documentation/runbooks for client handoff.
Senior Data Engineer
CAREWe work to fight poverty and achieve social justice by empowering women and girls. www.CARE.org
• Design and implement scalable, reliable, and efficient data pipelines to support clinical, operational, and business needs. • Develop and maintain architecture standards, reusable frameworks, and best practices across data engineering workflows. • Build automated systems for data ingestion, transformation, and orchestration leveraging cloud-native and open-source tools. • Optimize data storage and processing in data lakes and cloud data warehouses (Azure, Databricks). • Develop and monitor batch and streaming data processes to ensure data accuracy, consistency, and timeliness. • Maintain documentation and lineage tracking across datasets and pipelines to support transparency and governance. • Work cross-functionally with analysts, data scientists, software engineers, and business stakeholders to understand data requirements and deliver fit-for-purpose data solutions. • Review and refine work completed by other team members, ensuring quality and performance standards are met. • Provide technical mentorship to junior team members and collaborate with contractors and third-party vendors to extend engineering capacity. • Use Databricks, DBT, Azure Data Factory, and SQL to architect and deploy robust data engineering solutions. • Integrate APIs, structured/unstructured data sources, and third-party systems into centralized data platforms. • Evaluate and implement new technologies to enhance the scalability, observability, and automation of data operations. • Proactively suggest improvements to infrastructure, processes, and automation to improve system efficiency, reduce costs, and enhance performance.




