We support enterprises, product houses, and startups with custom software solutions development and IT consulting.
Senior Data Engineer – GovTech, Public Sector
Location
Ukraine
Posted
66 days ago
Salary
0
Seniority
Senior
Job Description
Senior Data Engineer – GovTech, Public Sector
Sigma Software Group
• Design and implement data ingestion pipelines for batch and streaming data • Configure and maintain data orchestration workflows (Airflow, NiFi) and CI/CD automation for data processes • Design and organize data layers within Data Lake architecture (HDFS, Iceberg, S3) • Build and maintain secure and governed data environments using Apache Ranger, Atlas, and SDX • Develop SQL queries and optimize performance for analytical workloads in Hive/Impala • Collaborate on data modeling for analytics and BI, ensuring clean schemas and dimensional models • Support machine learning workflows using Spark MLlib or Cloudera Machine Learning (CML)
Job Requirements
- Proven experience in building and maintaining large-scale data pipelines (batch and streaming)
- Strong knowledge of data engineering fundamentals: ETL/ELT, data governance, data warehousing, Medallion architecture
- Strong SQL skills for Data Warehouse data serving
- Minimum 3 years of experience in Python or Scala for data processing
- Hands-on experience with Apache Spark, Kafka, Airflow, and distributed systems optimization
- Experience with Apache Ranger and Atlas for security and metadata management
- Upper-Intermediate English proficiency
- BS in Computer Science or related field
Benefits
- remote work options
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Implement ETL/ELT solutions and data integration between multiple systems and data sources. • Design, implement, and maintain data pipelines to ingest, store, and process large volumes of data. • Collaborate with other teams to ensure data security and compliance. • Design, develop, and implement solutions to optimize the performance and scalability of data processing systems.
Staff Software Engineer, Data Engineering
AirbnbAirbnb is a community based on connection and belonging.
• Architect and productionize batch and real-time data systems to support various products and business needs. • Ensure the quality, performance, and stability of data systems through robust quality systems and monitoring practices. • Design and optimize data models for efficient storage and retrieval to meet critical product and business requirements. • Collaborate with cross-functional teams, including product managers, engineers, data scientists, and business partners, to align on data requirements and develop scalable systems. • Tune, productionize, and optimize data systems and machine learning models to enhance their effectiveness and efficiency. • Build and maintain strong relationships with partner engineering teams, including backend, client, data science, and ML teams, to ensure seamless integration and support. • Contribute to the development of long-term data strategies and roadmaps, and influence the technical direction of data engineering practices within the organization. • Mentor and coach team members, providing guidance and support to enhance their skills and performance.
Senior Software Engineer, Data Engineering
AirbnbAirbnb is a community based on connection and belonging.
• Design, build, and maintain robust and efficient data pipelines that collect, process, and store data from various sources, including user interactions, listing details, and external data feeds. • Develop data models that enable the efficient analysis and manipulation of data for merchandising optimization. Ensure data quality, consistency, and accuracy. • Build scalable data pipelines (SparkSQL & Scala) leveraging Airflow scheduler/executor framework • Collaborate with cross-functional teams, including Data Scientists, Product Managers, and Software Engineers, to define data requirements, and deliver data solutions that drive merchandising and sales improvements. • Contribute to the broader Data Engineering community at Airbnb to influence tooling and standards to improve culture and productivity • Improve code and data quality by leveraging and contributing to internal tools to automatically detect and mitigate issues
Ingeniero de datos, contabilidad y finanzas
XpertGroupAcompañamos a nuestros clientes en su proceso de transformación digital a través de soluciones ágiles y de valor.
• Desarrollar e Implementar Modelos Analíticos con Python, Flask o Django, Shell Linux, PostgreSQL o MySQL, SOLID. • Desarrollar modelos analíticos que involucren librerías de Python Avanzado como Numpy, Pandas, Scikit-Learn, Tensorflow, NLTK. • Extraer, procesar y consumir la información desde los repositorios de datos. • Crear el roadmap de los modelos analíticos que serán ejecutados. • Participar en el desarrollo de proyectos de analítica. • Trabajar con Lenguaje Fuente abierta. Python (pandas y matplot is), Open Refine remediación de datos normalización, SQL, algoritmos del blockchain Quicksha, códigos de sha 256 o sha 512. • Impulsar y mantener una arquitectura que permita asegurar datos de alta calidad, para que sean consumibles por los diferentes equipos. • Utilizar los datos para determinar qué tareas se pueden automatizar o emplear en modelos predictivos y prescriptivos. • Identificar cómo mejorar la calidad, eficiencia y confiabilidad de los datos. • Apoyar al equipo de trabajo del cliente y de XpertGroup en los temas asociados a la ingeniería de datos y que no se encuentren descritos en las funciones anteriores. • Generar estimaciones de proyectos y contribuir en la planificación y gestión de recursos necesarios para la ejecución exitosa del mismo. • Cumplir con las demás funciones que le sean asignadas afines a la descripción del cargo y de acuerdo con el nivel, naturaleza y área de desempeño.



