DeepSource GmbH logo
DeepSource GmbH

Build Artificial Intelligence Talented Teams

Data Engineer

Data EngineerData EngineerFull TimeRemoteSeniorTeam 1-10H1B No SponsorCompany SiteLinkedIn

Location

Egypt

Posted

3 days ago

Salary

0

Seniority

Senior

Job Description

Data Engineer

DeepSource GmbH

• Responsible for designing, building, and maintaining the data architecture and infrastructure required for our organization's data needs. • Develop, test, and deploy data solutions, ensuring they meet our organization's needs.

Job Requirements

  • Python data experience: You should have experience working with Python and be comfortable using it for data analysis and processing.
  • Pandas: You should be proficient in using the Pandas library for data manipulation and analysis.
  • Pyspark: You should have experience working with Pyspark for distributed data processing.
  • Databricks: You should be familiar with Databricks and have experience working with it to manage big data workloads.
  • Azure Synapse Spark: You should have experience working with Azure Synapse Spark to build and manage data pipelines.
  • Azure Data Factory: You should have experience working with Azure Data Factory to create and manage data integration workflows.
  • SQL query & Optimization: You should be proficient in SQL and have experience optimizing SQL queries for performance.
  • SQL Server migration: You should have experience migrating data from SQL Server to other platforms.
  • Bonus: Experience with Terraform for Azure.
  • L1: Someone with the knowledge assessed in the Databricks certification: Databricks Certified Data Engineering Associate
  • L2: Someone with the knowledge assessed in the Databricks certification: Databricks Certified Data Engineering Professional

Related Categories

Related Job Pages

More Data Engineer Jobs

Role Description Our client is looking for a Senior AWS/ Databricks Engineer in Sandton. - AWS VPC ownership (Databricks VPC): administer and maintain subnets, route tables, security groups, NACLs, NAT/Internet egress patterns (as applicable), and network segmentation to meet performance and security requirements. - Connectivity to the bank / enterprise network: troubleshoot and support end-to-end connectivity between the Databricks AWS VPC (IRE) and the bank network across cross-region/cross-account boundaries; coordinate and drive changes with the Cloud team where required. - Private access to AWS services: Design, implement, and operate VPC endpoints and related routing/DNS patterns to enable secure access to services such as S3 while reducing reliance on public internet paths. - S3 data access enablement (with security controls): Partner with platform/security teams to ensure Databricks workloads can reliably read/write required S3 data using appropriate IAM roles/policies and encryption controls; support diagnosis of access failures that present as platform incidents. - Operational support & reliability: Provide production support for the platform connectivity layer (incident response, RCA, preventative actions), maintain runbooks and reference diagrams, and implement improvements to reduce repeat incidents. - Cross-team change management: Raise, manage, and chase change requests with the Cloud team for items outside the Databricks VPC boundary; translate technical needs into clear implementation requirements and validate changes end-to-end. Qualifications - 6+ years of industry experience - AWS networking: strong hands-on experience with VPC design/operations, routing, security groups/NACLs, and network troubleshooting in production. - 5+ years in enterprise cloud operations: experience operating within a regulated/enterprise environment with change management, auditability, and strict security controls. - 3+ years in connectivity troubleshooting: ability to diagnose reachability issues across complex boundaries (cross-account/cross-region, enterprise network perimeters) and drive resolution across multiple teams. - 5+ years in AWS service access patterns: experience enabling secure access to services like S3 (and related IAM policy patterns) in a way that supports production workloads. - 3+ years in stakeholder management: proven ability to liaise with a central cloud/network team, raise and drive changes, and communicate clearly during incidents. - Databricks on AWS experience: understanding of Databricks workspace architecture and its connectivity constraints (data plane/control plane concepts, typical network dependencies). - Private connectivity patterns: experience with private endpoint patterns and enterprise connectivity services (e.g., endpoint-based access, centralized routing constructs). - Infrastructure-as-Code: Terraform/CloudFormation experience for repeatable, audited changes (nice-to-have). - Security tooling and monitoring: exposure to logging/monitoring approaches used for network and cloud operations. Ways of Working - Owns outcomes end-to-end (hands-on fixes inside the VPC; drives changes outside the boundary through the Cloud team). - Strong operational mindset prioritizes stability, clear communication, and measurable prevention of repeat incidents. - Documents and standardizes runbooks, network diagrams, and repeatable change patterns.

South Africa
ICF logo

Senior Education and Child Welfare Data Management and Visualization Specialist

ICF

Founded in 1969, ICF is a global advisory and technology services company headquartered in Reston, Virginia. It delivers data-driven solutions across energy, environment, infrastru

Data Engineer4 days ago

Role Description ICF seeks an on-call Senior Education and Child Welfare Data Management and Visualization Specialist to work within our Child and Family Well-Being (CFWB) Line of Business. This position specializes in Power BI and Tableau, connecting varied data sets to inform programmatic insights, data storytelling, and using data to help our clients understand progress towards achieving their goals. The Senior Education and Child Welfare Data Management and Visualization Specialist will apply academic knowledge and work experience to support education and child welfare evaluation and continuous quality improvement (CQI) on research and technical assistance projects in a state. Key Responsibilities - Lead the collection, management, and analysis of data to inform program insights and improve data accessibility for users. - Develop dashboards, reports, infographics, and data-driven stories. - Collect data through surveys, focus groups, and interviews. - Carry out essential administrative data-management tasks. - Help build the skills of data analysts by providing guidance on Power BI, data visualization best practices, and approaches for generating meaningful insights aligned with project needs. - Support, design, and build interactive and static dashboards, reports, and data visualizations using Power BI, Tableau, and/or Excel. - Support quality assurance tasks, building data infrastructure, data management and analysis, and data cleaning and quality control processes. - Design data-driven storytelling using data visualization techniques and best practices. - Maintain and rapidly produce regular reporting deliverables and user-friendly reports/presentations and infographics for a variety of audiences. - Partner with teams to define key performance indicators, business intelligence, and data reporting requirements. - Support and mentor data staff across multiple projects in building skills in Power BI, Tableau, and other data management and visualization tools. - Conduct basic descriptive analyses of data to generate reports and data visuals of program activities, outputs/deliverables, and outcomes. - Conduct occasional observations or visits to locations nationwide. Qualifications - Bachelor's degree in data science, education, sociology, social work, psychology, justice, or other social science related subject. - 4+ years of experience conducting qualitative and/or quantitative data collection or management, research analysis, or evaluation. - Demonstrated experience in using software such as Power BI, Tableau, and Power Query is required. - Demonstrated experience in data visualization and data storytelling techniques and principles is required. - Demonstrated experience in producing high-quality, professional visualizations, presentations, products, and/or resources is required. Preferred Skills/Experience - Master’s degree in data science, education, sociology, social work, psychology, justice, or other social science related subject. - Program certificate, coursework, or training in data visualization or data storytelling. - Strong organizational skills with proven ability to manage multiple tasks and maintain attention to detail. - Experience working independently to develop innovative solutions to client data interests and challenges. - Foundational knowledge of SQL for querying and validating data. - Experience sharing knowledge and building data management or visualization skills of colleagues. - Prior professional work experience in a consulting environment. Professional Skills - Demonstrated ability to take ownership of project tasks and follow through on commitments. - Strong interest and/or experience in areas such as early education; K–12 and higher‑education research and evaluation; refugee, immigrant, and asylee services; child welfare; capacity building; technical assistance; mental and behavioral health; implementation science; and continuous quality improvement (CQI). - Proven ability to prioritize and manage multiple activities in a fast‑paced, evolving environment. - Effective team builder and team player with the ability to navigate competing priorities, perspectives, and stakeholder needs. - Strong analytical, problem‑solving, and decision‑making skills. - Excellent verbal, written, and interpersonal communication skills, including experience developing and delivering presentations and technical writing. Pay Range The pay range for this position based on full-time employment is: $74,090.00 - $125,954.00.

United States
$74.1K - $126.0K / year
ICF logo

Senior Education and Child Welfare Data Management and Visualization Specialist

ICF

Founded in 1969, ICF is a global advisory and technology services company headquartered in Reston, Virginia. It delivers data-driven solutions across energy, environment, infrastru

Data Engineer4 days ago

Role Description ICF seeks an on-call Senior Education and Child Welfare Data Management and Visualization Specialist to work within our Child and Family Well-Being (CFWB) Line of Business. This position specializes in Power BI and Tableau, connecting varied data sets to inform programmatic insights, data storytelling, and using data to help our clients understand progress towards achieving their goals. The Senior Education and Child Welfare Data Management and Visualization Specialist will apply academic knowledge and work experience to support education and child welfare evaluation and continuous quality improvement (CQI) on research and technical assistance projects in a state. Key Responsibilities - Lead the collection, management, and analysis of data to inform program insights and improve data accessibility for users. - Develop dashboards, reports, infographics, and data-driven stories. - Collect data through surveys, focus groups, and interviews. - Carry out essential administrative data-management tasks. - Help build the skills of data analysts by providing guidance on Power BI, data visualization best practices, and approaches for generating meaningful insights aligned with project needs. - Support, design, and build interactive and static dashboards, reports, and data visualizations using Power BI, Tableau, and/or Excel. - Support quality assurance tasks, building data infrastructure, data management and analysis, and data cleaning and quality control processes. - Design data-driven storytelling using data visualization techniques and best practices. - Maintain and rapidly produce regular reporting deliverables and user-friendly reports/presentations and infographics for a variety of audiences. - Partner with teams to define key performance indicators, business intelligence, and data reporting requirements. - Support and mentor data staff across multiple projects in building skills in Power BI, Tableau, and other data management and visualization tools. - Conduct basic descriptive analyses of data to generate reports and data visuals of program activities, outputs/deliverables, and outcomes. - Conduct occasional observations or visits to locations nationwide. Qualifications - Bachelor's degree in data science, education, sociology, social work, psychology, justice, or other social science related subject. - 4+ years of experience conducting qualitative and/or quantitative data collection or management, research analysis, or evaluation. - Demonstrated experience in using software such as Power BI, Tableau, and Power Query is required. - Demonstrated experience in data visualization and data storytelling techniques and principles is required. - Demonstrated experience in producing high-quality, professional visualizations, presentations, products, and/or resources is required. Requirements - Master’s degree in data science, education, sociology, social work, psychology, justice, or other social science related subject (preferred). - Program certificate, coursework, or training in data visualization or data storytelling (preferred). - Strong organizational skills with proven ability to manage multiple tasks and maintain attention to detail. - Experience working independently to develop innovative solutions to client data interests and challenges. - Foundational knowledge of SQL for querying and validating data. - Experience sharing knowledge and building data management or visualization skills of colleagues. - Prior professional work experience in a consulting environment. Professional Skills - Demonstrated ability to take ownership of project tasks and follow through on commitments. - Strong interest and/or experience in areas such as early education; K–12 and higher‑education research and evaluation; refugee, immigrant, and asylee services; child welfare; capacity building; technical assistance; mental and behavioral health; implementation science; and continuous quality improvement (CQI). - Proven ability to prioritize and manage multiple activities in a fast‑paced, evolving environment. - Effective team builder and team player with the ability to navigate competing priorities, perspectives, and stakeholder needs. - Strong analytical, problem‑solving, and decision‑making skills. - Excellent verbal, written, and interpersonal communication skills, including experience developing and delivering presentations and technical writing. Pay Range The pay range for this position based on full-time employment is: $74,090.00 - $125,954.00.

United States
$74.1K - $126.0K / year

Role Description We are looking for someone to work in a team that thinks big, works with attention to detail, collaborates daily, and shares responsibility. We believe in autonomy, the exchange of different experiences, and solutions built together. Here, growth means sharing knowledge, respecting different ways of thinking, and building trusting relationships while delivering quality technology. If you identify with this way of working, come join us. What you will do: - Design, build, deploy, and support cloud-based solutions for ingesting and processing multiple data sources (e.g., weather, location, satellite, transactions); - Develop scalable data pipelines for structured and unstructured data; - Build APIs for accessing and processing geospatial data; - Implement event-driven data processing using Google Cloud Pub/Sub; - Integrate data across cloud platforms (AWS and GCP) and systems such as PostgreSQL/PostGIS and BigQuery; - Optimize data pipelines for performance, reliability, scalability, and cost; - Provide support and troubleshooting for production pipelines and services. Qualifications - Fluent English; - Strong experience in data engineering; - Proficiency in Python and SQL; - Experience with PostgreSQL/PostGIS; - Experience with large, complex datasets geospatial; - Solid understanding of data modeling best practices; - Hands-on experience with GCP services BigQuery, Pub/Sub, Cloud Run, Cloud Storage; - Experience with version control tools (GitHub or Azure DevOps); - Strong analytical and problem-solving mindset; - Ability to communicate with both technical and non-technical stakeholders. Requirements - Nice to have: - Multi-cloud experience (AWS + GCP); - Experience with event-driven architectures; - Experience with query optimization and large-scale data processing; Benefits - 🍽 Meal/Food Allowance - Caju multi-benefit card (credit format), offering flexibility for everyday needs. (CLT only) * - 🏥 Health Plan – Unimed - No waiting period or copayment for the holder. Possibility to include dependents (children, spouse, or stepchildren) with copayment. (CLT only) * - 🦷 Dental Plan – Uniodonto - Affordable options to include dependents. (CLT only) * - 💪 Wellhub - Access to gyms, physical activities, and wellness programs. - 🧘 Zenklub - For CLT: Two free sessions per month and special rates for additional sessions. For Cooperative/Contractor: Special rates for sessions. - 🛡 Life Insurance - More security for you and your family. (CLT only) * - 👶 Childcare Allowance - Financial support for Zallpers with children from 4 months to 6 years old, according to internal policy. (CLT only) * - 🎁 Baby Zallpy - A special gift to celebrate the arrival of new Zallpy babies. - 👥 Business Partner Support - Close and human-centered support from our People & Culture team throughout your journey at Zallpy. - 🌍 Volunteer Internal Communities - Diversity, Sports & Movement, and Technology. - 🎓 Educational Partnerships - Discounts on undergraduate and graduate programs, professional courses, and language schools. - ✈️ Experiences & Development - Participation in events, workshops, trips, and team-building activities. - 💼 Referral Program - Refer talents to Zallpy and receive bonuses from R$2,500 to R$5,000.

Brazil
$2.5K - $5K / year