Job Closed
This listing is no longer active.
The company behind edX. We connect 100M+ learners to world-class, free-to-degree online education.
Senior Data Engineer
Location
South Africa
Posted
172 days ago
Salary
0
Seniority
Senior
Job Description
Senior Data Engineer
2U
• Design and build robust, scalable, and high-performance ETL/ELT data pipelines • Develop and implement solutions using advanced programming languages (e.g., Python, SQL, DBT) • Optimize and performance-tune complex data workflows and queries to handle massive data volumes and ensure minimal latency • Assist with the design and implementation of data models for Data Warehouses (Snowflake etc.) • Implement data quality frameworks (validation, monitoring, and testing) to ensure data accuracy, consistency, and completeness • Establish comprehensive error logging and alerting systems for pipeline health and failure remediation • Apply DevOps practices to data infrastructure, often involving Infrastructure as Code (e.g., Terraform) • Effectively leverage knowledge, expertise and experience to lead and deliver projects • Effectively support and mentor data engineers • Stakeholder management throughout all business departments • Collaborate with other engineers and analysts to execute projects • Perform research and constantly stays on top of the industry’s trends • Engage and proactively seek opportunities for improvement through ongoing learning
Job Requirements
- 5+ years experience in the Data industry
- 4+ years experience in a Data Engineering role
- Expert in SQL to then implement where needed in Data Warehousing systems and Data Pipelines
- PostgreSQL / MySQL / Python / R / Redshift / Snowflake experience
- Experience in database design and database management
- A sound understanding of data management principles, including data quality assurance and data governance
- Experience in source code and version control systems
- Experience with infrastructure automation (Terraform)
- Familiarity working with containerized services (Docker and Kubernetes)
- Experience in the shipment of data using various methods, such as ETL/ELT, within tools such as Airflow (MWAA), Argo, dbt
- Experience using services within the AWS ecosystem (e.g, RDS, EC2, Redshift, EKS)
- Understanding or knowledge of Linux commands (bash scripting)
- Manages multiple projects, resolves challenges swiftly, delivers adaptable approaches, and values diverse opinions.
- Takes ownership, anticipates new problems, tests solutions, and mentors on effective problem-solving techniques.
- Sets quarterly goals aligned with team objectives, tracks progress in Workday, and communicates barriers to the manager. Takes personal responsibility for quality and timeliness of work, correcting mistakes and learning from them.
- Demonstrates expert understanding, introduces new tools/methodologies, and provides team training to enhance productivity.
- Tailors messages for audience understanding, engages respectfully in team meetings, synthesizes thoughts effectively, presents confidently, and actively listens to stakeholders.
- Consistently builds and maintains key stakeholder relationships, perceived as a trusted partner, offers sound guidance, and navigates challenging relationships effectively.
Benefits
- 2 complimentary Getsmarter short courses per year
- Subsidised medical aid with Discovery Health Medical Scheme
- 4% 2U contribution towards Discovery Life Pension Fund and Group Risk Benefit
- Employee Assistance Program (EAP)
- Generous leave policy including time off to volunteer for non-profit organization, study leave, sports leave and a company-wide festive season break
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Engineer
MediaRadar, Inc.Sales enablement platforms customized for media, and ad tech companies that help you close more deals.
• Involve in Design, development, and maintenance of scalable ETL/ELT pipelines on Azure Databricks using Apache Spark (PySpark/Spark SQL). • Design and implement both batch and real-time data ingestion and transformation processes. • Build and manage Delta Lake tables, schemas, and data models to support efficient querying and analytics. • Consolidate and process large-scale datasets from various structured and semi-structured sources (e.g., JSON, Parquet, Avro). • Write optimized SQL queries for large datasets using Spark SQL and PostgreSQL. • Develop, schedule, and monitor workflows using Databricks Workflows, Airflow or similar orchestration tools. • Design, build, and deploy cloud-native, containerized applications on Azure Kubernetes Service (AKS) and integrate with Azure services. • Ensure data quality, governance, and compliance through validation, documentation, and secure practices. • Collaborate with data analysts, data architects, and business stakeholders to translate requirements into technical solutions. • Contribute to and enforce best practices in data engineering, including version control (Git), CI/CD pipelines, and coding standards. • Continuously enhance data systems for improved performance, reliability, and scalability. • Mentor junior engineers and help evolve team practices and documentation. • Stay up to date on emerging trends, technologies, and best practices in the data engineering space. • Work effectively within an agile, cross-functional project team.
• Translates business requirements into specifications that will be used to implement the required data load (ETL), reports and dashboards. • Gathers business requirements, development and delivery of business intelligence & reporting services and lead end to end BI delivery. • Develops and implements business intelligence, analytics, and dashboards using out-of-box features customizations and visualizations to support organizational initiatives. • Designs, models, develops & optimizes stored procedures to meet data management and data load and reporting objectives. • Designs and models data flows and ETL procedures ensuring data quality and integrity. • Ensures data sets feeding the analytics platforms are tuned for optimal performance. • Works with power users of the analytics/reporting tools to advise on best practices for Business Intelligence reporting. • Troubleshoots and resolves issues with the processes used and the content produced by the BI platform. • Works with business stakeholders to define and improve standardized and ad-hoc analytics/reporting capabilities. • Works as part of the BI team to maintain and develop all components of the BI platform. • Provides ongoing maintenance support through troubleshooting, report modifications and optimization. • Provides input into the technical architecture and the data warehouse technology stack. • Creates functional and technical design documents and maintains documentation for all reports and dashboards created or modified. • Responsibilities include orienting and training employees; assigning workload; planning, monitoring, and appraising job results; and coaching, counseling, and disciplining employees.
• Lead planning and execution of data migration activities from four SAP ECC environments into SAP S/4HANA • Utilize tools such as SNP CrystalBridge (or similar) to perform data extraction, transformation, loading, and reconciliation • Define and implement required data transformation rules to ensure data integrity and alignment with the new ERP design • Collaborate with Functional Process Teams to support readiness activities, validate business requirements, and ensure successful migration outcomes • Participate in multiple integrated test cycles, including mock conversions and validation testing, to confirm migration accuracy • Identify defects, gaps, and inconsistencies between SAP and legacy systems; coordinate resolution with project teams • Support functional data design discussions and technical load processes as required • Provide expert guidance on migration strategy, tools, and best practices • Escalate risks, issues, and decision points to leadership in a timely manner • Provide deployment and post-go-live support to ensure a stable transition to S/4HANA
Data Engineer
Five Forty540 is a forward-thinking company that the government turns to in order to #getshitdone. We don’t just talk about innovation – we deliver it. We break down barriers, build impactful technology, and solve mission-critical problems.
This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more. Role Description 540 is seeking a Data Engineer to join the Advana team supporting the MHS Caduceus program. In this role, you’ll help build and maintain the Financial Management Data Analytics stream supporting the Defense Health Program portfolio. You’ll work hands-on with complex DoD and DHA data to develop scalable data products that provide actionable financial insight. This role focuses on delivering production-ready data capabilities that support mission-critical healthcare and financial decision-making. Location: Remote, with preference for candidates located near Arlington, VA for occasional in-person client meetings Citizenship & Clearance Requirement: per client requirements, candidates must be U.S. Citizens with an active DoD Secret clearance Education Requirement: Bachelor’s Degree 540 Internal Thrive Level: Data Engineer II or III Qualifications - 3+ years of data engineering or related experience - Strong Python experience - Experience engaging with clients and/or stakeholders - Experience building and maintaining data pipelines and ETL processes - Experience working with Databricks or similar modern data platforms - Experience consuming data via APIs - Familiarity with Git-based development workflows - Comfortable working in terminal / command-line environments - Experience with data visualization tools (Tableau or similar) Requirements - AWS cloud experience - Experience working with very large datasets (billions of records) - Experience with Palantir Foundry - Experience building, deploying and managing Tableau dashboards - Experience working with Advana data environments - GitLab experience - Experience working with DoD systems - Experience working with financial systems Benefits - Flexible PTO + all Federal holidays off - Health, dental and vision insurance plans - Flexible Spending Account (FSA) - 401k with employer match - Company-sponsored life insurance, short- and long-term disability - Professional development (training, certifications, conferences) - Paid cloud developer accounts - Referral bonuses - HQ office perks (parking / metro reimbursement, nitro coffee & lunches) - Annual social events (540 Week, hackathon, charity golf tournament, etc.) - Access to 540’s Washington Capitals & Nationals tickets



