• Design, develop, and optimize enterprise-scale data pipelines and ETL/ELT workflows using Azure and Databricks technologies. • Architect and implement scalable data ingestion, transformation, and orchestration processes using Azure Data Factory, Databricks, and Azure Synapse Analytics. • Develop high-performance data transformation frameworks using PySpark, Python, and Spark SQL for large-scale distributed data processing. • Optimize SQL queries, Spark jobs, and data workflows to improve performance, scalability, and cost efficiency. • Lead data migration initiatives, including SQL Server migrations and modernization of legacy data platforms. • Implement and maintain Delta Lake architecture, incremental data loading strategies, and enterprise data lake best practices. • Collaborate with architects and cross-functional teams to design robust and scalable data models aligned with business and governance standards. • Monitor and troubleshoot production pipelines, perform root-cause analysis, and implement preventive measures for recurring issues. • Support CI/CD implementation and infrastructure automation for data engineering workflows. • Mentor junior engineers and contribute to engineering standards, reusable frameworks, and technical best practices. • Create and maintain technical documentation including architecture diagrams, pipeline documentation, and operational runbooks. • Evaluate and recommend modern data engineering tools, frameworks, and optimization strategies.

Role Description As a Data Engineer, He/She will be responsible for designing, building, and maintaining the data architecture and infrastructure required for our organization's data needs. They will also be responsible for developing, testing, and deploying data solutions, ensuring that they meet our organization's needs. Qualifications - Experience working with Python for data analysis and processing. - Proficient in using the Pandas library for data manipulation and analysis. - Experience working with Pyspark for distributed data processing. - Familiar with Databricks for managing big data workloads. - Experience with Azure Synapse Spark for building and managing data pipelines. - Experience with Azure Data Factory for creating and managing data integration workflows. - Proficient in SQL and experienced in optimizing SQL queries for performance. - Experience migrating data from SQL Server to other platforms. - Bonus: Experience with Terraform for Azure. - Knowledge assessed in the Databricks certification: Databricks Certified Data Engineering Associate. - Knowledge assessed in the Databricks certification: Databricks Certified Data Engineering Professional. Company Description

• Responsible for designing, building, and maintaining the data architecture and infrastructure required for our organization's data needs. • Develop, test, and deploy data solutions, ensuring they meet our organization's needs.

Role Description Our client is looking for a Senior AWS/ Databricks Engineer in Sandton. - AWS VPC ownership (Databricks VPC): administer and maintain subnets, route tables, security groups, NACLs, NAT/Internet egress patterns (as applicable), and network segmentation to meet performance and security requirements. - Connectivity to the bank / enterprise network: troubleshoot and support end-to-end connectivity between the Databricks AWS VPC (IRE) and the bank network across cross-region/cross-account boundaries; coordinate and drive changes with the Cloud team where required. - Private access to AWS services: Design, implement, and operate VPC endpoints and related routing/DNS patterns to enable secure access to services such as S3 while reducing reliance on public internet paths. - S3 data access enablement (with security controls): Partner with platform/security teams to ensure Databricks workloads can reliably read/write required S3 data using appropriate IAM roles/policies and encryption controls; support diagnosis of access failures that present as platform incidents. - Operational support & reliability: Provide production support for the platform connectivity layer (incident response, RCA, preventative actions), maintain runbooks and reference diagrams, and implement improvements to reduce repeat incidents. - Cross-team change management: Raise, manage, and chase change requests with the Cloud team for items outside the Databricks VPC boundary; translate technical needs into clear implementation requirements and validate changes end-to-end. Qualifications - 6+ years of industry experience - AWS networking: strong hands-on experience with VPC design/operations, routing, security groups/NACLs, and network troubleshooting in production. - 5+ years in enterprise cloud operations: experience operating within a regulated/enterprise environment with change management, auditability, and strict security controls. - 3+ years in connectivity troubleshooting: ability to diagnose reachability issues across complex boundaries (cross-account/cross-region, enterprise network perimeters) and drive resolution across multiple teams. - 5+ years in AWS service access patterns: experience enabling secure access to services like S3 (and related IAM policy patterns) in a way that supports production workloads. - 3+ years in stakeholder management: proven ability to liaise with a central cloud/network team, raise and drive changes, and communicate clearly during incidents. - Databricks on AWS experience: understanding of Databricks workspace architecture and its connectivity constraints (data plane/control plane concepts, typical network dependencies). - Private connectivity patterns: experience with private endpoint patterns and enterprise connectivity services (e.g., endpoint-based access, centralized routing constructs). - Infrastructure-as-Code: Terraform/CloudFormation experience for repeatable, audited changes (nice-to-have). - Security tooling and monitoring: exposure to logging/monitoring approaches used for network and cloud operations. Ways of Working - Owns outcomes end-to-end (hands-on fixes inside the VPC; drives changes outside the boundary through the Cloud team). - Strong operational mindset prioritizes stability, clear communication, and measurable prevention of repeat incidents. - Documents and standardizes runbooks, network diagrams, and repeatable change patterns.