Data Engineer

Location

South Africa

Posted

7 days ago

Salary

0

Seniority

Mid Level

Job Description

Data Engineer

Sambe Consulting

Role Description You will be part of a multi-disciplinary technology team, working closely with our customers (business, software vendors, and partners). Within the team, you will be responsible for a suite of data processes and will participate in all aspects from design through to testing and implementation. You will be surrounded by data professionals that strive for excellence and data best practices to realize business value. As a Data Engineer, you will work in the data engineering team to build and maintain data pipelines to ingest data into the warehouse and support integration into other systems. You will apply your knowledge of good data engineering practices and standards, and data technologies to deliver target state design and implementation. The role is challenging, and you must be adept at problem-solving and able to respond to changing priorities and rapidly evolving requirements that may have a direct impact on services to users. This role would suit a professional who is keen to grow their career in a busy team that values cognitive diversity and diversity of lived experience. Duties & Responsibilities - Maintain, support, and monitor existing production SSIS packages and SQL Queries and Stored Procedures, CICD pipelines to ensure all data loads on the data warehouse meet data quality standards and business SLA requirements. - Build, maintain, support, and monitor Synapse data engineering pipelines on the data platform if required. - Participate and contribute to data architecture design, data modelling, gathering and analysis of data requirements; understand, document, communicate, and build appropriate solutions. - Participate, design, build, deliver, and document data-related projects with various environment-specific data analytics technologies. - Promote data engineering best practices with CICD pipelines and automation. - Collaborate and work closely with team members and contribute significantly to building a high-performing, collaborative, transparent, and result-driven data engineering team. - Support the Data Engineering Practice Team Manager with best fit-to-purpose data engineering solutions, quality engineering artifacts, and high standard documentation. - Follow Data Ethics standards to protect personal information and meet our customers’, partners’, and community’s expectations. Qualifications - 8+ years demonstrable experience in design, build, and support of data engineering pipelines in data warehousing, data ingestion, cleansing, manipulation, modelling, and reporting. - Experience in ETL using Microsoft technologies. - Strong experience in writing MS SQL server queries, stored procedures, and SSIS Packages. Experience with SSRS would be an advantage. - Experience of manipulating semi-structured data (XML, JSON). - Strong knowledge and extensive experience in working in an Agile framework with CI/CD using modern DevOps / Data Ops integrated processes with YAML pipelines. - Bachelor’s degree in computer/data science technical or related field is a must. Post-graduate is highly regarded. - Knowledge of Azure Synapse data engineering pipelines, PySpark notebooks, data platform lake house architecture, and Azure SQL ODS storage is desirable.

Related Categories

Related Job Pages

More Data Engineer Jobs

GitLab logo

Intermediate Fullstack Engineer – Data Products

GitLab

Build software faster. The One DevOps Platform enables your entire org to collaborate around your code. We're hiring.

Data Engineer7 days ago
Full TimeRemoteTeam 1,001-5,000Since 2014H1B No Sponsor

• Develop well-scoped features, with support from Senior and Staff Engineers, for how GitLab and third-party data is ingested, modeled, and synced into the knowledge graph, meeting team goals for data freshness, sync reliability, and graph coverage. • Build parts of integrations with external systems for business context, including Jira, observability tools, Zendesk, and ServiceNow, increasing indexed business context while meeting team goals for data completeness and freshness in the graph. • Build features for the Software Engineering Intelligence dashboards, including DORA, value stream reporting, GitLab Duo, and software development lifecycle metrics rolled up across organizational structures, meeting team goals for reporting accuracy, performance, and coverage. • Contribute to design discussions and technical direction for your features, with guidance from Senior and Staff Engineers, helping deliver secure, reliable, and performant solutions on schedule. • Raise blockers and risks early, and work with Senior Engineers to unblock your work so milestones stay on track and delivery risk is reduced. • Participate in design review and code review, and apply feedback to improve code quality, maintainability, and delivery velocity. • Build a deep understanding of the system, including the data model and the platform the team builds on, so you can ship changes more independently and resolve issues faster.

India
Software Mind logo

Corporate Data Architect

Software Mind

Software House focused on results since 1999

Data Engineer7 days ago
Full TimeRemoteTeam 1,001-5,000Since 1999H1B No Sponsor

Role Description We are looking for a Corporate Data Architect who will help organize and standardize how the organization understands and uses data. Your main task will be to design and develop data architecture at the level of the entire organization so that data is consistent, well defined, and ready to be used in analytics, AI, and system integrations. You will act as a bridge between the worlds of business, data, and technology, connecting strategy, standards, and practice. - Design semantic, logical, and conceptual data models based on business domains and organizational processes. - Define and standardize the corporate data model as part of the Data Governance initiative. - Create and maintain reference architecture (logical and technical reference architectures) and design patterns for data domains. - Co-create the strategy and roadmap for data architecture development that supports the vision of the Data Governance program and organizational goals. - Collaborate with Data Governance, Data Management, Analytics, AI, and Data Engineering teams to integrate models with the data catalog and metadata management tools. - Support technical teams in implementing models in data layers. - Participate in designing data integration, flow, and quality processes (data lineage, data contracts, data quality). - Identify and certify authoritative data sources within organizational domains. - Monitor the compliance of data solutions with Data Governance policies and standards as well as architectural recommendations. - Support the resolution of data conflicts and issues. - Create architectural documentation, metamodels, and inter-domain relationship diagrams. - Consult on and develop good practices in the area of modeling, metadata, and information architecture. - Collaborate on defining the vision, priorities, and directions for the development of data architecture in the organization. Qualifications - Experience in data modeling (conceptual, logical, physical), preferably in a large organization or within complex data ecosystems. - Practical knowledge of tools and methodologies such as ER, UML, IDEF1X, Data Vault, 3NF, Kimball/Inmon. - Ability to work with data catalogs and metadata management tools (e.g. DataHub, Collibra, Alation, Atlan). - Knowledge of SQL and relational data models. - Experience working with database systems and data architecture in a cloud environment (AWS). - Ability to work with engineering and analytics teams to translate business needs into data models. - Understanding of data security, availability, and control principles. - Good command of English (reading documentation, community discussions). Requirements - Experience in Data Governance, Data Quality, Master Data, and Metadata Management projects. - Knowledge of concepts and technologies related to the semantic data layer: ontologies, RDF, OWL, GraphQL, dbt Semantic Layer, Semantic Kernel. - Knowledge of Python for automation, model validation, and API integrations. - Knowledge of event-driven data architecture. - Experience in creating and maintaining architectural documentation. Benefits - Flexible employment and remote work. - International projects with leading global clients. - International business trips. - Non-corporate atmosphere. - Language classes. - Internal & external training. - Private healthcare and insurance. - Multisport card. - Well-being initiatives.

Worldwide
Checkr logo

Staff Data Engineer

Checkr

Checkr is self-described as a leading background service for small businesses, empowering clients to conduct background screenings on potential candidates in a

Data Engineer7 days ago

Title: Staff Data Engineer Location: Denver, Colorado, United States; San Francisco, California, United States Job Description: About Checkr Checkr is building the data platform to power safe and fair decisions. Over 140,000 companies and millions of people rely on Checkr for AI verification in the moments that matter most: getting a new job, a new place to live, a car ride, childcare, even a date. Customers include Uber, Pennymac, Airbnb, Doordash, Amazon, and Anthropic. We're a team that thrives on solving complex problems with innovative solutions that advance our mission. Checkr is recognized on Forbes Cloud 100 2025 List and is a Y Combinator 2024 Breakthrough Company. As a Staff Data Engineer on the People Data team, you''ll help build and evolve the centralized platform that powers every Checkr product. This platform stores and serves the identity and people records that underpin Checkr''s Workforce, Mortgage, Tenant, Trust, and Personal products, making it foundational to the company''s AI-powered verification platform and every high-stakes decision our customers make. In this role, you''ll own the core services, data pipelines, and architecture that keep this platform scalable, reliable, and ready for the next generation of Checkr products. You''ll solve complex distributed systems and data engineering challenges while influencing the technical direction of one of the company''s most critical platforms. What you''ll do - Architect, design, lead, and build an end-to-end, performant, reliable, scalable data platform. - Work as an independent contributor: solve problems and deliver high-quality solutions with minimal oversight and strong ownership. - Mentor and guide junior engineers to deliver complex, next-generation features. - Bring a customer-centric, product-oriented mindset. Collaborate with customers and internal stakeholders to resolve product ambiguities and ship features that solve real customer problems. - Partner with engineering, product, design, and other stakeholders to design and architect new features. - Experimentation mindset: autonomy and empowerment to validate a customer need, get team buy-in, and ship a rapid MVP. - Quality mindset: you treat quality as a non-negotiable part of your software deliverables. - Analytical mindset: instrument and deploy new product experiments with a data-driven approach. - Monitor, triage, and resolve production issues for the team''s services. - Create and maintain data pipelines and foundational datasets to support product and business needs. What you bring Required Experience - 10+ years designing, implementing, and delivering highly scalable, performant data platforms. - Experience building large-scale data processing pipelines using ETL/ELT, batch, and stream processing. - Expert-level proficiency in PySpark, Python, and SQL. - Expertise in data modeling, relational databases, and NoSQL data stores (e.g., MongoDB). - Experience with big data technologies such as Kafka, Spark, Iceberg, data lakes, and the AWS stack (EKS, EMR, Serverless, Glue, Athena, S3, etc.). - Knowledge of security best practices and data privacy concerns. - Strong problem-solving skills and attention to detail. Nice to have - Experience or knowledge of data processing platforms such as Databricks or Snowflake. #LI-TD1 Pay Transparency Disclosure We use geographic cost of labor as an input to develop ranges for our roles and as such, each location where we hire may have a different range. If this role is remote, we have listed the top to the bottom of the possible range, but we will specify the target range for an exact location when you are selected for a recruiting discussion. For more information on our compensation philosophy, see our website. On-target Earnings OR Base Salary range (San Francisco, CA) $196,000-$230,000 USD On-target Earnings OR Base Salary range (Denver, CO) $166,000-$195,000 USD What We Offer - A fast-paced and collaborative environment - Learning and development allowance - Competitive cash and equity compensation, and opportunity for advancement - 100% medical, dental, and vision coverage - Up to $25K reimbursement for fertility, adoption, and parental planning services - Flexible PTO policy - Monthly wellness stipend At Checkr, we believe an in office work environment strengthens collaboration, drives innovation, and encourages connection. Our hub locations are Denver, CO; San Francisco, CA; Nashville, TN; and Santiago, Chile. Individuals are expected to work from the office 3+ days a week. In-office perks are provided, such as lunch five times a week, a commuter stipend, and an abundance of snacks and beverages. A Equal Employment Opportunities at Checkr Checkr is committed to building the best product and company, which requires hiring talented and qualified individuals with a diverse set of perspectives and lived experiences. Checkr believes in hiring people of all backgrounds, including those whose histories are impacted by the justice system in accordance with local, state, and/or federal laws, including the San Francisco's Fair Chance Ordinance.

Colorado + 1 moreAll locations: Colorado | California
$196K - $230K / year

Data Engineer

Trilon Group

Trilon Group provides smart and sustainable infrastructure solutions across transportation, water, energy, environment, and community sectors. The firm offers a

Data Engineer7 days ago

Data Engineer Department: IT Job Description: Employment Type: Full Time Location: Remote- USA Compensation: $116,000 - $155,000 / year Description Trilon is building a supercharged, technology-enabled future for our people and partners. The Data Engineer plays a key role in that mission by building and maintaining the data platform that powers Trilon's enterprise analytics, automation, and AI capabilities. Reporting to the Vice President, Data & DevOps, this role is responsible for designing, developing, and maintaining scalable data integrations and transformations in Azure and Microsoft Fabric. The Data Engineer ensures that Trilon's data platform delivers reliable, high-quality, and well-structured data to support business intelligence, operations, and innovation. This role serves as the primary custodian of Trilon's integrated data model and is instrumental in developing a unified, extensible architecture that scales with continued acquisitions. The Data Engineer designs and builds secure Power BI semantic models for consumption by analysts and decision-makers, ensuring consistent and governed access to enterprise data. This role also partners closely with the AI and Innovation vTeam to prepare data for analytics, machine learning, and retrieval-augmented generation (RAG) applications. Key Responsibilities Data Platform Engineering and Maintenance - Serve as the primary owner and technical steward of the Trilon enterprise data platform - Design, develop, and maintain data pipelines and workflows using Azure Data Factory, Synapse, and Microsoft Fabric - Build and manage data transformations, orchestration, and automation across structured, semi-structured, and unstructured data sources - Ensure scalability, reliability, and performance of the data platform as Trilon continues to grow through acquisition - Implement monitoring and alerting to proactively detect and resolve pipeline or data quality issues Data Integration and Modeling - Develop and maintain integrations between Trilon's enterprise systems, cloud services, and acquired partner environments - Design and maintain a unified, scalable data model that harmonizes data across business systems - Build secure, governed, and high-performance Power BI semantic models optimized for analytics and self-service reporting - Collaborate with business analysts and data consumers to ensure data models support enterprise reporting needs and KPIs - Partner with cybersecurity and infrastructure teams to ensure data models and access patterns meet compliance and governance standards Data Quality and Governance - Implement validation and quality checks to ensure accuracy, completeness, and timeliness of enterprise data sets - Maintain metadata, lineage, and documentation to promote transparency and reusability - Define and enforce data quality and consistency standards across all integrated sources - Collaborate with the Technology Asset Manager and Service Platform Manager to align system integrations and data governance - Support data cataloging, discovery, and classification initiatives within Microsoft Purview or equivalent tools Automation, Optimization, and Resilience - Develop automated frameworks for ingestion, transformation, and validation using Azure-native tools and pipelines - Implement DevOps principles for data workflows including version control, testing, and deployment automation - Optimize pipeline performance, resource utilization, and data freshness - Build resilience and fault tolerance into data operations to ensure reliability and recovery - Create reusable components and templates to streamline integration of new data sources and partner systems AI and Innovation Enablement - Collaborate with the AI and Innovation vTeam to prepare and structure data for AI, ML, and RAG-based applications - Develop and maintain data pipelines that support model training, evaluation, and fine-tuning - Curate and transform unstructured data for retrieval, embedding, and vectorization within AI applications - Ensure data readiness for generative AI tools, chat interfaces, and knowledge retrieval systems - Stay informed of emerging AI data engineering trends and Microsoft Fabric AI integrations Collaboration and Cross-Domain Partnership - Partner with application and infrastructure teams to ensure reliable and secure data exchange across systems - Collaborate with business stakeholders and analysts to understand reporting needs and deliver usable data models - Support integration engineers in onboarding new firms and ensuring their data aligns with Trilon's enterprise model - Work closely with cybersecurity and compliance teams to enforce data protection, retention, and access policies - Provide documentation, architecture diagrams, and operational standards for the data platform and pipelines Skills, Knowledge and Expertise - 5 or more years of experience in data engineering, data integration, or data platform development - Strong hands-on experience with Azure Data Factory, Azure Synapse, Microsoft Fabric, and related Azure data services - Proficiency in SQL, DAX, Power Query, and data modeling for Power BI - Experience designing and maintaining Power BI semantic models, datasets, and row-level security configurations - Familiarity with data governance, cataloging, and lineage management in tools like Microsoft Purview - Experience building and optimizing cloud data pipelines with structured, semi-structured, and unstructured data - Understanding of data preparation for AI and machine learning applications, including RAG architectures - Exposure to engineering and geospatial data such as CAD, BIM, and GIS - Strong analytical and problem-solving skills with a focus on scalability and performance - Excellent collaboration and communication skills across technical and business audiences - Bachelor's degree in Computer Science, Data Engineering, or related field preferred - Microsoft certifications such as Azure Data Engineer Associate or Fabric Analytics Engineer Associate are a plus - May require occasional travel to Trilon offices or partner locations for integration or collaboration activities About Trilon Trilon was formed with the vision of building the next Top 20 infrastructure consulting firm in North America by bringing together some of the nation's best infrastructure consulting firms, focused on delivering practical and sustainable infrastructure solutions. Trilon is backed by Alpine Investors, a PeopleFirst Private Equity Firm. Trilon currently comprises 5,500+ staff across the US. For more information, visit www.trilon.com. Pay Transparency The base salary range for this role is indicated in the posting. This range reflects the company's good faith estimate of the compensation for this position at the time of posting. Final compensation will be determined based on factors such as experience, skills, qualifications, internal equity, and geographic location.

United States
$116K - $155K / year