An online platform to test and advance your skills in penetration testing and cybersecurity. #ThinkOutsideTheBox
Senior Data Engineer
Location
United States
Posted
2 days ago
Salary
$140K - $160K / year
Seniority
Senior
Job Description
Senior Data Engineer
Hack The Box
Role Description Let's redefine cyber security expertise standards and connect business - community through highly engaging hacking experiences. ✨The core mission of the Senior Data Engineer: - Own and evolve data pipelines on GCP. - Build new pipelines, harden existing ones, and improve data quality. - Make clean, trustworthy data available across the organization. - Work end-to-end on streaming and batch pipelines. - Design ELT/ETL processes on BigQuery and ClickHouse. - Build real-time pipelines on Pub/Sub and Kafka with Dataflow. - Orchestrate workflows with Airflow. - Ensure data is properly cleaned, modeled, and served for analytics, ML training, and online inference. - Partner with ML engineers on feature pipelines and monitoring data drift. - Consume and build REST APIs and integrate with third-party SaaS sources. - Treat infrastructure as code. Qualifications - Strong data modelling and warehouse architecture skills. - Hands-on experience with GCP data services. - Production experience with streaming pipelines. - Solid SQL and strong Python skills. - Experience with ClickHouse or another columnar OLAP engine. - Workflow orchestration experience with Airflow. - Comfortable with dbt or equivalent transformation frameworks. - Experience migrating off legacy warehouses onto cloud-native stacks is a plus. - Working knowledge of ML in production. - Docker & Kubernetes experience. - CI/CD mindset and infrastructure-as-code sensibility. Requirements - Design and build batch and streaming pipelines on Dataflow, Pub/Sub, and Kafka. - Drive the migration off Snowflake onto GCP-native stack. - Own the orchestration layer in Airflow. - Model data for analytics and ML. - Partner with ML engineers on feature stores and retraining workflows. - Capture requirements from stakeholders and translate them into data products. - Continuously improve data quality, reliability, observability, and cost efficiency. - Identify new data sources worth acquiring and integrate them cleanly. Benefits - Compensation: $140,000-160,000. - Medical, Dental & Vision (employee coverage 100% paid). - 401K with employer match. - Employer-paid Life and AD&D Insurance. - Supplemental Life Insurance. - Short-term and Long-term Disability. - Healthcare and Dependent Care FSA. - Paid parental leave. - 25 annual leave days. - Home Office Allowance. - Dedicated budget for training and professional development. - State-of-the-art equipment. - Full access to the Hack The Box lab offerings.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Design, build, and maintain scalable and efficient cloud-based data pipelines. • Develop and support ETL/ELT processes for data integration and transformation. • Ensure compliance with information security and data governance standards. • Maintain and enhance cloud data environments, including data lakes and data warehouses. • Collaborate closely with engineering teams, product teams, and stakeholders. • Contribute to technical research, continuous improvement, and adoption of new technologies. • Maintain code quality using version control systems and CI/CD practices. • Support the strategic evolution of data engineering solutions and infrastructure.
Senior Software Engineer, Data Focused
Smart WorkingEmpowering companies to work with the best engineers in the world
• Support and maintain data ingestion pipelines, including connectors that pull data from NHS systems, practice management systems, and laboratory or equipment sources using APIs and screen-scraping. • Own and support the BigQuery ingestion layer and the downstream SQL Server and SQL Azure processing responsible for calculating dentist pay. • Proactively identify, investigate and resolve data ingestion issues while improving platform observability and monitoring. • Handle enterprise customer support requests and tactical enhancement requests for the existing platform. • Work across a highly distributed, brownfield, heavily customised environment supporting multiple bespoke customer deployments. • Review and occasionally adjust SQL-driven commands used by the legacy on-premises Delphi connector where required. • Contribute to tactical process improvements, including introducing source control through GitHub Enterprise and improving CI/CD practices using Azure DevOps. • Perform light-touch maintenance of the existing .NET web application. • Trace and resolve complex issues across stored procedures, Common Table Expressions (CTEs), SQL Server business logic and distributed systems. • Solve ambiguous production problems independently while collaborating within a Kanban team. • Help improve SDLC controls and engineering processes in an environment where production changes currently occur without CI/CD safeguards. • Deliver careful production changes while responding quickly to live customer issues.
AI-First Data Engineer
Sport Alliance GmbHDigitalization of the fitness industry - Leading service provider for SaaS, Fintech, and lead generation.
• Build and optimize cloud-native data pipelines on AWS — ETL/ELT and the infrastructure underneath (Aurora, Redshift, dbt, Spark/EMR, Airflow, CDC) — using AI tooling as a standard part of the workflow. • Design and evolve the data models that power analytics, operational use cases, and AI/ML across thousands of studios. • Help raise the bar on data reliability — embed governance, testing, and lineage so both people and AI systems can trust the numbers by default. • Partner with product, and other engineering teams to turn ambiguous business problems into robust data solutions. • Evaluate and evolve new approaches with us — data mesh patterns, and emerging AI tooling — and help decide what genuinely earns a place in our stack. • Grow into our financial and regulatory reporting workstream (Finion Capital), where correctness and auditability matter most.
AI-First Data Engineer
Sport Alliance GmbHDigitalization of the fitness industry - Leading service provider for SaaS, Fintech, and lead generation.
• Build and optimize cloud-native data pipelines on AWS — ETL/ELT and the infrastructure underneath (Aurora, Redshift, dbt, Spark/EMR, Airflow, CDC) — using AI tooling as a standard part of the workflow. • Design and evolve the data models that power analytics, operational use cases, and AI/ML across thousands of studios. • Help raise the bar on data reliability — embed governance, testing, and lineage so both people and AI systems can trust the numbers by default. • Partner with product, and other engineering teams to turn ambiguous business problems into robust data solutions. • Evaluate and evolve new approaches with us — data mesh patterns, and emerging AI tooling — and help decide what genuinely earns a place in our stack. • Grow into our financial and regulatory reporting workstream (Finion Capital), where correctness and auditability matter most.



