Job Closed
This listing is no longer active.
People and Company Intelligence
Data Engineer
Location
United States
Posted
139 days ago
Salary
$160K - $180K / year
Seniority
Senior
Job Description
Data Engineer
People Data Labs
• Build infrastructure for ingestion, transformation, and loading an exponentially increasing volume of data from a variety of sources using Spark, SQL, AWS, and Databricks • Building an organic entity resolution framework capable of correctly merging hundreds of billions of individual entities into a number of clean, consumable datasets. • Developing CI/CD pipelines and anomaly detection systems capable of continuously improving the quality of data we're pushing into production. • Dreaming up solutions to largely undefined data engineering and data science problems.
Job Requirements
- 4-6+ years of industry experience with clear examples of strategic technical problem-solving and implementation
- Strong software development fundamentals.
- Experience with Python
- Expertise with Apache Spark (Java, Scala, and/or Python-based)
- Experience with SQL
- Experience building scalable data processing systems (e.g., cleaning, transformation) from the ground up.
- Experience using developer-oriented data pipeline and workflow orchestration (e.g., Airflow (preferred), dbt, dagster or similar)
- Knowledge of modern data design and storage patterns (e.g., incremental updating, partitioning and segmentation, rebuilds and backfills)
- Experience working in Databricks (including delta live tables, data lakehouse patterns, etc.)
- Experience with cloud computing services (AWS (preferred), GCP, Azure or similar)
- Experience with data warehousing (e.g., Databricks, Snowflake, Redshift, BigQuery, or similar)
- Understanding of modern data storage formats and tools (e.g., parquet, ORC, Avro, Delta Lake)
Benefits
- Stock
- Competitive Salaries
- Unlimited paid time off
- Medical, dental, & vision insurance
- Health, fitness, and office stipends
- The permanent ability to work wherever and however you want
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Ingeniero de Datos
Metova, Inc.Helping companies transform their business through technology to meet the growing expectations of their customers.
• Design, develop, and maintain data pipeline architectures. • Optimize data ingestion, storage, and processing workflows. • Collaborate with data scientists and analysts to understand data needs and convert requirements into technical specifications. • Ensure data quality and integrity throughout the data lifecycle. • Implement data security and compliance measures. • Monitor and troubleshoot data systems performance issues. • Stay up-to-date with industry trends, technologies, and best practices in data engineering.
• Enable efficient data access by creating and maintaining data pipelines. • Collaborate with ML engineers to design and maintain automation for machine learning training, quality assessment, and model release process. • Build data infrastructure from the vast amount of data for analytics, hypothesis testing and company metrics. • Identify, design and implement improvement to internal processes allowing to optimize data delivery, automate manual processes. • Design new and improve current patterns for building data models and implement necessary modifications.
Big Data Engineer
Sigma Software GroupWe support enterprises, product houses, and startups with custom software solutions development and IT consulting.
• Develop and maintain ETL pipelines and data integration services using Python and SQL • Work with AWS services (S3, DynamoDB, Lambda, Glue) and NoSQL databases (MongoDB, DynamoDB) • Design, optimize, and validate data flows, ensuring data quality across systems • Collaborate with Senior engineers on architecture and performance improvements • Troubleshoot production data issues and perform root cause analyses • Contribute to the continuous improvement of development practices and performance monitoring
• Design, build, and maintain scalable data pipelines that ingest, clean, transform, and aggregate data from disparate sources across the VA ecosystem • Develop and implement data models that support analytical use cases and measurement frameworks • Partner with data scientists, service designers, and product managers to translate analytical requirements into infrastructure that delivers analytics-ready datasets • Implement data quality checks, validation processes, monitoring, and governance practices to ensure data accuracy and integrity • Communicate technical concepts and infrastructure decisions using plain language to non-technical stakeholders to support effective decision-making and gain buy-in where needed • Serve as a technical leader, mentor junior practitioners, and advance how data engineering supports human-centered design • Stay current on data engineering developments and bring best practices into constrained technical and policy environments




