Job Closed
This listing is no longer active.
Shaping the next generation of e-commerce for the next-gen consumer.
Senior Data Engineer
Location
United States
Posted
68 days ago
Salary
$140K - $160K / year
Seniority
Senior
Job Description
Senior Data Engineer
StockX
• Design and build mission critical data pipelines with a highly scalable distributed architecture - including data ingestion (streaming, events and batch), data integration, data curation • Help continually improve ongoing reporting and analysis processes, simplifying self-service support for business stakeholders • Build and support reusable framework to ingest, integration and provision data • Automation of end to end data pipeline with metadata, data quality checks and audit • Build and support a big data platform on the cloud • Define and implement automation of jobs and testing • Optimize the data pipeline to support ML workloads and use cases • Support mission critical applications and near real time data needs from the data platform • Capture and publish metadata and new data to subscribed users • Work collaboratively with business analysts, product managers, data scientists as well as business partners and actively participate in design thinking session • Participate in design and code reviews • Motivate, coach, and serve as a role model and mentor for other development team associates/members that leverage the platform
Job Requirements
- 3 to 5 years' experience in data warehouse / data lake house technical architecture
- 3+ years of experience in using programming languages (Python / Scala / Java / C#) Minimum
- 3 years of Big Data and Big Data tools in one or more of the following: Batch Processing (e.g. Hadoop distributions, Spark), Real time processing (e.g. Kafka, Flink/Spark Streaming)
- Minimum of 2 years' experience with AWS or engineering in other cloud environments
- Strong Knowledge of Databricks SQL/Scala - Data Engineering Pipelines
- Experience with Database Architecture/Schema design
- Strong familiarity with batch processing and workflow tools such as dbt, AirFlow, NiFi
- Ability to work independently with business partners and management to understand their needs and exceed expectations in delivering tools/solutions
- Strong interpersonal, verbal and written communication skills and ability to present complex technical/analytical concepts to executive audience
- Strong business mindset with customer obsession; ability to collaborate with business partners to identify needs and opportunities for improved data management and delivery
- Experience providing technical leadership and mentoring other engineers for best practices on data engineering
- Bachelor's degree in Computer Science, or a related technical field.
Benefits
- medical
- dental
- equity
- discretionary bonuses
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Engineer II
CrunchtimeEmpowering restaurant brands to operate every location like their best location.
Global restaurant brands run their operation on the Crunchtime platform. Delivering a consistent guest experience across every location and managing food and labor costs are at the core of how Crunchtime’s software is used today in over 150,000 locations across 100+ countries by the world’s top restaurant and foodservice operators. Customers including Chipotle, Culver’s, Domino’s, Dunkin’, Five Guys and P.F. Chang’s rely on our top-ranked platform which now includes Zenput to manage inventory, staff scheduling, learning and development, food safety, operational tasks and audits. About the role Crunchtime is looking for a highly motivated Data Engineer to join our Connex product development team. The Connex team creates bi-directional real-time integration to every POS that our customers deploy. The ideal candidate is someone who enjoys integration, continual learning, and a fast-paced development environment. If you possess a drive for excellence, thrive in a fast-paced environment, and desire to be part of a team that is dedicated to being an industry leader, then we want to hear from you. What you’ll do as a Data Engineer - Own the planning, estimation, and delivery of data engineering tasks and projects, partnering with project management to track progress, milestones, and risks. - Design and implement scalable data pipeline solutions by analyzing customer data, identifying patterns, and performing data profiling to reverse engineer schemas and APIs. - Build, optimize, and maintain high-performance data pipelines, delivery networks, and bi-directional data integrations between customers and partners. - Ensure data reliability and quality through robust development of automated unit, integration, and regression tests. - Troubleshoot complex data and integration issues, providing escalated technical support and driving root cause resolution. - Collaborate cross-functionally to deliver integration solutions, and provide technical guidance, training, and documentation to internal teams and stakeholders. - Monitor data pipeline health, proactively identify performance or reliability issues, and implement improvements to ensure system stability. - Contribute to best practices in data engineering, including code quality, testing standards, and scalable architecture. What we’re looking for - 3+ years of professional experience in data engineering or software development with strong proficiency in SQL and Python. - Strong analytical and problem-solving skills with attention to detail and ability to work with complex datasets. - Experience designing and building scalable ETL/ELT pipelines, including working with APIs and handling structured and semi-structured data. - Solid understanding of data structures, algorithms, and data modeling concepts. - Experience with data validation, testing strategies (manual and automated), and ensuring data quality. - Strong communication and interpersonal skills, with the ability to work across teams and explain technical concepts clearly. - Ability to take ownership of tasks end-to-end, manage priorities, and deliver in a fast-paced environment. - Experience optimizing data pipelines for performance, scalability, and reliability. What you’ll get - Great mission-driven team members from diverse backgrounds with a strong company culture - Competitive pay - Unlimited PTO - Paid company holidays - Yearly team off-sites - International travel opportunities - Medical, dental, and vision benefits (FSA, HSA & HRA options) - Basic & Voluntary Life Insurance - 401k employer match - Wellness benefits (Headspace, OneMedical, Omada, Ginger.io, Gympass, Carrot) - Commuter benefits - Work in an open environment on solutions that are reshaping the way businesses operate - Fun team events - Ability to have a big impact - 10 weeks of paid parental leave - Fitness reimbursement - Learning & development funds Compensation: The annual base compensation range for this position is $75,000 - $105,000. The reasonably estimated total cash compensation (base + variable) range for this full-time position will be $80,625 - $112,875. Actual compensation is based on factors unique to each candidate, including, but not limited to: job-related skills, qualifications, education, experience, and location. Crunchtime has a location-based compensation system, so someone doing the same job in a different city might have a different pay range. Accessibility Accommodation Statement Crunchtime complies with federal and state disability laws and makes reasonable accommodations for applicants and employees with disabilities. If reasonable accommodation is needed to participate in the job application or interview process, to perform essential job functions, and/or to receive other benefits and privileges of employment, please contact recruiting@crunchtime.com. Equal Employment Opportunity Statement At Crunchtime, we are an equal opportunity employer. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, disability, veteran status, or any other characteristic. We have an inclusive and diverse workplace where every team member is respected and valued. As a global software development company, we believe that our diverse employee population enriches our work environment. Join us in our mission to provide the best software solutions to the food industry. We encourage you to apply regardless of your background or experience; even if you don’t meet all the qualifications, because we believe unique experiences and perspectives make us a stronger team.
Staff Software Engineer, Data Engineering
AirbnbAirbnb is a community based on connection and belonging.
Airbnb was born in 2007 when two hosts welcomed three guests to their San Francisco home, and has since grown to over 5 million hosts who have welcomed over 2 billion guest arrivals in almost every country across the globe. Every day, hosts offer unique stays and experiences that make it possible for guests to connect with communities in a more authentic way. The Community You Will Join: At Airbnb, we need to ensure every area of the business has trustworthy data to fuel insight and innovation. Understanding the business need, securing the right data sources, designing usable data models, and building robust & dependable data pipelines are essential skills to meet this goal. We are currently hiring for the following teams: The Data Stewardship team: is a group of passionate data practitioners with a diverse background in analytics, data modeling, governance, compliance, and scaled data quality. We are responsible for ensuring Airbnb is meeting its compliance obligations across our data ecosystem and ensuring data consumers are able to easily identify the best data for their needs. We support the pipelines, programs, and policy-bodies that make this possible. You’ll be part of the overall Data Infrastructure organization that is responsible for online and offline data infrastructure across the company, and the components that transition data between these environments. The Users and Contextualization Data & AI team: a vital part of the Marketplace Data & AI, focuses on building foundational data and systems to provide Airbnb teams with a deeper understanding of key domains. Specifically, this team concentrates on user data (Guest & Host), aiming to create high-quality, foundational, and well-governed user data and insights. These insights are crucial for enabling personalized and context-aware experiences that enhance trip quality both on and off the Airbnb platform, ultimately helping Airbnb better understand and serve its users throughout their journey. The Difference You Will Make: Data Stewardship: is foundational to Airbnb. High quality data is critical to our business decisions and the future of our AI initiatives. We own the overall strategy for data quality, identifying critical data and its provenance, measuring the effectiveness of our internal data products, and partnering with our core catalog team to provide offline interfaces to this data. Additionally we work across the organization to ensure our governance and compliance obligations are met in collaboration with infosec, privacy, and other stakeholders. The quality of data related to Users, Listings, and Marketplaces is foundational to Airbnb's success, directly impacting our products and business. To enhance our data foundation and maximize the value derived from our data and machine learning models, we are significantly expanding our Data Engineering efforts in these crucial areas. A Typical Day: - Architect and productionize batch and real-time data systems to support various products and business needs. - Ensure the quality, performance, and stability of data systems through robust quality systems and monitoring practices. - Design and optimize data models for efficient storage and retrieval to meet critical product and business requirements. - Collaborate with cross-functional teams, including product managers, engineers, data scientists, and business partners, to align on data requirements and develop scalable systems. - Tune, productionize, and optimize data systems and machine learning models to enhance their effectiveness and efficiency. - Build and maintain strong relationships with partner engineering teams, including backend, client, data science, and ML teams, to ensure seamless integration and support. - Contribute to the development of long-term data strategies and roadmaps, and influence the technical direction of data engineering practices within the organization. - Mentor and coach team members, providing guidance and support to enhance their skills and performance. Your Expertise: - 9+ years of relevant industry experience with a Bachelor’s and/or Master’s degree in CS/EE, or equivalent experience, or 6+ years of experience with a PhD - Extensive experience designing, building, and operating robust distributed data platforms (e.g., Spark, Kafka, Flink, HBase) and handling data at the petabyte scale. - Strong knowledge of Java, Scala, or Python, and expertise with data processing technologies and query authoring (SQL). - Proven ability to design, productionize, and optimize batch and real-time data pipelines and systems, ensuring their quality, performance, and stability. - Excellent ability to collaborate with cross-functional teams, including product managers, engineers, data scientists, and business partners, to align on requirements and drive data-driven decision-making. - Advanced analytical and problem-solving skills with a focus on data quality, governance, and system reliability. - Exceptional written and verbal communication skills, capable of influencing stakeholders and conveying complex technical concepts effectively. - Expertise in data modeling, warehousing, and working with relational (e.g., PostgreSQL, MySQL) and columnar databases (e.g., Redshift, BigQuery). - Experience working with machine learning engineers to integrate ML models into data systems and products (preferred but optional). - Ability to provide technical leadership and mentorship, guiding teams on best practices and contributing to the development of data engineering strategies. - Flexibility and innovative thinking to evaluate and incorporate new technologies and methodologies to improve data processes and solutions. Your Location: This position is US - Remote Eligible. The role may include occasional work at an Airbnb office or attendance at offsites, as agreed to with your manager. While the position is Remote Eligible, you must live in a state where Airbnb, Inc. has a registered entity. Click here for the up-to-date list of excluded states. This list is continuously evolving, so please check back with us if the state you live in is on the exclusion list. If your position is employed by another Airbnb entity, your recruiter will inform you what states you are eligible to work from. Our Commitment To Inclusion & Belonging: Airbnb is committed to working with the broadest talent pool possible. We believe diverse ideas foster innovation and engagement, and allow us to attract creatively-led people, and to develop the best products, services and solutions. All qualified individuals are encouraged to apply. We strive to also provide a disability inclusive application and interview process. If you are a candidate with a disability and require reasonable accommodation in order to submit an application, please contact us at: reasonableaccommodations@airbnb.com. Please include your full name, the role you’re applying for and the accommodation necessary to assist you with the recruiting process. We ask that you only reach out to us if you are a candidate whose disability prevents you from being able to complete our online application. How We'll Take Care of You: Our job titles may span more than one career level. The actual base pay is dependent upon many factors, such as: training, transferable skills, work experience, business needs and market demands. The base pay range is subject to change and may be modified in the future. This role may also be eligible for bonus, equity, benefits, and Employee Travel Credits. Pay Range $191,000—$225,000 USD
Senior Software Engineer, Data Engineering
AirbnbAirbnb is a community based on connection and belonging.
Airbnb was born in 2007 when two hosts welcomed three guests to their San Francisco home, and has since grown to over 5 million hosts who have welcomed over 2 billion guest arrivals in almost every country across the globe. Every day, hosts offer unique stays and experiences that make it possible for guests to connect with communities in a more authentic way. The Community You Will Join: We are currently hiring for multiple teams: Guest Data: Understanding how guests and hosts use Airbnb allows us to give them the best possible experience. The Guest Data team, part of the Guest & Host organization, is dedicated to developing top-tier tools, frameworks, and data models that offer a data-driven perspective on guest experiences on Airbnb products. Our primary stakeholders include analytics and data science partners, ML engineers, and product and engineering leaders involved in the core booking flow. Airbnb is powered by our Host and Guest communities. Our teams work on everything from driving improvements for the end-to-end Host and Guest experience to launching new product verticals from the ground up. Our job is to build world-class, comprehensive platforms and features to create high-quality experiences for our outstanding community of Hosts and Guests. Listings and Host Tools Data and AI (LHT-DnA) team is one of the 4 teams of the Marketplace-DnA org: The LHT-DnA team supports host personalization products and provides data driven solutions to achieve superior host experience on Airbnb. These products include but are not limited to managing your space (MYS), host quality standards etc. We own data pipelines and ML models and will build services for serving that are used in the above areas. Data Foundation: Our mission in the Foundational Data team is to build and maintain high quality datasets frequently used across all of Airbnb. We set company-wide standards that decide how locations are grouped into regions, visitors are measured based upon site traffic, bot traffic is separated from organic traffic, and cloud costs are attributed to Airbnb services. This data is used to build public financial reports, drive strategic marketing decisions, and manage operational costs. By collaborating with cross-functional teams, you will have the opportunity to drive significant business impact. The Difference You Will Make: Guest Data: As a Senior Data Engineer, you will help build our data strategy, develop top-tier data models and tooling, and provide technical leadership to a team of senior data engineers. You have the opportunity to influence critical aspects of the Airbnb experience, including Search and Checkout, while collaborating with long-standing xfn partners across Product, Analytics Engineering, Data Science, Infrastructure, and AI/ML. Your work will impact the quality of the Airbnb product experience for millions of daily visitors, making it an exciting opportunity for anyone passionate about data and its transformative potential. Data Foundation: You will bring a unique lens to our data strategy and provide in-depth technical mentorship and leadership to the team. We are looking for someone with experience that spans a wide range of skills centered around data. You will be responsible for developing data models that balance competing requirements, interacting with technical & non-technical data users, managing large-scale data projects, and maintaining data pipelines. A Typical Day: - Design, build, and maintain robust and efficient data pipelines that collect, process, and store data from various sources, including user interactions, listing details, and external data feeds. - Develop data models that enable the efficient analysis and manipulation of data for merchandising optimization. Ensure data quality, consistency, and accuracy. - Build scalable data pipelines (SparkSQL & Scala) leveraging Airflow scheduler/executor framework - Collaborate with cross-functional teams, including Data Scientists, Product Managers, and Software Engineers, to define data requirements, and deliver data solutions that drive merchandising and sales improvements. - Contribute to the broader Data Engineering community at Airbnb to influence tooling and standards to improve culture and productivity - Improve code and data quality by leveraging and contributing to internal tools to automatically detect and mitigate issues Your Expertise: - 5+ years of relevant industry experience with a BS/Masters, or 2+ years with a PhD - Experience with distributed processing technologies and frameworks, such as Hadoop, Spark, Kafka, and distributed storage systems (e.g., HDFS, S3) - Demonstrated ability to analyze large data sets to identify gaps and inconsistencies, provide data insights, and advance effective product solutions - Expertise with ETL schedulers such as Apache Airflow, Luigi, Oozie, AWS Glue or similar frameworks - Solid understanding of data warehousing concepts and hands-on experience with relational databases (e.g., PostgreSQL, MySQL) and columnar databases (e.g., Redshift, BigQuery, HBase, ClickHouse) - Excellent written and verbal communication skills Your Location: This position is US - Remote Eligible. The role may include occasional work at an Airbnb office or attendance at offsites, as agreed to with your manager. While the position is Remote Eligible, you must live in a state where Airbnb, Inc. has a registered entity. Click here for the up-to-date list of excluded states. This list is continuously evolving, so please check back with us if the state you live in is on the exclusion list. If your position is employed by another Airbnb entity, your recruiter will inform you what states you are eligible to work from. Our Commitment To Inclusion & Belonging: Airbnb is committed to working with the broadest talent pool possible. We believe diverse ideas foster innovation and engagement, and allow us to attract creatively-led people, and to develop the best products, services and solutions. All qualified individuals are encouraged to apply. We strive to also provide a disability inclusive application and interview process. If you are a candidate with a disability and require reasonable accommodation in order to submit an application, please contact us at: reasonableaccommodations@airbnb.com. Please include your full name, the role you’re applying for and the accommodation necessary to assist you with the recruiting process. We ask that you only reach out to us if you are a candidate whose disability prevents you from being able to complete our online application. How We'll Take Care of You: Our job titles may span more than one career level. The actual base pay is dependent upon many factors, such as: training, transferable skills, work experience, business needs and market demands. The base pay range is subject to change and may be modified in the future. This role may also be eligible for bonus, equity, benefits, and Employee Travel Credits. Pay Range $191,000—$223,000 USD
Senior Data engineer
ExperianBased in Dublin, Leinster, Ireland, Experian is a global information services company that operates in 40 countries around the world and has additional headquar
Company Description Experian is a global data and technology company, powering opportunities for people and businesses around the world. We help to redefine lending practices, uncover and prevent fraud, simplify healthcare, create marketing solutions, and gain deeper insights into the automotive market, all using our unique combination of data, analytics and software. We also assist millions of people to accomplish their financial goals and help them save time and money. We operate across a range of markets, from financial services to healthcare, automotive, agribusiness, insurance, and many more industry segments. We invest in people and new advanced technologies to unlock the power of data. As a FTSE 100 Index company listed on the London Stock Exchange (EXPN), we have a team of 22,500 people across 32 countries. Our corporate headquarters are in Dublin, Ireland. Learn more at experianplc.com. Job Description We are looking for an experienced Senior Data Engineer with expertise in AI‑powered tools, workflow automation, and modern data platform engineering. You will develop data software applications, data platforms, and data products. It partners with application engineers, data engineers, data scientists, and CRM teams to develop scalable solutions. The Engineer will contribute both independently and collaboratively. The Engineer's responsibilities will include defining engineering standards, performing data analysis and modeling, writing high‑quality code, conducting testing and code reviews, preparing deployments, ensuring operational readiness, and supporting Agile development practices. Additionally, the Engineer will keep Security first in mind. You will report to the Development and Operations Manager. - Lead the implementation of AI and LLM‑based tooling to automate data engineering workflows (e.g., code generation, data quality monitoring, schema drift detection). - Build automated frameworks for data ingestion, transformation, orchestration, and deployment using AI‑powered accelerators. - Integrate AI/ML capabilities into pipelines, including anomaly detection, metadata enrichment, and pattern recognition. - Develop automated data validation frameworks and implement AI‑assisted data quality checks. - Take technical ownership of all stages and iterations of assigned software components. - Design technology solutions meet all functional and non‑functional our requirements. - Provide ongoing production support by overseeing critical data pipelines, resolving incidents, ensuring system stability, and driving root‑cause analysis for continuous improvement. - Produce clear documentation of designs, architectural decisions, and code artifacts; participate in knowledge sharing. - Provide regular updates and communications to project partners. - Ensure solution adoption through training, documentation, and readiness planning. - Work across multiple projects and departments, supporting engineering tasks and cross‑functional programs. - Apply industry best practices, engineering standards, and governance principles. - Be a important contributor to deliver high‑value, on‑time, on‑budget complex projects. Qualifications - Bachelor's degree in Computer Science or equivalent experience. - 5+ years in commercial software development environments. - 5 + years of hands‑on experience in data engineering or similar roles. - Demonstrated experience using AI/LLM tools for code generation, testing, monitoring, and automation. - Hands‑on experience with: - PySpark, Python, SQL (expert‑level) - Data engineering pipelines, frameworks, and automation - Knowledge of data platform paradigms and modern software architecture. - Cloud experience with AWS, including services such as API Gateway, Lambda, EC2, ECS, SQS, SNS, DynamoDB, Redshift, and Aurora. - Understand detailed technical specifications and adapt quickly to new technologies Experience making any architectural tradeoffs required to provide software solutions. - Experience building scalable, resilient, and observable distributed systems and problem‑solving skills across a range of complexities. - Experience troubleshooting issues with users. - Familiarity with MLOps, ML orchestration, and model deployment workflows. Additional Information This is a permanent remote home-based role in Costa Rica. No relocation available. Our benefits include: Medical, life and dental insurance, Asociacion Solidarista, International Share Save Plan, Flex Work/Work from home, Paid time off, Annual Performance Bonus, Education Reimbursement, Family Bonding, Bereavement Leave, Referral Program, and more. Experian is proud to be an Equal Opportunity and Affirmative Action employer. Innovation is a critical part of Experian's DNA and practices, and our diverse workforce drives our success. Everyone can succeed at Experian and bring their whole self to work, irrespective of their gender, ethnicity, religion, colour, sexuality, physical ability or age. If you have a disability or special need that requires accommodation, please let us know at the earliest opportunity. #LI-ML2 #LI-Remote This is a remote position. - Employee Status: Regular - Role Type: Hybrid - Department: Product Development - Schedule: Full Time - Shift: Day Shift



