Senior Lead Data Engineer, Content Engineering
Location
New York
Posted
71 days ago
Salary
$156.8K - $235.2K / year
Seniority
Senior
Job Description
Senior Lead Data Engineer, Content Engineering
Paramount
• Build & Operate Large-Scale Feature Pipelines: Design and maintain batch/streaming pipelines (Spark, Flink, Databricks, Airflow) producing ML features for ranking models. • Ensure Point-in-Time Correctness: Develop feature sets that enable unbiased offline training and credible online inference. • Develop Embedding & Content Pipelines: Build scalable workflows for metadata, imagery, and multimodal representations; partner with Science teams to operationalize new models. • Architect Data Foundations: Design Delta/Parquet data models and medallion layers, optimizing storage layout and partitioning for latency and cost. • Real-Time Engineering: Build Kafka-based systems for real-time features and user-activity aggregations, ensuring robust handling of out-of-order events and exactly-once semantics. • Governance & Leadership: Define data quality rules and schema evolution processes while collaborating across ML pods to translate model needs into infrastructure.
Job Requirements
- 7+ years of experience in large-scale data or software engineering
- Deep experience with Spark (PySpark/Scala), Databricks, Airflow, and Kafka.
- Proficiency in feature pipelines, temporal joins, and mitigating training-serving skew.
- Experience with AWS/Azure/GCP and high-performance engines like Snowflake or Redshift.
- Proficient programming skills in Python and SQL with a focus on performance optimization.
- Experience in personalization domains (search, ranking, or recommender systems).
- Experience supporting petabyte-scale data lakehouses or feature stores.
- Familiarity with GenAI/RAG systems, multimodal content, or Delta Live Tables.
- Knowledge of Causal Inference, experimentation signals, or ML evaluation workflows.
- Experience with Terraform for governed, repeatable deployments.
Benefits
- medical
- dental
- vision
- 401(k) plan
- life insurance coverage
- disability benefits
- tuition assistance program
- PTO
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Design, develop, and maintain ETL/ELT data pipelines supporting enterprise data platforms. • Implement and enhance data ingestion, transformation, and integration processes using SQL and modern programming languages. • Design and implement automated data quality and validation checks (e.g., schema integrity, completeness, freshness, volume). • Embed data validation and testing logic directly into data pipelines and CI/CD workflows. • Refactor and modernize existing data pipelines to improve reliability, performance, and maintainability. • Collaborate with technical leads, architects, and engineering teams to define and promote standard data engineering and quality practices. • Support deployment, monitoring, and troubleshooting of data pipelines across development, test, and production environments. • Contribute to technical documentation, runbooks, and reusable components to improve team efficiency and consistency. • Participate in design and code reviews with a focus on sustainability, automation, and operational excellence.
• Design, build, and maintain scalable data pipelines and APIs on Google Cloud Platform • Develop automated workflows and data platforms that support analytics, reporting, and AI/ML use cases • Implement best practices for data security, governance, CI/CD, and automated deployment • Collaborate with data engineers, architects, data scientists, and business stakeholders • Produce high-quality, reusable code and mentor team members on best practices • Support testing, deployment, monitoring, and production troubleshooting
Senior Data Engineer
Hunt StWe help Aussie companies find top 3% remote talent in the Philippines & Nepal for a single finder's fee.
• Design, build, and maintain scalable data pipelines in modern lakehouse architectures. • Develop clean, efficient, and production-ready Python and SQL code. • Implement ETL/ELT processes, transformations, and orchestration workflows. • Model data using medallion architecture (Bronze/Silver/Gold), star schemas, and SCDs. • Integrate multiple data sources (APIs, databases, SaaS platforms, flat files). • Deploy pipelines using CI/CD tools and version control best practices. • Leverage AI tools (e.g., agent-based workflows, automation scripts) to improve delivery speed and quality. • Collaborate directly with clients on requirements, architecture, and delivery updates. • Monitor, troubleshoot, and optimise pipelines for performance and reliability. • Ensure data quality, integrity, and production readiness.
Senior Data Engineer
Hunt StWe help Aussie companies find top 3% remote talent in the Philippines & Nepal for a single finder's fee.
Role Description We’re looking for a Senior Data Engineer to join a high-performing delivery team and work on end-to-end data solutions across modern lakehouse environments. This is a hands-on role, focused on building and shipping production pipelines—not people management. You’ll work directly with clients and senior engineers, owning delivery from ingestion through to transformation and consumption. - Design, build, and maintain scalable data pipelines in modern lakehouse architectures. - Develop clean, efficient, and production-ready Python and SQL code. - Implement ETL/ELT processes, transformations, and orchestration workflows. - Model data using medallion architecture (Bronze/Silver/Gold), star schemas, and SCDs. - Integrate multiple data sources (APIs, databases, SaaS platforms, flat files). - Deploy pipelines using CI/CD tools and version control best practices. - Leverage AI tools (e.g., agent-based workflows, automation scripts) to improve delivery speed and quality. - Collaborate directly with clients on requirements, architecture, and delivery updates. - Monitor, troubleshoot, and optimise pipelines for performance and reliability. - Ensure data quality, integrity, and production readiness. Qualifications - 5+ years of professional experience in data engineering. - Must have experience with Databricks or Fabric. - Strong Python skills for production environments. - Advanced SQL (CTEs, window functions, performance tuning, complex joins). - Hands-on experience with modern data platforms (lakehouse, pipelines, distributed processing). - Experience with dimensional modelling and analytics-ready data design. - Solid understanding of CI/CD, Git workflows, and deployment practices. - Strong communication skills with the ability to work directly with clients. - Ability to translate business requirements into technical solutions. - AI Proficiency (Required) Requirements - This role requires strong, practical experience using AI in development workflows: - Comfortable working in modern development environments with AI-assisted tooling. - Proven experience building AI-powered tools, automations, or agents with real business impact. - Ability to use AI across the development lifecycle (design, coding, debugging, testing, documentation). - Strong judgment in validating and refining AI-generated outputs. - Experience with prompt design, context handling, and tool integrations. Nice to Have - Experience with integration platforms or middleware tools. - Exposure to dbt or modern data transformation frameworks. - Experience with cloud-based data ecosystems. - Dashboarding or semantic modelling experience. - Familiarity with AI/agent frameworks or orchestration tools. - Previous consulting or client-facing project delivery. Work Arrangement & Expectations This is a remote role that will be set up as an employer of record. To ensure alignment and transparency, successful candidates will be expected to: - Disclose any existing ongoing roles or client work. - Reflect this engagement on their LinkedIn profile.



