Role Description The Relevancy Sciences Team is responsible for creating relevant and personalized customer experiences for Kroger's E-commerce platform, which ranks among the top 10 ecommerce companies in the US. We generate trillions of recommendations at scale and deliver them to millions of Kroger customers daily. Our team maintains a comprehensive portfolio of machine learning solutions for search & product recommendations. We are seeking a talented and experienced Senior ML Data Engineer to join our data science team, with specialized expertise in building search and recommender systems. You will architect, build, and operate the critical data infrastructure that powers our machine learning models, spanning from feature engineering to training data generation. This role serves as the bridge between ML requirements and production data systems, with ownership of feature stores, training/evaluation pipelines, and ML-specific data operations. You will enable data scientists to iterate rapidly while ensuring production-grade reliability and scalability. What You'll Do - Feature Store Operations & Governance (40%) - Own the feature request lifecycle from intake through deployment, driving reusability and maintaining a searchable feature catalog. - Design and build scalable feature pipelines that compute features from diverse sources (BigQuery, Azure Data Lake) and write to Feature Store infrastructure (Vertex AI Feature Store + BigQuery). - Build streaming feature engineering pipelines using Apache Beam/Dataflow for real-time feature computation and low-latency model serving with sub-second data freshness. - Ensure point-in-time correctness and online/offline feature consistency to prevent data leakage. - Implement drift detection, data quality monitoring, and alerting mechanisms. - Develop self-service tools and templates that enable teams to independently create features. - Training & Evaluation Data Pipelines (30%) - Build automated pipelines that generate ML-ready training datasets by combining features with labeled target variables. - Implement point-in-time correctness logic and sophisticated sampling strategies to ensure balanced, representative datasets. - Maintain comprehensive dataset versioning for full traceability across model versions. - Generate detailed evaluation reports with performance metrics segmented by business dimensions. - Support operations across both Azure and Vertex AI environments during platform migration. - ML Data Operations & Reliability (20%) - Serve as Tier 2/3 on-call responder for feature data quality incidents, diagnosing and resolving pipeline failures and performance issues. - Maintain comprehensive lineage tracking and metadata management for full data traceability. - Support regulatory compliance through proper data governance and documentation. - Standards, Education & Collaboration (10%) - Establish and enforce feature naming conventions, data quality thresholds, and point-in-time correctness patterns. - Conduct workshops on feature engineering best practices and provide expert guidance on feature design. - Partner with Data Scientists, ML Engineers, Data Engineering, and MLOps teams to optimize infrastructure and align with technical strategy. Qualifications - 3+ years of hands-on experience building and maintaining ML data pipelines in production environments with demonstrated expertise in scaling and reliability. - Expert-level SQL skills and advanced Python programming capabilities with experience in data processing frameworks and ML libraries. - Proven experience with cloud data platforms, with strong preference for GCP ecosystem including BigQuery, Dataflow, Vertex AI Feature Store, and associated ML services. - Deep understanding of end-to-end ML workflows including training data preparation, model evaluation methodologies, and serving infrastructure requirements. - Production operations mindset with experience in monitoring, alerting, on-call responsibilities, and meeting SLA commitments. Requirements - Hands-on experience with Feature Store platforms such as Vertex AI Feature Store, Feast, Tecton, or similar enterprise solutions. - Deep knowledge of point-in-time correctness principles, temporal joins, and time-series data modeling best practices. - Multi-cloud experience with both Azure and GCP platforms, including data migration and hybrid cloud architectures. - Strong familiarity with core ML concepts including feature engineering, label creation, train/test/validation splits, and data leakage prevention. - Background spanning both analytics engineering and ML-specific data engineering with understanding of the unique requirements of each domain. Success Indicators - Improved Data Science Productivity: Data Scientists spend significantly less time on data preparation and infrastructure concerns, enabling more focus on model development and experimentation. - Increased Feature Reuse: Measurable increase in feature reuse across multiple models and teams, reducing redundant development effort and improving consistency. - Reliable Automation: Training and evaluation data generation processes operate reliably with minimal manual intervention and high uptime. - Efficient Incident Response: Data quality incidents are triaged quickly with clear escalation paths and rapid resolution times. - Accelerated ML Iteration: Overall ML model development and iteration velocity improves measurably across all teams using the platform. Benefits - Health: Medical with competitive plan designs and support for self-care, wellness, and mental health. Dental and vision benefits available. - Wealth: 401(k) with Roth option and matching contribution. Health Savings Account with matching contribution (requires participation in qualifying medical plan). AD&D and supplemental insurance options. - Happiness: Paid time off with flexibility to meet your life needs, including 5 weeks of vacation time, 7 health and wellness days, 3 floating holidays, and 6 company-paid holidays per year. Paid leave for maternity, paternity, and family care instances. Pay Transparency and Benefits The stated salary range represents the entire span applicable across all geographic markets from lowest to highest. Actual salary offers will be determined by multiple factors including but not limited to geographic location, relevant experience, knowledge, skills, other job-related qualifications, and alignment with market data and cost of labor. In addition to salary, this position is also eligible for variable compensation. Pay Range: $97,000 - $166,750 USD