• Create the conceptual, logical and physical data models. • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of sources like Hadoop, Spark, AWS Lambda, etc. • Lead and/or mentor a small team of data engineers. • Design, develop, test, deploy, maintain and improve data integration pipelines. • Develop pipeline objects using Apache Spark / Pyspark / Python or Scala. • Design and develop data pipeline architectures using Hadoop, Spark and related AWS Services. • Load and performance test data pipelines built using the above-mentioned technologies. • Communicate effectively with client leadership and business stakeholders. • Participate in proposal and/or SOW development

• Develop and maintain MLOps pipelines to automate the lifecycle of machine learning models. • Implement and optimize CI/CD pipelines specific to machine learning workflows. • Continuously monitor model performance in production, including logs, metrics, and alerts. • Design and apply strategies to detect and mitigate model performance degradation. • Ensure the security, governance, and compliance of data and models in production environments. • Collaborate with multidisciplinary teams to document and standardize MLOps-related processes.

• Assist in the analysis, design and development of a roadmap, design pattern, and implementation based upon a current vs. future state in a cohesive architecture viewpoint. • Gather and analyze data and develop architectural requirements at project level. • Responsible for multiple software projects as a project leader or internal consultant; support highly complex software projects that require in-depth domain knowledge or specialized architecture areas. • Participate in the data domain technical and business discussions relative to future architect direction. • Participate in the Data Governance Council. • Participate in development data and data delivery platforms that are service-oriented with reusable components that can be orchestrated together into different methods for different businesses. • Research and evaluate emerging technology, industry and market trends to assist in project development and/or operational support activities. • Effectively monitor and control costs in the organization.

Role Description We are hiring a Senior / Staff Data Engineer to build and evolve the data processing and pipeline layer that powers reporting, billing systems, and real-time data products at Index Exchange. This role focuses on designing and operating large-scale batch and streaming data pipelines , enabling reliable, scalable, and efficient data transformation across the platform. You will work on systems that transform raw, high-volume event data into clean, queryable, and production-grade datasets, supporting both API-driven data products and analytical workflows. - Process billions of events per day across distributed pipelines - Power core business datasets (reporting, billing, marketplace metrics) - Operate across batch (Spark) and streaming (Kafka / Flink) architectures - Require careful balancing of: - data correctness - processing efficiency - latency vs cost trade-offs You will solve problems such as: - Designing pipelines that scale without exploding compute costs - Managing data correctness at scale (deduplication, late data, joins) - Building systems that support both historical backfills and near real-time updates - Evolving pipelines from centralized processing (Hadoop) toward more distributed and efficient patterns - Streaming pipelines and Streaming DWs. Qualifications - Strong experience in data engineering at scale - Deep expertise in: - Spark (required) - SQL and data modeling - Experience with: - Airflow or workflow orchestration - Kafka or streaming systems - Strong understanding of: - distributed data processing - data modeling (large-scale datasets) - performance optimization - Ability to: - own pipelines end-to-end - debug complex data issues - work in high-scale, evolving environments Requirements - Define data processing standards and patterns across teams - Lead large-scale pipeline and platform initiatives - Influence data architecture and modeling decisions - Drive improvements across: - reliability - cost efficiency - scalability Benefits - Comprehensive health, dental, and vision plans for you and your dependents - Paid time off, health days, and personal obligation days plus flexible work schedules - Competitive retirement matching plans - Equity packages - Generous parental leave available to birthing, non-birthing, and adoptive parents - Annual well-being allowance plus fitness discounts and group wellness activities - Commuter benefits and discounts, where available - Employee assistance program - Mental health first aid program that provides an in-the-moment point of contact and reassurance - One day of volunteer time off per year and a donation-matching program - Bi-weekly town halls and regular community-led team events - Multiple resources and programming to support continuous learning - A workplace that supports a diverse, equitable, and inclusive environment