Senior Data Engineer
Location
Florida
Posted
6 hours ago
Salary
0
Seniority
Senior
Job Description
Senior Data Engineer
Instructure
• End-to-End Pipeline Engineering: Design, build, and deploy scalable ETL/ELT pipelines from diverse source systems into our Snowflake Data Cloud. • Cloud Infrastructure: Manage and optimize data flows within an AWS environment (S3, Lambda, IAM), ensuring high availability, security, and cost-efficiency. • High-Scale Processing: Leverage Databricks and Python (PySpark) to handle complex transformations and high-volume data processing. • Implement the Semantic Layer: Collaborate with the team to define, implement and scale our Semantic Layer (via dbt Semantic Layer, MetricFlow, or similar) to standardize business logic, metrics, and dimensions for all downstream consumers. • Model for Truth: Use dbt to build modular, version-controlled, and tested data models that serve as the definitive foundation for business intelligence. • Data Governance & Quality: Implement automated testing and monitoring to ensure the integrity and reliability of finished data marts.
Job Requirements
- Data Warehousing: Expert-level proficiency in Snowflake (clustering, Snowpipe, streams, and tasks) or similar Cloud Data Warehouses.
- Analytics Engineering: Advanced mastery of dbt and complex SQL transformation logic, with specific experience in building Semantic Models and metric definitions.
- Big Data & Code: Strong Python skills and hands-on experience with Databricks for Spark-based orchestration.
- Cloud Infrastructure: Practical experience managing data workloads within AWS.
- Version Control: Deep understanding of Git-based workflows and CI/CD for data.
- A Data Quality Champion: You believe data is a liability until it’s governed. You have a passion for data modeling and reducing "metric drift" across the organization.
- Scale-Oriented & Efficient: You design for the long term. You take pride in building streamlined, cohesive data environments that eliminate redundancy and fragmentation. By focusing on modular design and automation, you ensure our infrastructure is never fragile, remaining stable and easy to navigate even as data complexity and volume grow.
- A Problem Solver: You find the root cause of data discrepancies and build automated solutions to prevent them from recurring.
Benefits
- Competitive compensation, plus all full-time employees participate in our ownership program - because everyone should have a stake in our success.
- Flexible work culture. Our remote, hybrid and in-office collaboration spaces vary by role, team and location.
- Generous time off, including local holidays and our annual “Dim the Lights” period in late December, when teams are encouraged to step back and recharge based on departmental needs.
- Comprehensive wellness programs and mental health support
- Learning and development resources, including professional development tools and tuition reimbursement, to support your growth
- The technology and tools you need to do your best work
- Motivosity employee recognition program
- A culture rooted in inclusivity, support, and meaningful connection
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Work with AI team members to operationalize data pipelines and ML tasks. • Provide day-to-day support of deploying Python-native ML pipelines and perform data engineering tasks to enable AI/ML capabilities. • Present results to a diverse audience in presentation or report form. • Support architectural leadership, technical support, and advisement services to ensure identity management system technologies are integrated and meeting the appropriate security requirements. • Support leadership who engage with senior level executives at a public facing Federal agency and provide subject matter expertise in security architecture and other key domain areas.
• Maintain and enhance the Snowflake data platform and data pipelines, leveraging AI-assisted development tools to improve efficiency, reliability, and maintainability. • Drive innovation by adopting modern engineering practices, AI-assisted development, and automation to continuously improve the data platform and accelerate delivery. • Design, build, and optimize incremental data integration pipelines with a strong focus on data quality, performance, scalability, and cost efficiency. • Implement and maintain data security controls, support audit and compliance requirements, and monitor platform usage and costs. • Collaborate closely with Engineering, Product, and Operations teams to deliver reliable, scalable data solutions.
Data Architect
Weekday (YC W21)We are a Y-Combinator-backed startup building your AI-powered Recruiter Agent
• Collaborate closely with internal technical teams to deliver multiple data initiatives simultaneously. • Design, develop, and optimize scalable data solutions using Python, Snowflake, and SQL. • Review and maintain high-quality code developed by offshore engineering teams. • Lead technical discussions and translate business requirements into scalable technical solutions. • Work directly with Business and IT stakeholders to gather, refine, and validate requirements. • Build data solutions independently whenever rapid execution is required. • Monitor project progress, identify risks proactively, and communicate updates to leadership. • Ensure project deliverables meet quality, performance, and timeline expectations. • Provide technical guidance and mentor team members to achieve successful project outcomes.
Senior Data Engineer
CapcoCapco, a Wipro company, is a management & technology consultancy dedicated to the financial services & energy industries
Job Title: Data Engineer About Us “Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery. WHY JOIN CAPCO? You will work on engaging projects with the largest international and local banks, insurance companies, payment service providers and other key players in the industry. The projects that will transform the financial services industry. MAKE AN IMPACT Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services. #BEYOURSELFATWORK Capco has a tolerant, open culture that values diversity, inclusivity, and creativity. CAREER ADVANCEMENT With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands. DIVERSITY & INCLUSION We believe that diversity of people and perspective gives us a competitive advantage. MAKE AN IMPACT Role Description Experience : 10-12 Years Job Location : Bangalore/Pune/Hyderabad Notice Period: 0 - 30 days JD: Role : - Key Responsibilities · Design, build, and maintain scalable cloud-native data pipelines for trade lifecycle data processing. · Develop and optimize ETL/ELT workflows for ingesting, transforming, and loading high-volume financial datasets. · Create and maintain robust data models and schema designs aligned to trade lifecycle (front-to-back processing). · Onboard and integrate DTCC data sources, including RTTM and Settlement Web, into a Unified Trade Record (UTR) schema. · Design and implement ingestion frameworks using messaging systems (MQ) and API-based data extraction. · Process and normalize financial messages, including ISO 15022 formats. · Populate and maintain Trade Exceptions databases, ensuring accuracy and consistency of exception workflows. · Collaborate with business stakeholders (operations, reconciliation, settlements teams) to align data structures with business processes. · Optimize SQL Server queries, indexing strategies, and database performance. · Work with AWS cloud-native data services (e.g., S3, EKS (Kubernetes/Docker), RDS) for scalable data solutions. · Ensure data quality, lineage, governance, and observability across pipelines. · Troubleshoot complex data issues and provide analytical insights into trade discrepancies and exceptions. Required Skills & Qualifications · 10+ years of experience in data engineering, with a strong focus on capital markets. · Deep understanding of trade lifecycle (execution, clearing, settlement, reconciliation). · Strong experience with ETL/ELT frameworks and large-scale data processing. · Expertise in data modeling and schema design, particularly for trade and transaction data. · Hands-on experience with DTCC data sources, especially RTTM and Settlement Web. · Strong knowledge of ISO 15022 message formats and financial data standards. · Experience building ingestion frameworks using MQ (e.g., IBM MQ, Kafka) and REST APIs. · Advanced SQL skills with strong experience in SQL Server (performance tuning, indexing, stored procedures). · Experience with AWS cloud-native data platforms and services. · Strong analytical, debugging, and problem-solving skills. Preferred Qualifications · Experience with unified data models such as Unified Trade Record (UTR) and Schema development and Trade Exceptions management experience. · Familiarity with data governance, lineage, and cataloging tools. · Experience with real-time or near-real-time data processing frameworks. · Knowledge of regulatory reporting and compliance requirements in capital markets. · Knowing FIX messages and processing FxML messages will be an add-on experience. · Prior consulting or client-facing experience. Soft Skills · Excellent communication and stakeholder management skills. · Strong attention to detail and data accuracy. · Ability to work independently and in collaborative environments. · Proactive approach to problem-solving and continuous improvement. If you are keen to join us, you will be part of an organization that values your contributions, recognizes your potential, and provides ample opportunities for growth. For more information, visit www.capco.com. Follow us on Twitter, Facebook, LinkedIn, and YouTube.




