Cloud Confidently®
Senior Data Engineer
Location
United States
Posted
5 days ago
Salary
$150K - $180K / year
Seniority
Senior
Job Description
Senior Data Engineer
Effectual
• Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis • Develop real-time data pipelines that handle high-volume, high-velocity data streams • Implement event-driven architectures and microservices patterns for streaming data processing • Create and optimize data streaming topologies for complex event processing scenarios • Design fault-tolerant streaming systems with proper error handling and data recovery mechanisms • Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments • Implement Kafka Connect pipelines for streaming data integration • Design optimal Kafka topic partitioning strategies and replication configurations • Monitor and optimize Kafka cluster performance, throughput, and latency • Implement Kafka security configurations including SSL/TLS, SASL, and ACLs • Manage Kafka Schema Registry for data serialization and evolution • Design and implement Amazon Kinesis Data Streams and Kinesis Data Firehose solutions • Configure Kinesis Analytics applications for real-time stream processing • Optimize Kinesis shard management and auto-scaling configurations • Implement Kinesis data retention and archival strategies • Integrate Kinesis with other AWS services for comprehensive streaming solutions • Develop real-time stream processing applications using Apache Spark Streaming, Kafka Streams, or AWS Lambda • Implement complex event processing (CEP) patterns for real-time analytics • Build streaming ETL pipelines that transform data in motion • Create real-time aggregations, windowing operations, and stateful stream processing • Optimize streaming query performance and resource utilization • Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases • Implement data lineage and monitoring for streaming data pipelines • Create automated data quality checks and validation for streaming data • Manage data serialization formats (Avro, JSON, Protobuf) and schema evolution • Coordinate with data scientists and analysts to ensure streaming data meets analytical requirements • Implement Infrastructure as Code (IaC) for streaming data platforms using Terraform or CloudFormation • Automate deployment and management of streaming infrastructure through CI/CD pipelines • Monitor streaming system health, performance metrics, and alerting • Implement disaster recovery and high availability strategies for streaming systems • Stay current with emerging trends in streaming technologies and cloud-native solutions • Collaborate with data architects, data scientists, and application teams on streaming data requirements • Support rigorous project governance through daily progress reviews and time tracking • Provide technical leadership and mentorship to junior data engineers • Communicate complex streaming concepts to technical and non-technical stakeholders • Operate with transparency and responsiveness to support high-performing teams.
Job Requirements
- 7+ years of experience in the data engineering field with significant streaming data specialization
- Bachelor's degree in Computer Science, Engineering, or related STEM field
- Extensive hands-on experience with Apache Kafka including cluster management, performance tuning, and ecosystem tools
- Proven experience with AWS MSK and Amazon Kinesis services in production environments
- Strong background in real-time data processing and stream analytics.
Benefits
- Medical, dental, and vision health insurances,
- Short term disability, long term disability and life insurances,
- 401k with Company match
- Paid time off (PTO) (120 hours PTO that accrue over one year)
- Paid time off for major holidays (14 days per year)
- These and any other employee benefit offerings are subject to management’s discretion and may change at any time.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Design and implement scalable data platforms and pipelines across cloud environments • Develop reliable batch, streaming, and near-real-time pipelines • Build ingestion, transformation, and curation workflows for both structured and unstructured data • Implement modern data architectures including lakehouse patterns and medallion layering • Deliver high-quality datasets that support analytics, machine learning, causal modeling, and optimization systems • Design scalable logical and physical data models • Orchestrate workflows using tools such as Airflow, dbt, Lakeflow, or equivalents • Apply modern architecture patterns including event-driven and streaming architectures • Establish strong data observability • Enable data serving layers to support downstream systems • Collaborate with data scientists, ML engineers, analysts, and business stakeholders
• Data Platform Development & AI Enablement: You will build, maintain, and optimise ELT pipelines that ingest data from internal operational systems, APIs, third-party platforms, and event sources into Snowflake using tools such as Fivetran and cloud-native integrations. • Data Modelling, Transformation & Analytics Enablement: You will develop and maintain raw, curated, and business-ready data models within Snowflake, ensuring data is structured, documented, and optimised for analytics and self-service reporting. • Platform Optimisation & Continuous Improvement: You will help improve the reliability, scalability, performance, and observability of the modern data platform, identifying opportunities to optimise Snowflake usage, streamline workflows, improve automation, and enhance developer experience as TLC’s data capabilities continue to evolve. • Cross-Functional Collaboration: You will work closely with Analysts, Technology, Product, and Client teams to understand data requirements and deliver scalable data solutions that support business decision-making. • Operational System Integration: You will support integrations across TLC’s operational ecosystem, including COSMOS, Mixpanel, external plugins, and third-party platforms, helping ensure reliable, accurate, and scalable movement of data across business systems. • Data Quality, Governance & Engineering Best Practices: You will contribute to data governance, testing, monitoring, documentation, and engineering standards to ensure trusted, high-quality datasets and scalable development practices across the data platform. This includes version control, code reviews, automation, and continuous improvement of engineering processes.
Data Engineering Lead
Zup InnovationWe create digital assets to build, grow and accelerate your applications with efficiency, security and scalability.
• Design and implement reusable data platforms and systems with a focus on security • Define architectural standards for data flows, ensuring scalability and resilience • Lead the development of data ingestion, processing, and governance pipelines across cloud and on-premises environments • Propose and evolve integration solutions between dependent teams using modern data tools • Implement best practices for observability, cost control, and security in distributed environments • Support the dissemination of development standards, code review practices, and release automation
Role Description We are looking for an Analytics Consultant with sound experience in the development of data integration solutions. SQL and working knowledge of Data Integration techniques necessary (ETL, Replication, Profiling) to support business needs. - Develop and maintain SQL code and SSIS packages. - Analyze data and solve new and existing business issues. - Reviewing query performance and optimizing code. - Provide production level support. - Fully document all processes that are being created. - Sound experience on Microsoft on-premise data integration stack (SSIS, SQL, SQL Server). - Skilled in PowerBI. - Fluent in English, able to communicate both with the team and client. - Knowledge of insurance industry is a plus. - Reside in Argentina. Qualifications - Sound experience in data integration solutions. - Proficiency in SQL and Data Integration techniques (ETL, Replication, Profiling). - Experience with Microsoft on-premise data integration stack (SSIS, SQL, SQL Server). - Skilled in PowerBI. - Fluent in English. - Knowledge of the insurance industry is a plus. - Must reside in Argentina. Requirements - Experience developing and maintaining SQL code and SSIS packages. - Ability to analyze data and solve business issues. - Experience in reviewing query performance and optimizing code. - Capability to provide production level support. - Ability to fully document processes.
