Powering access to brighter lives in Africa, Asia, and beyond
Data Engineer
Location
India
Posted
1 day ago
Salary
0
Seniority
Mid Level
Job Description
Data Engineer
Greenlight Planet
• ETL/ELT Pipeline Development: Build, and maintain scalable data pipelines using AWS. Implement both batch and incremental load patterns for BI reporting and application data needs. • Real-Time Data Streaming: Develop and manage real-time data ingestion pipelines using Kafka. Ensure low-latency, fault-tolerant data flow for critical business workflows. • Workflow Orchestration: Build, schedule, and monitor end-to-end data workflows using Apache Airflow. Manage dependencies, retries, and alerting for production DAGs. • Data Warehouse Management: Administer and optimize Amazon Redshift clusters including schema design, query performance tuning, distribution/sort keys, and vacuuming to ensure high availability and cost efficiency. • Data Quality & Observability: Implement automated data quality checks at ingestion and transformation stages. Define validation rules, build alerting for anomalies and discrepancies, and establish SLAs to ensure stakeholders can trust the data they use. • API Integrations: Integrate third-party and internal REST APIs into data pipelines to pull operational and product data into the warehouse. • Cloud Cost Optimization: Monitor and right-size data processing and storage resources across S3, EMR, Redshift, EC2, and Lambda. Proactively identify inefficiencies and propose cost-saving improvements. • BI & Analytics Collaboration: Partner with the BI team to align data models, preprocessing logic, and Redshift schema design with reporting and dashboard needs.
Job Requirements
- Bachelor’s degree in Computer Science or a related quantitative field.
- 2+ years of experience working as a Data Engineer
- Good proficiency in Python and SQL for data transformation and pipeline development
- Hands-on experience with Apache Spark (PySpark) for large-scale data processing
- Working knowledge of Kafka for real-time data ingestion and stream processing
- Hands-on experience managing and maintaining Airflow DAGs in production environments
- Familiarity with Redshift performance tuning, schema design, and query optimization
- Experience implementing automated data validation and quality checks within pipelines
- Detail-oriented with a keen interest in data transformations and their impact on business outcomes
- Problem-solving and time management skills
- Prior experience in project or team management is preferred, enthusiasm for mentoring and guiding others is a plus.
Benefits
- Professional growth in a dynamic, rapidly expanding, high-social-impact industry
- An open-minded, collaborative culture made up of enthusiastic colleagues who are driven by the challenge of innovation towards profound impact on people and the planet.
- A truly multicultural experience: you will have the chance to work with and learn from people from different geographies, nationalities, and backgrounds.
- Structured, tailored learning and development programs that help you become a better leader, manager, and professional through the Sun King Center for Leadership.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Engineer
CorpayCorpay helps companies control business expenses and make payments more simply, safely and securely than ever before.
Role Description Corpay is currently looking to hire a Data Engineer – Contractor to join our team. In this role, the Data Engineer - Contractor will join our Data & Analytics team and help support, enhance, and scale our enterprise data platform. This role will be responsible for developing and maintaining data pipelines, data models, and reporting datasets across our Microsoft Fabric environment, enabling reliable and actionable data for business stakeholders. The ideal candidate will have hands-on experience with Microsoft Fabric, Azure Data Factory, Azure SQL technologies, Data Warehouses, and Lakehouses. This role will focus on ensuring the stability, performance, and scalability of our data ecosystem while supporting ongoing business initiatives within the payments industry. Responsibilities - Supporting and enhancing enterprise data solutions built within Microsoft Fabric, ensuring reliability, scalability, and performance. - Developing, maintaining, and optimizing data pipelines using Microsoft Fabric and Azure Data Factory. - Building and managing data ingestion processes from internal and external data sources. - Supporting and maintaining Fabric Data Warehouses, Lakehouses, and related data assets. - Monitoring data pipelines and platform performance, proactively identifying and resolving issues. - Ensuring data quality, consistency, and governance standards are maintained across the platform. - Creating and maintaining technical documentation for data pipelines, data models, and operational processes. - Participating in platform modernization initiatives and continuous improvement efforts. - Supporting testing, deployment, and production validation activities for data engineering projects. - Working closely with analytics, reporting, and business intelligence teams to enable accurate and timely reporting. - Assisting in troubleshooting complex data integration and performance issues across multiple systems. Qualifications - 3–5 years of experience in Data Engineering, Data Integration, or a related technical field. - Hands-on experience with Microsoft Fabric, including Data Warehouses, Lakehouses, and data pipelines. - Knowledge of data modeling, data warehousing, and data integration best practices. - Experience troubleshooting and optimizing large-scale data processing workloads. - Strong analytical and problem-solving skills. - General payments industry experience, including exposure to payment processing, financial services, fintech, commercial payments, AP automation, or related domains. - Familiarity with broad Azure data services. - Experience working in fintech, payments, banking, or financial services organizations. - Microsoft Fabric, Azure, or related cloud certifications. Benefits - Company-issued equipment. - Hands-on training.
Lead Data Engineer
AccentureAccenture is a leading global professional services company that helps the world’s leading businesses, governments and other organizations build their digital core, optimize their operations, accelerate revenue growth and enhance citizen services—creating tangible value at speed and scale. We are a talent- and innovation-led company with approximately 791,000 people serving clients in more than 120 countries. Technology is at the core of change today, and we are one of the world’s leaders in helping drive that change, with strong ecosystem relationships. Our broad range of services, solutions and assets across Strategy & Consulting, Technology, Operations, Industry X and Song, together with our culture of shared success and commitment to creating 360° value, enable us to help our clients reinvent and build trusted, lasting relationships.
Role Description Accenture Flex offers you the flexibility of local fixed-duration project-based work powered by Accenture, a leading global professional services company. As an Accenture Flex employee, you will apply your skills and experience to help drive business transformation for leading organizations and communities. Responsibilities: - Responsible for the design and implementation of scalable & fault tolerant data applications on on-prem & AWS to store and process terabytes of data from upstream sources with high availability. - Partner with information architects, platform architects, data scientists, and product management on solution requirements to design solutions. - Hands-on experience with Advanced Ab Initio concepts, including ETL project development, solution design, metaprogramming, PDL. - Design generic ETL frameworks using Ab Initio, coupled with strong hands-on involvement in audit and reconciliation workflows. - Strong cloud engineering skills with AWS, including hands-on experience with S3, Amazon Redshift and related data services. - Work with batch and real-time data pipelines in a DevOps environment. - Hands-on understanding and working knowledge of the Hadoop ecosystem, including Hive. - Utilize innovative frameworks to avoid redundancy by promoting automation. - Identify enablers and level of effort required to properly ingest and transform data for the DWH & data. - Stay up to date on latest trends in data engineering, coach team members and recommend best practices, insights and/or expertise in deliverables. - Work closely with Product Owners, Product Managers, Program Manager, Scrum Masters and Team Members in a Scaled Agile framework. - Develop and maintain strong collaborative working relationships with Technology and Business Stakeholders. Qualifications - Minimum of 5 years of data warehouse and ETL Ab Initio experience, including hands-on technical and lead capabilities. - Minimum of 5 years of experience working with Ab Initio GDE, Conduct IT, Meta Programming, PDL, Hadoop, Shell Scripting, and SQL. - Minimum of 5 years of experience working in the banking industry. - High School Diploma or GED. Requirements - Bachelors or Associates degree (preferred). - Cloud Migration experience and PySpark (preferred). Benefits - Market competitive suite of benefits including medical, dental, vision, and long-term disability coverage. - 401(k) plan. - Paid time off.
Comm Ops Data Engineer
Johnson & Johnson Innovative MedicineAt Johnson & Johnson, we believe health is everything. Our strength in healthcare innovation empowers us to build a world where complex diseases are prevented, treated, and cured, where treatments are smarter and less invasive, and solutions are personal. Through our expertise in Innovative Medicine and MedTech, we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow, and profoundly impact health for humanity.
Role Description Johnson & Johnson is hiring for a Comm Ops Data Engineer – Shockwave Medical to join our team. The position is FULLY REMOTE and can sit anywhere in the US. As a Data Engineer reporting to the Sr. Manager, Data Engineering, you will play a pivotal role in enabling our business groups to access and interpret meaningful data. You will need to bridge detailed knowledge of our data with an understanding of end users’ needs, crafting accessible, well-structured data for reporting, advanced analytics, and use in AI modeling. Essential Job Functions - Develop and maintain SQL-based data pipelines to extract, transform, and load (ETL) data from enterprise data warehouses. - Aid in semantic model and view development within Snowflake to enable advanced, self-service analytics aligned to the newest analytical practices. - Collaborate with team members to understand data needs and provide technical solutions. - Experiment with new data engineering techniques and tools, embracing the unknown to drive innovation and improvement. - Monitor data systems and implement enhancements driving performance and efficiency. - Coordinate with IT to ensure seamless flow of data across systems, maintaining high standards of data availability and security. This is a hybrid/remote role, depending on location and business needs. Travel up to 20% may be required for business needs. Qualifications - Bachelor’s degree and 2+ years of experience as a Data Engineer or similar technical role. - Expertise in SQL and ETL tools for data transformation and warehouse management. - Experience working with Snowflake; Streamlit experience a plus. - Understanding of Power BI or other business intelligence tools. - Familiarity with data governance, security, and compliance best practices. - Propensity to problem-solving and ability to optimize data applications. - Understanding of Python, R, or other scripting languages. - Biotech, Medical Device, Medtech, Life Sciences experience a plus. Requirements - Custom Programs - Data Analysis - Data Pipelines - Data Visualization - Impact Management Benefits - Vacation – 120 hours per calendar year - Sick time - 40 hours per calendar year; for employees who reside in the State of Colorado – 48 hours per calendar year; for employees who reside in the State of Washington – 56 hours per calendar year - Holiday pay, including Floating Holidays – 13 days per calendar year - Work, Personal and Family Time - up to 40 hours per calendar year - Parental Leave – 480 hours within one year of the birth/adoption/foster care of a child - Bereavement Leave – 240 hours for an immediate family member; 40 hours for an extended family member per calendar year - Caregiver Leave – 80 hours in a 52-week rolling period - Volunteer Leave – 32 hours per calendar year - Military Spouse Time-Off – 80 hours per calendar year
Role Description The Engineer III is an experienced software engineer who independently designs and delivers complex features with minimal oversight. This role applies strong technical judgment to solve difficult problems, upholds team engineering standards through code review and documentation, and begins to mentor junior engineers. The Engineer III is a reliable owner of significant work and a growing technical voice within their team. ESSENTIAL FUNCTIONS AND BASIC DUTIES: Technical Execution & Excellence - 80% - Independently design and implement complex features and improvements from requirements through production. - Own medium-to-large features and initiatives end-to-end with minimal oversight. - Write high-quality, well-tested, maintainable code that serves as a model for junior engineers. - Lead code reviews, providing thorough and constructive technical feedback. - Diagnose and resolve complex bugs and production issues, including those spanning multiple systems. - Identify and address technical debt within their area; propose improvements with clear rationale. - Make sound technical decisions within well-understood domains; escalate architectural decisions appropriately. - Build deep knowledge of the team's systems and contribute to architectural discussions. - Drive improvements to team engineering practices - testing, observability, CI/CD, and code quality. - Contribute to technical documentation, runbooks, and design documents. - Participate in incident response and post-incident review within their domain. - Balance delivery speed with code quality and long-term maintainability. Team Collaboration & Technical Contribution - 15% - Communicate progress, risks, and technical tradeoffs clearly to team and stakeholders. - Collaborate with Product and QA on requirements, acceptance criteria, and feature quality. - Mentor Engineer I and Engineer II teammates through pairing, code review, and coaching. - Participate in technical interviews and hiring screens. - Help decompose and estimate complex work for sprint and quarterly planning. - Share knowledge through documentation, design documents, and team discussions. - Partner with Platform on infrastructure and tooling needs for features they own. Technical Growth & Influence - 5% - Contribute to team-level technical direction and approach. - Propose improvements to engineering processes, tooling, and standards. - Participate in architecture and design reviews for teamwork. - Develop breadth across the team's systems beyond their primary ownership area. SECONDARY FUNCTIONS (IF APPLICABLE) - May work on special projects or other duties as assigned. - Research and develop necessary skills to support organizational objectives. SUPERVISORY/BUDGETARY/EXTERNAL COMMUNICATION RESPONSIBILITY - No direct supervisory responsibility. - Communicates with immediate team and cross-functional partners (Product, QA, Platform) and external vendors as needed. Qualifications - Bachelor's Degree in Computer Science, related technical field, or equivalent practical experience, required. - 4-6 years of professional software engineering experience. - Proven experience independently delivering complex features from design through production deployment. - Experience working with distributed systems, APIs, data pipelines, or cloud infrastructure at production scale. - Experience providing meaningful code review feedback and beginning to mentor junior engineers. Requirements - Ability to adhere to and exhibit the Company Values at all times. - Solid proficiency in the technology stack and domain relevant to the team (e.g. application, data, or cloud/infrastructure engineering). - Understanding of CI/CD pipelines, observability practices, and modern software delivery. - Familiarity with healthcare or regulated industries (HIPAA, SOC2) preferred. - Solid proficiency in programming languages and tools relevant to the team's domain. - Experience designing and building production systems - APIs, data pipelines, cloud infrastructure, or web applications depending on discipline. - Ability to independently design and deliver complex features with minimal guidance. - Strong code review skills; provides thorough, constructive technical feedback. - Experience writing unit, integration, and end-to-end tests; advocates for appropriate test coverage. - Understanding of cloud platforms, CI/CD pipelines, observability, and software delivery best practices. - Ability to diagnose and resolve complex bugs spanning multiple systems or services. - Emerging technical leadership - can guide junior engineers and contribute to team technical direction. - Clear written and verbal communication; can document design decisions and tradeoffs. - Collaborative and accountable; owns outcomes, not just tasks. - Solid critical thinking and creative problem-solving skills. - Strong organizational and time management skills. - Flexibility and adaptability to change. - Working knowledge of Microsoft Word, Excel, PowerPoint, Outlook, and Teams. - Solid understanding of software design fundamentals (data structures, algorithms, design patterns). - Eagerness to learn and apply new technologies and frameworks. - Ability to consistently meet goals, commitments, and deadlines. - Ability to work with sensitive information and maintain confidentiality. Benefits - Company-paid benefits (Basic Life and AD&D, Short and Long-Term Disability, Employee Assistance Program, Compass Health Advocate and Transitions). - Healthcare benefit options (Value Plan, High Deductible Plan with HSA, Healthcare FSA, Dependent Care FSA, Prepaid Legal Services, 529 Savings Plan, Pet Insurance). - Paid parental leave. - Company sponsored 401k plan with company matching. - PTO that accrues at a rate of 15 days/year for 1st year and continues to increase with tenure. Disclosures - Smoking/vaping and the use of tobacco products are prohibited on all Company premises, including indoor and outdoor areas, parking lots, and Company-owned vehicles. - As part of our employment process, candidates who receive a conditional offer may be required to undergo pre-employment drug testing. - We are an Equal Opportunity Employer and do not discriminate based on race, color, religion, sex, national origin, age, disability, veteran status, or any other protected status under the law.


