Job Closed
This listing is no longer active.
We are an equal opportunity employer with a commitment to diversity. All individuals, regardless of personal characteristics, are encouraged to apply. All qualified applicants will receive consideration for employment without regard to age, race, color, national origin, ancestry, sex, sexual orientation, gender, gender identity, gender expression, marital status, pregnancy, religion, physical or mental disability, military or veteran status, genetic information, or any other status protected by applicable state or local law.
Lead ETL Data Engineer
Location
India
Posted
109 days ago
Salary
0
Seniority
Senior
Job Description
Lead ETL Data Engineer
Empower
• Own the overall design and technical direction of ETL pipelines and backend data workflows using Python and SQL across multiple projects or domains • Architect and oversee large-scale, highly available data processing solutions on AWS using services such as Glue, EMR, Lambda, and S3 • Define, standardize, and continuously improve CI/CD pipelines and engineering practices for data and application code across the team • Partner with enterprise and solution architects to ensure alignment with approved architecture patterns and long-term data platform strategy • Collaborate with product and business stakeholders to shape technical roadmaps and ensure solutions meet functional, performance, and security requirements • Serve as the primary escalation point for complex technical issues, data quality challenges, and performance bottlenecks that span teams or systems • Lead code reviews and technical design reviews, enforcing high standards for code quality, testing, documentation, and operational readiness • Mentor and coach engineers at all levels, building technical capability within the team and promoting a strong engineering culture • Drive continuous improvement in data and application architecture, development processes, observability, and reliability practices • Provide technical leadership during production incidents, root cause analysis, and long-term remediation efforts • Evaluate and champion new tools, frameworks, and approaches, including proof-of-concepts, to improve scalability, reliability, and delivery speed of the data platform.
Job Requirements
- Bachelor’s degree in Computer Science or related field, or equivalent practical experience
- 7+ years of experience in software development, with deep experience in backend or data engineering and exposure to complex, distributed systems
- Expert-level proficiency in Python for backend development and data processing
- Expert-level proficiency in SQL, including advanced optimization, data modeling, and working with large-scale or complex datasets
- Extensive hands-on experience with AWS data and compute services, including Glue, EMR, Lambda, and S3, and kafka topics
- Proven experience designing and governing CI/CD pipelines and engineering standards for teams or multiple projects
- Strong understanding of data and software architecture, including patterns for scalable, resilient, and secure data-intensive systems
- Demonstrated track record of technical leadership, including mentoring, setting standards, and driving architecture or platform decisions
- Strong problem-solving and analytical skills with experience leading resolution of complex, cross-team issues
- Excellent communication and influencing skills, with the ability to work effectively across engineering, architecture, and business stakeholders.
Benefits
- Flexible work environment
- Internal mobility celebration
- Purpose and well-being recognition
- Work-life balance emphasis
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
ETL Data Engineer
EmpowerWe are an equal opportunity employer with a commitment to diversity. All individuals, regardless of personal characteristics, are encouraged to apply. All qualified applicants will receive consideration for employment without regard to age, race, color, national origin, ancestry, sex, sexual orientation, gender, gender identity, gender expression, marital status, pregnancy, religion, physical or mental disability, military or veteran status, genetic information, or any other status protected by applicable state or local law.
• Design, build, and maintain ETL pipelines and backend data workflows using Python and SQL • Develop and optimize data processing solutions on AWS using services such as Glue, EMR, Lambda, and S3 under guidance from senior engineers and architects • Implement and support CI/CD pipelines to automate build, test, and deployment processes • Collaborate with data engineers, architects, product owners, and business partners to understand requirements and translate them into clear technical tasks • Write clean, maintainable, and well-tested code that aligns with team standards and architectural guidelines • Troubleshoot and resolve data, performance, and reliability issues in lower and production environments with support from senior team members as needed • Participate in code reviews to provide and receive feedback and help uphold coding and quality standards • Contribute to design and technical discussions by sharing ideas and raising risks or improvement opportunities • Continuously learn and adopt tools, technologies, and development practices relevant to data engineering and cloud platforms
• Create and edit logical and physical data models, XML schemas, and file formats using ER/Studio, XML Spy, or equivalent tools. • Build tables, define columns and data types, and perform database scripting (DDL, inserts, updates, data movement). • Analyze and profile data sources; develop source-to-target mappings with detailed transformations and business rules using Erwin Mapping Manager or similar. • Collaborate with business and technical stakeholders to define transformation rules and data flow, ensuring mapping rules and data governance are followed. • Write SQL scripts to analyze, validate, or correct data, including complex stored procedures, views, and functions. • Monitor data quality, investigate issues, and execute remediation plans. • Provide guidance to teams on solutions, enterprise architecture best practices, and standards. • Review project deliverables for quality and compliance; recommend improvements.
• Develop and optimize PySpark-based ETL pipelines for processing large-scale data • Develop, test, and maintain robust Python applications • Collaborate with data engineers, analysts, and other developers to design and implement data solutions • Work with Azure Data Lake, Azure Databricks, and Azure Data Factory to build scalable data solutions • Implement data classification, policy enforcement, and metadata extraction within Purview • Collaborate with data engineers and business teams to ensure smooth data flow across systems • Troubleshoot performance bottlenecks in Spark jobs and improve data pipeline efficiency • Ensure data security, compliance, and quality following best practices.
• Perform Data Warehouse Architecture at an intermediate level • Focus on design and development of data models and ETL specifications • Conduct data analysis, system design and architecture • Propose requirements, standards, and best practices for data modeling • Conduct product and vendor research, and present recommendations
