The Nation’s Leading Residential and Commercial Solar Performance Plan And Cash-Back Energy Guarantee
Manager, Data Engineering
Location
California
Posted
38 days ago
Salary
$124K - $155K / year
Seniority
Senior
Job Description
Manager, Data Engineering
Omnidian
• Lead, mentor and coach a team of six Data Engineering professionals • Own workload management for the Data Engineering team, including intake, prioritization, resource allocation • Ensure team has measurable SMART goals and understands their impact on company objectives • Collaborate cross functionally to drive initiatives • Establish and provide visibility for performance metrics
Job Requirements
- 5+ years experience as a Data Engineer
- 3+ years experience as a people manager
- Proficiency at successfully managing large scale data engineering projects
- Hands on experience building data pipelines within a structured deployment environment
- Experience with Agile project management
- Familiarity with distributed systems like Spark, Kafka and data lakes
- Experience with cloud platforms like AWS or Azure
- Understanding of data modeling principles
Benefits
- Health insurance coverage (100% of employee premiums, 50% of dependent premiums)
- 401(k) administration with $1k match per year
- Performance bonus opportunities
- Stock options for employees
- Up to $500 in annual learning reimbursement for courses, certifications, or conferences
- 12 weeks of paid parental leave after 1 year of employment
- Four-week paid sabbatical leave after 4 years of employment
- Flexible work arrangements with opportunities to work remotely
- Affinity groups to help employees feel seen and supported
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Data Collection: Collect data from various internal and external sources, following established procedures and proposing process improvements when necessary. • Data Preparation: Perform data cleaning, transformation, and integration tasks to ensure data quality and consistency with minimal supervision. • Project Support: Collaborate with team members on the development and maintenance of data pipelines, taking on larger responsibilities and leading small subprojects. • Process Documentation: Make significant contributions to data engineering process documentation by recording procedures and workflows and ensuring they remain up to date. • Exploratory Data Analysis: Conduct initial data analyses to identify patterns, trends, and business-relevant insights with greater autonomy. • Query Support: Execute more complex SQL queries and extract information from relational databases, providing technical support to the team. • Quality Monitoring: Proactively monitor data quality, identify and resolve issues or inconsistencies, and suggest improvements to quality control practices. • Continuous Learning: Participate in and contribute to training and development programs, and mentor junior team members. • Cross-Functional Collaboration: Work effectively with other departments to ensure proper data integration into solutions and systems, contributing specialized knowledge. • Meeting Deadlines: Manage tasks and ensure completion within established deadlines, proactively communicating any challenges or obstacles to team leadership.
Data Engineer – Web Scraping, ETL
TechBiz GlobalTechBiz Global is a leading IT recruitment and software development company
• Monitor and manage overnight scraper and ingestion runs, triaging failures and applying fixes in real time to minimize data gaps before US market open • Verify data completeness and quality across all automated feeds, flagging anomalies and coordinating with the Houston team on persistent issues • Maintain run logs, error documentation, and escalation notes for seamless async handoffs • Build and maintain scrapers, parsers, and ingestion pipelines across a growing set of energy market data domains • Contribute to the design and build-out of our broader ETL infrastructure, including scheduling, orchestration, and error handling • Write transformation logic to clean, normalize, and load raw data into PostgreSQL staging and production tables • Optimize existing pipelines for performance, reliability, and cost efficiency • Help build monitoring dashboards and alerting for pipeline health and data freshness • Document data lineage, schema changes, and pipeline dependencies
• Transform financial lives through innovative data solutions • Support internal mobility and create a diverse environment
• Lead data architecture design, API assessment, and ETL requirements gathering during the Discovery & Design phase. • Develop and configure CMIC ERP API integration to establish reliable data exchange between the ERP system and the AWS platform. • Design and implement data pipelines using AWS Glue for ETL processing of subcontractor documents and ERP data. • Integrate Amazon Textract to extract structured data from insurance certificates, bonding letters, and financial documents. • Build and maintain data models to support AI-powered validation, risk profiling, and executive reporting. • Configure Amazon S3 data lake architecture to store and manage raw, processed, and curated data assets. • Implement AWS Lambda and AWS Step Functions to orchestrate data workflows and automated processing pipelines. • Develop and expose data through Amazon API Gateway to support application and dashboard consumption. • Ensure data quality, validation, and integrity across all integration points and pipeline outputs. • Conduct data integration testing and support user acceptance testing (UAT) for data-dependent features. • Collaborate with Full Stack, AI/ML, and DevOps team members to ensure seamless end-to-end data flows. • Contribute to knowledge transfer documentation, data pipeline runbooks, and operations guides.




