Teamwork | Excellence | Integrity | Diversity, Equity and Inclusion | Community Investment
Data Engineer II
Location
Texas
Posted
3 days ago
Salary
$58K - $165.5K / year
Seniority
Mid Level
Job Description
Data Engineer II
GM Financial
• Help evaluate and implement emerging batch and streaming data engineering technologies • Drive innovation and support the adoption of modern data solutions • Collaborate with data scientists, architects, developers, and business partners • Build, deploy, and optimize scalable data pipelines and automation processes that enable advanced analytics and business insights.
Job Requirements
- 2-4 years of hands-on experience with data engineering required
- Bachelor’s degree in related field or equivalent experience required
- Experience with processing large data sets using Hadoop, HDFS, Spark, Kafka, Pulsar, Flume or similar distributed systems.
- Experience with ingesting various source data formats such as JSON, Parquet, CSV, SequenceFile, Cloud Databases, Document Databases like CosmosDB, MQ, Relational Databases such as Oracle.
- Experience with Cloud technologies (such as Azure, AWS, GCP) and native toolsets such as Azure ARM Templates, Hashicorp Terraform, AWS Cloud Formation.
- Understanding of cloud computing technologies, business drivers and emerging computing trends.
- Thorough understanding of Hybrid Cloud Computing: virtualization technologies, Infrastructure as a Service, Platform as a Service and Software as a Service Cloud delivery models and the current competitive landscape.
- Working knowledge of Object Storage technologies to include but not limited to Data Lake Storage Gen2, S3, ADLS etc.
- Working knowledge of Agile development /SAFe, Scrum and Application Lifecycle Management.
- Strong background with source control management systems (GIT or Subversion); Code Quality (Sonar); Artifact Repository Managers (Artifactory), Continuous Integration/ Continuous Deployment (Azure DevOps).
- Experience with NoSQL data stores such as CosmosDB, MongoDB.
- Experience in working with vehicle telemetry and auto insurance data is a plus.
- Creating and maintaining ETL processes.
- Knowledgeable of best practices in information technology governance and privacy compliance.
- Experience with Adobe solutions (ideally Adobe Experience Platform) and REST APIs.
- Troubleshoot complex problems and works across teams to meet commitments.
- Excellent computer skills and proficiency in digital data collection.
- Ability to work in an Agile/Scrum team environment
- Strong interpersonal, verbal, and writing skills.
- Understanding of big data platforms and architectures, data stream processing pipeline/platform, data lake and data lake houses
- SQL experience: querying data and sharing what insights can be derived
- Understanding of cloud solutions such as Microsoft Azure & Amazon AWS cloud architecture & services
- Understanding of GDPR, privacy & security topics. Understanding of data management and governance tools like Atlan, Immuta etc. is a plus.
- Strong in the use of Microsoft Office software, data querying platforms (Databricks is a plus) and statistical programming tools such as Python
Benefits
- 401K matching
- bonding leave for new parents (12 weeks, 100% paid)
- tuition assistance
- training
- GM employee auto discount
- community service pay
- nine company holidays
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Director, Product Management – AI Platform, Data
SurveyMonkeyBuilt for business, loved by users. Get real results with the world’s most popular platform for surveys and forms.
• Lead product strategy for SurveyMonkey’s agentic and conversational surface areas. Influencing how intelligent, automated workflows change what it means to run a survey program. • Own the AI/ML platform layer, enabling AI capabilities across the product, in close partnership with LLM infrastructure and data science teams. • Define and execute a multi-phase strategy to make our data corpus a customer-facing competitive advantage, from cross-survey intelligence to industry benchmarks to predictive signals. • Lead a team of PMs focused on AI platform and data products
Role Description Projeto Temporário - 160 horas de Projeto. Freelancer em RPA (recibo de pagamento para autônomo). Buscamos um(a) Engenheiro(a) de Dados Sênior, com perfil hands-on, para atuar no desenvolvimento e evolução de soluções modernas de dados e Inteligência Artificial, utilizando Databricks como plataforma principal. Procuramos um profissional com forte capacidade técnica para projetar, desenvolver e otimizar pipelines de dados, além de atuar na construção de soluções de IA Generativa, com destaque para o uso do Databricks Genie. - Desenvolver e manter pipelines de dados escaláveis utilizando Databricks, Spark e PySpark. - Projetar soluções de ETL/ELT para ambientes de alta volumetria. - Desenvolver notebooks e workflows em Databricks com foco em performance, qualidade e governança. - Atuar na implementação de arquiteturas modernas de dados utilizando Unity Catalog, Delta Live Tables e demais recursos da plataforma Databricks. - Criar e evoluir soluções baseadas em IA Generativa utilizando Databricks Genie, Mosaic AI e integração com LLMs. - Desenvolver APIs e automações em Python para integração entre plataformas e soluções analíticas. - Atuar na modelagem, catalogação e governança de dados. - Participar da definição de arquitetura, padrões técnicos e boas práticas de Engenharia de Dados. - Trabalhar em conjunto com arquitetos, cientistas de dados e áreas de negócio na construção de soluções analíticas. Qualifications - Experiência sólida como Engenheiro(a) de Dados. - Perfil hands-on, com atuação direta no desenvolvimento de soluções. - Experiência avançada em: - Databricks; - Apache Spark / PySpark; - Python; - SQL; - ETL/ELT; - Data Lake e Data Lakehouse. - Vivência com ambientes Cloud (AWS, Azure, GCP ou OCI). - Experiência com versionamento de código, CI/CD e DevOps. - Conhecimento em modelagem de dados, governança e otimização de performance. Requirements - Experiência prática com Databricks Genie (Genie Spaces), Mosaic AI e desenvolvimento de soluções utilizando IA Generativa e LLMs. - Experiência com Unity Catalog e Delta Live Tables. - Vivência em desenvolvimento de APIs (FastAPI ou similares). - Experiência em Analytics Avançado e BI. - Certificações Databricks. - Ter atuado em projetos do segmento de energia (Utilities/Oil & Gas/Distribuição de Energia/Gás). Competências - Perfil analítico e orientado à solução de problemas. - Facilidade para atuar em ambientes colaborativos. - Proatividade e autonomia. - Boa comunicação e capacidade de interação com times multidisciplinares. - Foco em qualidade, performance e inovação.
Senior Data Engineer, Snowflake
BlueCloudGlobal leader in Data, Analytics and AI with exceptional focus on Innovation, Customer Service and Employee engagement
• Design, develop, and maintain scalable data pipelines and transformation workflows within Snowflake. • Build and optimize ELT/ETL processes using SQL, Python, Snowpark, dbt, and Snowflake-native capabilities. • Modernize existing Python- and stored-procedure-based pipelines using set-based processing, incremental loading, and reusable engineering patterns. • Implement Snowflake-native data-processing solutions using Dynamic Tables, Streams, Tasks, and stored procedures. • Improve platform performance and cost efficiency through query tuning, warehouse right-sizing, clustering strategies, workload isolation, and auto-suspend and auto-resume policies. • Develop reliable orchestration, dependency management, monitoring, error handling, retry, and recovery mechanisms. • Design and maintain dimensional, relational, and domain-oriented data models. • Implement data-quality checks, observability, testing, lineage, and documentation standards. • Integrate data from batch and real-time sources using tools such as Fivetran, Kafka, Airflow, or similar technologies. • Contribute to CI/CD pipelines, infrastructure-as-code practices, and automated deployment processes. • Work closely with architects and client stakeholders to translate business and technical requirements into production-ready solutions. • Participate in code reviews, technical design discussions, estimation, and Agile delivery activities.
Data Architect
BlueCloudGlobal leader in Data, Analytics and AI with exceptional focus on Innovation, Customer Service and Employee engagement
• Lead the architecture, assessment, and modernization of enterprise Snowflake data platforms. • Define current-state and target-state architectures across data ingestion, transformation, storage, orchestration, governance, security, and consumption. • Lead inventories and assessments of Python-, SQL-, and stored-procedure-based pipelines, classifying workloads by business criticality, frequency, volume, runtime, reliability, and cost. • Analyze Snowflake usage telemetry to establish cost and performance baselines across pipelines, warehouses, and workloads. • Identify architectural and engineering anti-patterns, including procedural row-by-row processing, unnecessary full refreshes, missing incremental patterns, redundant transformations, warehouse mis-sizing, and insufficient workload isolation. • Recommend modernization strategies using set-based processing, incremental data patterns, Dynamic Tables, Streams and Tasks, Snowpark, dbt, and appropriate orchestration technologies. • Design scalable data models, governed data domains, semantic layers, and context models that support analytics and AI use cases. • Architect Snowflake Cortex solutions using capabilities such as Cortex Analyst, Cortex Search, and Cortex LLM functions. • Establish architectural standards for security, access control, data governance, lineage, observability, performance, resilience, and business continuity. • Design data lakehouse and real-time data-processing architectures across AWS, Azure, or GCP. • Evaluate and recommend appropriate use of Snowflake Iceberg Tables and other open data architecture patterns. • Build prioritized optimization backlogs and sequenced delivery roadmaps based on business impact, technical risk, cost, performance, and implementation effort. • Provide technical leadership across multiple concurrent projects and guide engineers, analysts, QA specialists, and DevOps teams. • Support project planning, estimation, scope definition, dependency management, risk mitigation, and technical governance. • Lead discovery workshops and interviews with client stakeholders and subject-matter experts. • Present architecture decisions, assessment findings, quantified impact projections, and roadmap recommendations to technical and executive audiences. • Act as a trusted advisor to clients throughout architecture and delivery engagements. • Create and maintain architecture diagrams, decision records, technical standards, and platform documentation.


