Responsible recruiting!
Data Engineer
Location
United States
Posted
3 days ago
Salary
$75 - $105 / hour
Seniority
Mid Level
Job Description
Data Engineer
OmegaHires
Role Description We are seeking skilled Data Engineers to join our dynamic team in a remote capacity. In this role, you will be responsible for designing, developing, and maintaining robust data pipelines and architectures that support various analytics and business intelligence initiatives. You will work with cutting-edge technologies such as Databricks, Snowflake, and cloud platforms like AWS and Azure to ensure efficient data processing and integration. This contract position offers the opportunity to collaborate with cross-functional teams and contribute to the overall data strategy of the organization. Qualifications - 5+ years of experience in data engineering or related field - Proficiency in Python and SQL - Experience with ETL/ELT processes - Familiarity with cloud platforms such as AWS or Azure - Experience with data pipeline development using Databricks or Snowflake - Ability to work in a remote environment - US citizenship or work authorization Requirements - Experience with data visualization tools like Power BI - Knowledge of data governance and quality frameworks - Familiarity with Kubernetes and container orchestration Benefits - Indicative rate: $75–$105/hr
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Desenvolver, manter e otimizar pipelines de dados escaláveis em ambiente cloud • Trabalhar com processamento distribuído de dados utilizando Apache Beam • Orquestrar workflows e pipelines de dados utilizando Apache Airflow • Desenvolver soluções em Python seguindo boas práticas de engenharia de software • Atuar em ambientes Google Cloud Platform (GCP) • Criar e manter pipelines de deploy utilizando práticas de CI/CD • Desenvolver e gerenciar configurações e automações utilizando YAML/YML • Utilizar Docker para conteinerização de aplicações e serviços • Desenvolver, testar e documentar modelos de transformação de dados com dbt • Colaborar com times de engenharia, analytics e negócio para garantir qualidade, confiabilidade e performance das soluções
• Colaborar com as equipes de engenharia de dados para entender os requisitos e fornecer soluções eficientes; • Atuar no desenvolvimento e sustentação das tecnologias Azure, Databricks (PysPark), Python, Data Lake e SQL; • Utilizar Python ElementTree para manipulação eficiente de dados XML e integração de dados heterogêneos; • Desenvolver scripts em Python utilizando Pandas para manipulação e análise de dados estruturados; • Atuar com a disponibilidade, estabilidade e evolução contínua dos sistemas do cliente assegurando alta performance; • Atuar com a parte de segurança e conformidade dos requisitos de negócio, por meio de uma gestão proativa, automação de processos e melhoria contínua dos serviços de sustentação; • Trabalhar com a qualidade e integridade dos dados, implementando práticas de teste e monitoramento; • Manter documentação técnica abrangente para os processos e soluções implementadas; • Apoio com ações corretivas, suportando as entregas em produção, acompanhando CHGs;
• Collaborate with data engineering teams to understand requirements and deliver efficient solutions; • Develop and maintain Azure, Databricks (PySpark), Python, Data Lake, and SQL technologies; • Use Python ElementTree for efficient XML data handling and integration of heterogeneous data; • Develop Python scripts using Pandas for manipulation and analysis of structured data; • Ensure availability, stability and continuous evolution of client systems, ensuring high performance; • Work on security and compliance for business requirements through proactive management, process automation, and continuous improvement of support services; • Maintain data quality and integrity by implementing testing and monitoring practices; • Keep comprehensive technical documentation for processes and solutions implemented; • Support corrective actions, assist with production deliveries and monitor change requests (CHGs);
• Perform the complete migration of the current environment, today hosted on Databricks on Azure, to AWS, including creating a new data model and restructuring legacy pipelines and routines; • Define and evolve the Corporate Data Platform architecture (Lakehouse); • Ensure adherence to the target model based on AWS + Databricks; • Define architecture standards, frameworks and best practices; • Drive the definition of the migration strategy (waves, prioritization, dependencies); • Migration and Modernization: Lead the modernization of the legacy Data Warehouse (Azure/DataStage → AWS/Databricks); • Define migration approaches: Incremental vs Big Bang; • Ensure operational continuity during the transition; • Governance & Security: Define and implement standards for: Data governance/Access control/Data quality and lineage; • Ensure compliance with corporate policies and LGPD (Brazilian data protection law); • DataOps & Standardization: Structure standardized and reusable pipelines; • Implement best practices for CI/CD for data; • Reduce dependence on manual processes and low standardization; • Integration and Ecosystem: Design integrations with multiple sources and on-premises systems;


