Two Six Technologies logo
Two Six Technologies

Making a positive impact across the globe.

Data Collection Engineer

Data EngineerData EngineerFull TimeRemoteLeadTeam 501-1,000H1B No SponsorCompany SiteLinkedIn

Location

United States

Posted

12 hours ago

Salary

$160K - $200K / year

Seniority

Lead

Bachelor Degree7 yrs expEnglishElasticSearchKubernetesPython

Job Description

Data Collection Engineer

Two Six Technologies

• Design and deploy high-performance, distributed web scrapers using Python and Scrapy • Utilize Browser Scripting tools to navigate, interact with, and extract data from websites • Deploy, scale, and manage scraping workloads on Kubernetes • Define strict JSON Schemas and leverage Pydantic for data validation • Build and optimize search and storage pipelines using Elasticsearch • Architect robust pipeline workflows to manage the end-to-end data lifecycle • Manage complex proxy rotation and session handling to maintain high success rates

Job Requirements

  • 7+ years of professional software engineering experience
  • Excellent reverse-engineering skills
  • Expert-level proficiency in Python
  • Deep experience with Scrapy
  • Proven experience with browser automation tools
  • Mastery of JSON and data validation using Pydantic
  • Hands-on experience indexing, querying, and optimizing Elasticsearch clusters
  • Strong proficiency in managing and scaling applications within Kubernetes environments
  • Experience building structured pipeline workflows
  • Bachelor’s degree in Computer Science or Engineering

Benefits

  • medical, dental, and vision insurance
  • life and disability insurance
  • retirement benefits
  • paid leave
  • tuition assistance
  • professional development

Related Categories

Related Job Pages

More Data Engineer Jobs

Dadoteca logo

Senior Data Engineer | Support

Dadoteca

Transformando dados em conhecimento

Data Engineer13 hours ago
Full TimeRemoteTeam 51-200H1B No Sponsor

Role Description Estamos em busca de uma pessoa Engenheira de Dados Sênior para atuar em um sustetanção de projetos de dados. Procuramos um(a) profissional com forte conhecimento técnico, perfil proativo, autonomia e capacidade analítica para atuar na evolução de soluções existentes e na melhoria contínua dos processos. Mais do que executar demandas, buscamos alguém que questione, proponha melhorias e contribua ativamente para sustentação. - Atuar na sustentação e evolução de pipelines e soluções de dados; - Modernizar processos e fluxos atualmente suportados por tecnologias legadas; - Desenvolver e otimizar rotinas de processamento e transformação de dados; - Modelar estruturas de dados utilizando modelagem dimensional; - Identificar oportunidades de melhoria de performance, qualidade e governança dos dados; - Trabalhar de forma colaborativa com times multidisciplinares na definição e implementação de soluções técnicas; - Garantir a estabilidade, confiabilidade e eficiência dos ambientes de dados. Qualifications - Experiência sólida como Engenheiro(a) de Dados Sênior; - Domínio de Databricks; - Experiência avançada com Python; - Conhecimento em Modelagem Dimensional; - Experiência FORTE com bancos de dados, especialmente MongoDB; - Forte capacidade analítica, autonomia e perfil voltado para resolução de problemas. Requirements - Experiência com banco de dados Oracle legado; - Vivência em projetos de modernização de ambientes legados; - Experiência em sustentação e melhoria contínua de plataformas de dados. Benefits - Proativa e com senso de dono; - Dinâmica e orientada à solução de problemas; - Comunicativa e colaborativa; - Capaz de trabalhar com autonomia e tomada de decisão; - Comprometida com qualidade técnica e melhoria contínua.

Brazil

Senior Data Engineer

Pie Insurance - Contracts

Pie Insurance is a technology-driven small business insurance company focused on making commercial insurance simpler, smarter, and more affordable. Our Data Engineering team sits at the intersection of technology and business operations, and this role is a critical part of how we scale. This is a contract role. The position is fully remote and based in Brazil. Candidates must be authorized to work as contractors in Brazil.

Data Engineer14 hours ago

Role Description This is a hands-on engineering role — you'll be writing production Python and SQL, building Airflow DAGs, and contributing to our Data Vault 2.0 warehouse alongside a team of staff engineers. - Develop complex and efficient data pipelines to transform raw data sources into reliable, well-tested components of our data models. - Design, build, and maintain data pipelines that deliver accurate, trusted data with the freshness our stakeholders depend on. - Make data modeling decisions within our Data Vault 2.0 warehouse that balance raw fidelity in the vault with the consumption patterns of downstream marts and analytics. - Administer and optimize Snowflake across warehouse sizing, query performance, access controls, RBAC and user/role management, and ongoing cost tuning. - Build and maintain resilient Airflow DAGs and CI/CD pipelines that make deployments predictable, repeatable, and safe to roll back. - Implement automated testing (unit, integration, and data quality) so issues are caught in CI before they reach production. - Own production observability through our internal tooling, tuning alerts, responding to incidents, and closing the loop from production issues back into pre-load validation and CI checks. - Leverage AI-powered tools (e.g., Claude Code, Cursor, Snowflake Cortex) as a core part of your development workflow to accelerate code generation, automate documentation, and improve code quality. - Work with stakeholders across Executive, Product, Engineering, and business teams to translate the "why" behind a request into a technical solution that meets the business need. - Lead technical projects end-to-end, scoping with stakeholders, documenting requirements, and explaining technical trade-offs without relying on a Product Manager. - Drive cross-team initiatives that require influence and alignment to achieve a common goal. - Drive best practices for data governance, privacy, and security, including the change management and validation discipline required for SOX-relevant reporting. - Take an active part in the operational responsibilities of running our data infrastructure, with a focus on reliability, cost efficiency, and observability. Qualifications - Minimum 5 years experience as a software engineer or data engineer with a focus on data systems. - Advanced proficiency writing complex SQL and manipulating large structured and semi-structured datasets. - Proficiency in Python for building production-grade data pipelines. - Hands-on Snowflake administration experience, including warehouse management, RBAC and role design, access controls, and cost governance. - Demonstrable experience designing and implementing modern data warehouses, with an understanding of best practices. - Experience modeling data in cloud data warehouses such as Snowflake, Redshift, or BigQuery. Data Vault 2.0 experience strongly preferred, deep dimensional modeling experience acceptable with willingness to learn Data Vault. - Demonstrated experience using testing frameworks to validate data and code in a production environment. - Hands-on experience with data observability tooling. - Proficiency using AI coding assistants (e.g., Claude Code, Cursor, Snowflake Cortex) as a core part of your development workflow. - Track record of leading technical projects end-to-end without a Product Manager — from scoping through documentation through stakeholder communication. - Comfort working in a regulated environment (e.g., SOX-relevant financial reporting). - Willingness to develop deep domain expertise in insurance and the Pie business over time. Requirements - The best data engineers here understand premium, loss, and policy lifecycle as well as they understand Snowflake. Company Description Pie Insurance is a technology-driven small business insurance company focused on making commercial insurance simpler, smarter, and more affordable. Our Data Engineering team sits at the intersection of technology and business operations, and this role is a critical part of how we scale. This is a contract role. The position is fully remote and based in Brazil. Candidates must be authorized to work as contractors in Brazil.

Brazil
Verndale logo

Data Engineer

Verndale

Agence d’expérience numérique

Data Engineer16 hours ago
Full TimeRemoteTeam 201-500H1B Sponsor

• Design, build, and maintain robust ETL/ELT pipelines that move data from diverse source systems into analytics, AI, and machine learning platforms. • Develop and optimize data architectures that support real-time and batch analytics, ML workflows, and personalization. • Partner with data architects, analysts, and client stakeholders to translate business requirements into technical designs. • Ensure data quality, lineage, observability, and security through validation, monitoring, and governance best practices. • Optimize performance and cost-efficiency of pipelines and storage. • Support cloud-first architectures across AWS, Azure, and GCP.

California + 1 moreAll locations: California | Montana
$70K - $85K / year
Mashreq logo

Assistant Vice President – Technology, Data Warehouse

Mashreq

We are Mashreq - Inspiring you to Rise Every Day

Data Engineer17 hours ago
Full TimeRemoteTeam 1,001-5,000Since 1967H1B No Sponsor

• The Enterprise Data Platform stream delivers outcomes to help different parts of the bank gain meaningful insights from their Data Assets. • Our core function spans multiple Data Warehouses, Hadoop, Business Intelligence and Analytics Cluster within Information Management. • The work involves providing delivery and support of platforms spanning over Hadoop, DWH, ETL, Reporting & Analytics application landscape. • You will be handling critical applications that influence bank’s management decision making process, regulatory reporting (which can influence financial/non-financial penalties) and operational reporting.

Pakistan