Job Closed

This listing is no longer active.

Pavago

Pavago specializes in connecting businesses with top-tier offshore talent in operations, sales, and marketing, offering a comprehensive recruitment solution designed to reduce cost

Data Engineer

Location

Brazil

Posted

96 days ago

Salary

0

Seniority

Mid Level

Job Description

Data Engineer

Pavago

Job Title: Data Engineer Position Type: Full-Time, Remote Working Hours: U.S. client business hours (with flexibility for pipeline monitoring and data refresh cycles) About the Role: Our client is seeking a Data Engineer to design, build, and maintain reliable data pipelines and infrastructure that deliver clean, accessible, and actionable data. This role requires strong software engineering fundamentals, experience with modern data stacks, and an eye for quality and scalability. The Data Engineer ensures data flows seamlessly from source systems to warehouses and BI tools, powering decision-making across the business. Responsibilities: Pipeline Development: - Build and maintain ETL/ELT pipelines using Python, SQL, or Scala. - Orchestrate workflows with Airflow, Prefect, Dagster, or Luigi. - Ingest structured and unstructured data from APIs, SaaS platforms, relational databases, and streaming sources. Data Warehousing: - Manage data warehouses (Snowflake, BigQuery, Redshift). - Design schemas (star/snowflake) optimized for analytics. - Implement partitioning, clustering, and query performance tuning. Data Quality & Governance: - Implement validation checks, anomaly detection, and logging for data integrity. - Enforce naming conventions, lineage tracking, and documentation (dbt, Great Expectations). - Maintain compliance with GDPR, HIPAA, or industry-specific regulations. Streaming & Real-Time Data: - Develop and monitor streaming pipelines with Kafka, Kinesis, or Pub/Sub. - Ensure low-latency ingestion for time-sensitive use cases. Collaboration: - Partner with analysts and data scientists to provide curated, reliable datasets. - Support BI teams in building dashboards (Tableau, Looker, Power BI). - Document data models and pipelines for knowledge transfer. Infrastructure & DevOps: - Containerize data services with Docker and orchestrate in Kubernetes. - Automate deployments via CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI). - Manage cloud infrastructure using Terraform or CloudFormation. What Makes You a Perfect Fit: - Passion for clean, reliable, and scalable data. - Strong problem-solving skills with debugging mindset. - Balance of software engineering rigor and data intuition. - Collaborative communicator who thrives in cross-functional environments. Required Experience & Skills (Minimum): - 3+ years in data engineering or back-end development. - Strong Python and SQL skills. - Experience with at least one major data warehouse (Snowflake, Redshift, BigQuery). - Familiarity with pipeline orchestration tools (Airflow, Prefect). Ideal Experience & Skills: - Experience with dbt for transformations and data modeling. - Streaming data experience (Kafka, Kinesis, Pub/Sub). - Cloud-native data platforms (AWS Glue, GCP Dataflow, Azure Data Factory). - Background in regulated industries (healthcare, finance) with strict compliance. What Does a Typical Day Look Like? A Data Engineer’s day revolves around keeping pipelines running, improving reliability, and enabling teams with high-quality data. You will: - Check pipeline health in Airflow/Prefect and resolve any failed jobs. - Ingest new data sources, writing connectors for APIs or SaaS platforms. - Optimize SQL queries and warehouse performance to reduce costs and latency. - Collaborate with analysts/data scientists to deliver clean datasets for dashboards and models. - Implement validation checks to prevent downstream reporting issues. - Document and monitor pipelines so they’re reproducible, scalable, and audit-ready. In essence: you ensure the business has accurate, timely, and trustworthy data powering every decision. Key Metrics for Success (KPIs): - Pipeline uptime ≥ 99%. - Data freshness within agreed SLAs (hourly, daily, weekly). - Zero critical data quality errors reaching BI/analytics. - Cost-optimized queries and warehouse performance. - Positive feedback from data consumers (analysts, scientists, leadership). Interview Process: - Initial Phone Screen - Video Interview with Pavago Recruiter - Technical Task (e.g., build a small ETL pipeline or optimize a SQL query) - Client Interview with Engineering/Data Team - Offer & Background Verification

Related Categories

Related Job Pages

More Data Engineer Jobs

Full TimeRemoteTeam 11-50Since 2021H1B No Sponsor

• Q Development: Write efficient and concise code in the Q language for manipulating in-memory and disk (splayed/partitioned) tables; • Fluent English; • Feed Architecture: Develop and optimize Ticker Plants, Chained Tickers, and Real-time Databases (RDB); • Historical Data Management: Maintain and optimize the Historical Database (HDB), ensuring that queries on terabytes of data return in milliseconds; • Performance Optimization: Identify I/O and CPU bottlenecks, applying vector processing techniques to avoid slow loops and iterators; • Integration: Create connectivity interfaces (APIs) between kdb+ and other languages such as Python (PyQ/EmbedPy), Java, or C++.

Brazil
Data Engineer97 days ago
Full TimeRemoteTeam 11-50Since 2021H1B No Sponsor

• Q development: Write efficient, concise code in the Q language • Feed architecture: Develop and optimize ticker plants • Historical data management: Maintain and optimize the historical database • Performance optimization: Identify I/O and CPU bottlenecks • Integration: Build connectivity interfaces (APIs)

Brazil
Full TimeRemoteTeam 201-500Since 2004H1B Sponsor

Full‑Stack data engineer We are seeking a highly self‑sufficient, motivated engineer based with strong full‑stack data engineering skills to join our innovative, dynamic team. This is a remote/offshore role that requires autonomy, excellent communication, and the ability to deliver high‑quality work with limited supervision while collaborating with a predominantly US‑based team. You will build reliable, scalable data products and user experiences that power AI/ML modeling, agentic workflows, and reporting—working end‑to‑end from data ingestion and transformation through to UI. Our Python‑based data platform is undergoing a major evolution toward a modern, cloud‑native ELT architecture. We are standardizing on Snowflake as our central data platform and dbt as our core transformation framework, implementing scalable, maintainable ELT practices that simplify ingestion, modeling, and deployment. This role will be pivotal in independently designing and building robust data pipelines and semantic layers that directly power our AI and machine learning initiatives—delivering clean, reliable, and well‑modeled data assets to our data science team for feature engineering, model training, and production inference. You will collaborate closely (primarily via remote channels) with data scientists and ML engineers to ensure our data ecosystem is optimized for experimentation speed, model performance, and seamless integration into downstream products and services. Key Responsibilities - Remote collaboration & communication: Operate effectively as an offshore member of a distributed team, proactively communicating status, risks, and blockers across time zones and coordinating overlap with US working hours as needed. - Full‑stack data engineering: Build across the entire stack, including data ingestion/acquisition and transformation, APIs, front‑end components, and automated test suites, delivering production‑grade solutions with minimal hand‑holding. - Autonomous delivery & ownership: Take end‑to‑end ownership of features and projects—clarifying requirements, breaking work into milestones, estimating timelines, and delivering high‑quality, well‑documented solutions. - Specification and design: Translate short‑ and long‑term business requirements, architectural considerations, and competing timelines into clear, actionable technical specifications and design documents. - Code quality: Write clean, maintainable, efficient code that adheres to evolving standards and quality processes, including unit tests and isolated integration tests in containerized environments. - Continuous improvement: Contribute to agile practices and provide input on technical strategy, architectural decisions, and process improvements, continuously suggesting better tools, patterns, and automation. Required Skills & Experience - Professional experience: 5+ years in software engineering, with a full‑stack background building complex, scalable data‑engineering pipelines using data warehouse technology, SQL with dbt, Python, AWS with Terraform, and modern UI technologies. - Modern data engineering: Strong experience with medallion data architecture patterns using data warehouse technologies (e.g., Snowflake), data transformation tooling (e.g., dbt), BI tooling, and NoSQL data marts (e.g., Elasticsearch/OpenSearch). - Testing and QA: Solid understanding of unit testing, CI/CD automation, and quality assurance processes for both data pipeline testing and operational data quality tests. - Remote work & autonomy: Proven track record working in a remote or distributed environment, demonstrating self‑motivation, reliable execution, and the ability to make sound technical decisions independently. - Agile methodology: Working knowledge of Agile development practices and workflows (e.g., sprint planning, stand‑ups, retrospectives) in a distributed team setting. - Education: Bachelor’s or Master’s degree in Computer Science, Statistics, Informatics, Information Systems, or a related quantitative field. Preferred Skills & Experience - Machine learning and AI: Hands‑on experience with large language models (LLMs) and agentic frameworks/workflows. - Search and analytics: Familiarity with the ELK stack (Elasticsearch, Logstash, Kibana) for search and analytics solutions. - Cloud expertise: Experience with AWS cloud services; familiarity with SageMaker; and CI/CD tooling such as GitHub Actions or Jenkins. - Front‑end expertise: Experience building user interfaces with Angular or a modern UI stack. - Financial domain knowledge: Broad understanding of equities, fixed income, derivatives, futures, FX, and other financial instruments.

India
Microsoft logo

Software Engineer II - Azure Data Lake Storage Gen 2

Microsoft

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to any characteristic protected by applicable local laws, regulations, and ordinances.

Data Engineer97 days ago
Full TimeRemoteTeam 10,001+H1B Sponsor

Overview Are you excited about the possibility of working on the largest Cloud Storage service in the world? Azure Storage team is chartered with building, managing and running the Persistent Cloud Storage for Microsoft Azure Cloud. Azure Storage consists of a Storage Platform which currently powers five different storage services: Blob Storage, Table Storage (NoSQL Key Value store), Queue Storage, Data Lake Storage, File Storage and Disk Storage. We are one of the foundational services in the Azure Cloud. The service hosts data from some of the largest companies in the world plus supports all of Microsoft’s largest online businesses including Xbox, Teams and Bing and has scaled to store exabytes of customer data with the roadmap looking even better in terms of growth. We are looking for a Software Engineer II - Azure Storage with strong coding and problem solving skills and a solid foundation in designing and building reliable software. This position will contribute to building and improving key components of Azure Storage, delivering customer and partner facing features and reliability improvements with high quality. The candidate will work on modern server technologies and highly distributed systems, with a focus on reliability, scalability, performance, consistency and debuggability. If you are ready to take on the challenge of solving impactful problems, designing and implementing well scoped components, and debugging live systems in partnership with experienced engineers, then this is the position for you. Strong collaboration skills, the ability to communicate clearly with teammates and partners, and a strong commitment to our customers will be required. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond. Responsibilities - Collaborates with appropriate stakeholders to determine user requirements for a scenario, asks clarifying questions, and helps translate requirements into engineering tasks. - Contributes to the development of design documents for a product, application, service, or platform by proposing options, identifying tradeoffs, and reviewing designs with senior engineers. - Creates, implements, optimizes, debugs, refactors, and reuses code to improve correctness, performance, and maintainability. - Writes and maintains unit, integration, and scenario tests, and uses telemetry and diagnostics to validate health and quality in production. - Delivers well scoped work items end to end, including implementation, code reviews, testing, safe rollout, and follow up based on signals and feedback. - Partners with appropriate stakeholders (for example, project managers and partner teams) to execute on project plans, release plans, and work items for owned areas. - Participates in on call rotation and live site operations by following established playbooks, escalating appropriately, and contributing to mitigation and post incident follow up. Qualifications Required Qualifications: - Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python - OR equivalent experience. Other Requirements: - Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: - Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter. Preferred Qualifications: - Master's Degree in Computer Science or related technical field AND 3+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python - OR Bachelor's Degree in Computer Science or related technical field AND 5+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python - OR equivalent experience. #azurecorejobs This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled. Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.

Australia
Job Closed