Senior Data Engineer

Data EngineerData EngineerFull TimeRemoteSeniorTeam 51-200

Location

United States

Posted

50 days ago

Salary

0

Seniority

Senior

Job Description

Senior Data Engineer

Confi

Role Description Você busca uma oportunidade 100% remota, com times distribuídos e uma cultura de colaboração, valorizando a confiança, autonomia e responsabilidade? Buscamos um(a) Engenheiro(a) de Dados Sênior para atuar em uma arquitetura moderna de dados na OCI (Oracle Cloud Infrastructure), com ingestão near–real time e processamento distribuído em PySpark. O papel envolve construir e evoluir pipelines robustos, garantir qualidade das camadas do modelo medalhão (Bronze → Platinum) e disponibilizar dados para consumo via MySQL HeatWave e Metabase. - Desenvolver e otimizar pipelines em PySpark (Bronze/Silver/Gold/Platinum). - Ingestão via EventHub, flat files e MongoDB (API). - Operar arquitetura Lakehouse em OCI (buckets, HeatWave, external tables). - Criar e manter DAGs no Airflow. - Automatizar processos com GitHub + GitHub Actions. - Implementar integrações e exports com Python + Metabase API. - Garantir padrões de qualidade, observabilidade e performance. Qualifications - Domínio avançado em PySpark, Python e SQL. - Domínio avançado em Modelagem de dados. - Domínio de Data Lake / Delta Lake. - Experiência com arquitetura medalhão e pipelines near–real time. - Experiência com Airflow e Git/GitHub Actions. - Experiência em infraestrutura em cloud (AWS, GCP, Azure, OCI...). - Integrações com APIs e MongoDB. Requirements - Conhecimento na Cloud OCI (Dataflow, buckets, HeatWave, external tables). - DataOps. - Governança de dados. - Observabilidade.

Related Categories

Related Job Pages

More Data Engineer Jobs

Airwallex logo

Senior Data Engineer

Airwallex

Airwallex is a financial services company that has developed a “global financial platform for modern businesses.” As an employer, the company strives to cul

Data Engineer50 days ago
Full TimeRemoteTeam 2,200Since 2015

About Airwallex Airwallex is the only unified payments and financial platform for global businesses. Powered by our unique combination of proprietary infrastructure and software, we empower over 200,000 businesses worldwide - including Brex, Rippling, Navan, Qantas, SHEIN and many more - with fully integrated solutions to manage everything from business accounts, payments, spend management and treasury, to embedded finance at a global scale. Proudly founded in Melbourne, we have a team of over 2,000 of the brightest and most innovative people in tech across 26 offices around the globe. Valued at US$8 billion and backed by world-leading investors including T. Rowe Price, Visa, Mastercard, Robinhood Ventures, Sequoia, Salesforce Ventures, DST Global, and Lone Pine Capital, Airwallex is leading the charge in building the global payments and financial platform of the future. If you're ready to do the most ambitious work of your career, join us. Attributes We Value We hire successful builders with founder-like energy who want real impact, accelerated learning, and true ownership. You bring strong role-related expertise and sharp thinking, and you're motivated by our mission and operating principles. You move fast with good judgment, dig deep with curiosity, and make decisions from first principles, balancing speed and rigor. You're humble and collaborative; turn zero-to-one ideas into real products, and you "get stuff done" end-to-end. You use AI to work smarter and solve problems faster. Here, you'll tackle complex, high-visibility problems with exceptional teammates and grow your career as we build the future of global banking. If that sounds like you, let's build what's next. Senior Data Engineer Hiring Location: Singapore What you'll do Part 1. Data Modeling - Design and implement robust and scalable data models that support business intelligence, machine learning, and operational needs. - Possess a deep understanding of data schemas and be able to select appropriate schema designs (e.g., star schema, snowflake, normalized vs denormalized) based on use cases. - Collaborate closely with business teams to translate their data needs into clean, structured, and well-documented models. - Understand and promote the concept of SSOT (Single Source of Truth) throughout the data layers and pipelines. - Maintain data consistency, traceability, and quality across multiple data sources and domains. Part 2. ETL and Data Pipeline Management - Experience building and maintaining both batch and streaming ETL pipelines, with a strong understanding of end-to-end data workflow - from data ingestion to transformation and delivery. - Able to work closely with Data Platform Engineers (DPEs) and Product Managers (PMs) to quickly identify root causes of data issues and provide efficient, scalable solutions. - Bonus if you've worked with data across distributed or multi-datacenter systems, including solving challenges related to data migration, duplication, and consistency. Part 3. Data Governance - Participate in and contribute to data governance strategies, policies, and standards. - Be familiar with any of the six key pillars of traditional data governance (e.g., data quality, data stewardship, metadata management, master data management, data privacy/security, data lifecycle). Part 4. Data + AI - We hope you have a basic understanding of AI and enjoy thinking about how data engineering and AI can work together in practical and creative ways. These four areas are the main focuses of the DE team. Ideally, you should be strong in at least one of them - especially data modeling or data ETL. If you also have experience or skills in the other areas, that would be a big plus. Who you are We're looking for people who meet the minimum qualifications for this role. The preferred qualifications are great to have, but are not mandatory. Minimum qualifications: - Bachelor's degree or higher in Computer Science, Information Systems, Finance, Mathematics, or a related field. - Minimum 5 years of proven experience designing and implementing ETL pipelines, with a strong understanding of the strategies, rules, and processes involved in building scalable and reliable data pipelines. - Experience with streaming data pipelines or real-time data processing technologies, such as Apache Kafka or Apache Flink. - Proficiency in SQL, database management systems (e.g., MySQL, PostgreSQL, Oracle), and data warehousing solutions. - Familiarity with Google Cloud Platform (GCP) and modern data infrastructure tools, including Databricks and Airflow. - Knowledge of data governance practices and regulatory requirements within the financial industry. - Excellent problem-solving skills with strong attention to detail and a commitment to producing high-quality work. - Strong communication and collaboration skills, with the ability to work effectively in a fast-paced, team-oriented environment. - Outstanding verbal communication skills, with the ability to collaborate effectively with globally distributed teams. Preferred qualifications: - Experience with financial industries, payment systems, or fintech platforms. - Knowledge of data governance practices and regulatory requirements in the financial industry. - Experience with scripting languages (e.g., Python, R) for data analysis and automation. - Certification in data management or related technologies At Airwallex, you can make an impact in a rapidly growing, global fintech. We want you to share in our success, which is why you'll be offered a competitive salary plus valuable equity within Airwallex. We also like to ensure we create the best environment for our people by providing collaborative open office space with a fully stocked kitchen. We organise regular team-building events and we give our people the freedom to be creative. Applicant Safety Policy: Fraud and Third-Party Recruiters To protect you from recruitment scams, please be aware that Airwallex will not ask for bank details, sensitive ID numbers (i.e. passport), or any form of payment during the application or interview process. All official communication will come from an @airwallex.com email address. Please apply only through careers.airwallex.com or our official LinkedIn page. Airwallex does not accept unsolicited resumes from search firms/recruiters. Airwallex will not pay any fees to search firms/recruiters if a candidate is submitted by a search firm/recruiter unless an agreement has been entered into with respect to specific open position(s). Search firms/recruiters submitting resumes to Airwallex on an unsolicited basis shall be deemed to accept this condition, regardless of any other provision to the contrary. Equal opportunity Airwallex is proud to be an equal opportunity employer. We value diversity and anyone seeking employment at Airwallex is considered based on merit, qualifications, competence and talent. We don't regard color, religion, race, national origin, sexual orientation, ancestry, citizenship, sex, marital or family status, disability, gender, or any other legally protected status when making our hiring decisions. If you have a disability or special need that requires accommodation, please let us know.

Singapore
Cisco ThousandEyes logo

Lead Site Reliability Engineer, Network Assurance Data Platform - Cisco ThousandEyes

Cisco ThousandEyes

Cisco ThousandEyes is a Digital Experience Assurance platform that empowers organizations to deliver flawless digital experiences across every network – even the ones they don’t own. Powered by AI and an unmatched set of cloud, internet and enterprise network telemetry data, ThousandEyes enables IT teams to proactively detect, diagnose, and remediate issues – before they impact end- user experiences. ThousandEyes is deeply integrated across the entire Cisco technology portfolio and beyond, helping customers deploy at scale while also delivering AI-powered assurance insights within Cisco’s leading Networking, Security, Collaboration, and Observability portfolios.

Data Engineer50 days ago
Full TimeRemoteTeam 1,100Since 2010

Meet the Team Cisco ThousandEyes is a Digital Assurance platform that empowers organizations to deliver flawless digital experiences across every network - even the ones they don't own. Powered by AI and an unmatched set of cloud, internet and enterprise network telemetry data, ThousandEyes enables IT teams to proactively detect, diagnose, and remediate issues - before they impact end- user experiences. ThousandEyes is deeply integrated across the entire Cisco technology portfolio and beyond, helping customers deploy at scale while also delivering AI-powered assurance insights within Cisco's leading Networking, Security, Collaboration, and Observability portfolios. Your Impact As a Site Reliability Engineering (SRE) Technical Leader on the Network Assurance Data Platform (NADP) team, you will play a key role in ensuring the reliability, scalability, and security of our cloud and big data platforms. You will represent the NADP SRE team, working in a dynamic environment, tackling challenges with creativity, providing technical leadership in defining and delivering on the team's technical roadmap. You will collaborate with cross-functional teams, including software development, product management, customers and security teams, to design, influence, build, and maintain SaaS systems operating at multi-region scale. Your work will directly impact the success of our machine learning (ML) and AI initiatives by ensuring the underlying platform infrastructure is robust, efficient, and aligned with operational excellence. Responsibilities: - Design, build, and optimize cloud and data infrastructure to ensure the high availability, reliability, and scalability of big-data and ML/AI systems to meet customer needs, while implementing SRE principles such as monitoring, alerting, error budgets, and fault analysis. - Collaborate closely with cross-functional teams, including customers, development, product management, and security teams, to create secure, scalable solutions that support ML/AI workloads and enhance operational efficiency through automation. - Troubleshoot complex technical problems in production environments, perform root cause analyses, and contribute to continuous improvement efforts through postmortem reviews and proactive performance optimization. - Lead the architectural vision and shape the team's technical strategy and roadmap, balancing immediate needs with long-term goals, driving innovation, and influencing the technical direction. - Serve as a mentor and technical leader, guiding teams and fostering a culture of engineering and operational excellence by sharing your deep knowledge and experience. - Engage with customers and stakeholders to understand use cases and feedback, translating them into actionable insights and effectively influencing stakeholders at all levels. - Utilize your strong programming skills to integrate software and systems engineering, building core data platform capabilities and automation to meet enterprise customer needs and roadmap objectives. - Develop strategic roadmaps, processes, plans, and infrastructure to efficiently deploy new software components at an enterprise scale while enforcing engineering best practices. Minimum Qualifications - Ability to design and implement scalable and well tested solutions, with focus on operational efficiency. - Strong hands-on cloud experience, preferably AWS. - Infrastructure as a Code expertise, especially Terraform and Kubernetes/EKS. - Hands-on expertise scaling and orchestrating production AI/ML infrastructure using the Hadoop ecosystem (Spark, Hive, HDFS, Gobblin), Airflow , and AWS big data services like EMR and SageMaker . - Ability to write high quality code in Python, Go, or equivalent programming languages. Preferred Qualifications - Solid understanding of Unix/Linux systems, the kernel, system libraries, file systems, and client-server protocols. - Have experience with architecting software and infrastructure at scale with a sense of ownership and accountability. - Experience with observability tools including Prometheus (Alertmanager), Grafana, Thanos, CloudWatch, OpenTelemetry, and the ELK stack. - Certifications: CKA (Certified Kubernetes Administrator), CKAD (Certified Kubernetes Application Developer), AWS Certified DevOps Engineer, or equivalent certifications in cloud and security domains. Why Cisco? At Cisco, we're revolutionizing how data and infrastructure connect and protect organizations in the AI era - and beyond. We've been innovating fearlessly for 40 years to create solutions that power how humans and technology work together across the physical and digital worlds. These solutions provide customers with unparalleled security, visibility, and insights across the entire digital footprint. Fueled by the depth and breadth of our technology, we experiment and create meaningful solutions. Add to that our worldwide network of doers and experts, and you'll see that the opportunities to grow and build are limitless. We work as a team, collaborating with empathy to make really big things happen on a global scale. Because our solutions are everywhere, our impact is everywhere. We are Cisco, and our power starts with you.

India
Job Closed
Full TimeRemoteTeam 51-200

Job DetailsJob Location: Anywhere USA, VA 22314Position Type: Full TimeEducation Level: Graduate DegreeSalary Range: $155,000.00 - $185,000.00 SalaryTravel Percentage: NegligibleJob Category: Professional ServicesCurrently, RG is looking to hire a Data Science Lead. The Data Science Lead is an exempt customer-facing business professional that provides a variety of services directly related to the customer’s overarching mission. Additionally, the Data Science Lead will be responsible for: Providing advanced analytics, ML model design, technical leadership. Collecting, analyzing and interpreting large data sets to identify trends, patterns, and provide key business insights. Performs data mining, cleaning, and aggregation processes to prepare data, implement data models, conduct analysis, and develop databases. Develops insights and reports from multiple structured and unstructured data sources using programming, statistical, and analytical techniques and tools. Maintains continuous collaboration with teams to understand the underlying purpose, focus, and objective of each data analysis project to ensure alignment and support. Designs, develops, and implements the most valuable data-driven solutions for the organization. Works autonomously, but may provide a leadership role for the work group through knowledge in the area of specialization. Required Skills The Data Science Lead is an experienced visualization professional who combines technical expertise with design sensibility. This individual translates complex data into clear visual stories that drive operational decision-making. QualificationsRequired Experience To be successful in this role, all candidates must meet the below qualifications to be considered. Master's degree in computer science, information systems, design, or a related field 10+ years of data science experience Ability to travel 5% to Jacksonville, FL and/or Fairfax, VA Current NACI or ability to obtain Bring the WOW Factor In addition to our core requirements that are an absolute must for this role, we want to highlight some of the qualifications that we consider to “Bring the WOW Factor”. Below are some of those “WOW Factors” that we would love to have, but that are not considered hard requirements to do the job. Experience working in the Navy Fleet Readiness Center (FRC) Physical Requirements & Disclaimer Sedentary work: Exerting up to 10 lbs. of force occasionally and/or a negligible amount of force frequently or constantly to lift, carry, push, pull or otherwise move objects, including the human body. Sedentary work involves sitting most of the time. Jobs are sedentary if walking and standing are required, only occasionally and all other sedentary criteria are met. EOE, including disability/vets

Spain
$155K - $185K / year
Peraton Corporation logo

Configuration / Data Management Specialist

Peraton Corporation

Peraton Corporation, a national security company headquartered in Herndon, Virginia, supplies solutions for mission-critical programs and systems. Founded in 2017, Peraton's missio

Data Engineer50 days ago

Responsibilities The Office of Space Weather Observations (SWO) under NESDIS is responsible for advancing space weather observational capabilities to meet NOAA programmatic needs. NOAA’s Space Weather Next (SWX) program maintains and extends space weather observations from various vantage points, selected to most efficiently provide comprehensive knowledge of the Sun and the near-Earth space environment needed to protect our technological infrastructure.  The Space Weather Ground Services (SWGS) is responsible for comprehensive ground services for all SWX projects, ensuring successful implementation and operation of observing assets and ensuring the continuity of space weather measurements made by SWFO-L1 and the GOES-R series satellites.  The SWGS Mission Operations Services (MOS) program must provide a full mission satellite command and control solution to support the L1 Series with two new independently launched observatories. Overview: Peraton seeks a Configuration & Data Management specialist to support a new satellite ground system development program supporting National Oceanic and Atmospheric Administration (NOAA). The candidate will be part of a team that is responsible for maintaining the technical integrity, traceability and control of all program artifacts throughout the full system lifecycle - from system architecture and design through integration, accreditation, delivery, and operational sustainment.  This role requires close collaboration with other program functional elements including Cyber Engineering, Systems Engineering Integration and Test (SEIT), Software Engineering, Networks, Infrastructure, Systems Architects, Operations and program leadership to ensure disciplined configuration control, accurate documentation and compliance with all contractual and program requirements. - Supports creation and delivery of contractor deliverable (CDRLs), completed reports, change requests, and documents under configuration control follow established workflows and processes as described in the SWFO CM procedures. - Support the change control process and identifies and prepares change requests. - Controls changes to product requirements, design, documentation, and implementation via configuration control boards (CCBs) in coordination with the program team to determine the effect on the end product design and function and determines and prepares any necessary documentation. - Uses CM tools (Atlassian Jira and Confluence); and configuration auditing scripts to monitor changes to operating systems, software, configuration setting, GFP, COTS products, applications, and firmware - Performs configuration audits to ensure compliance with the published baseline, addressing Technical Requirements (TRs), Configuration Change Requests (CCRs), and Discrepancy Reports (DRs), documentation, release notes, and delivery media, documenting differences as DRs per program requirements. - Develops document and drawing trees - Performs configuration management and release engineering tasks to ensure new software product operating parameters are documented, comply with standard hardware configurations, and are logistically sustainable - Designs, develops, automates, and maintains productivity tools using programming, databases or scripting languages to improve software modeling and development. - Designs and implements build procedures that are used to support software product development and use - May provide guidance or training to other team members on the use of configuration management tools  **This position is contingent on contract award.** #SWOMOS Qualifications - Minimum of 5 years with BS/BA; Minimum of 3 years with MS/MA - 9 years of relevant experience will be considered in lieu of a bachelor’s degree - Strong understanding of configuration control processes, version management, and technical documentation governance.  - Experience working with the Atlassian tool suite, including Jira, Insight, Confluence, Risk Register, Crucible, etc - Proficiency in NASA-STD-0005 (Configuration Management) and NOAA/NESDIS procedural directives for CM/DM - Ability to manage hardware, software, and firmware configuration items (CIs) from development through deployment and operations - Experience working in Scaled Agile Framework (SAFe) environments or similar Agile methodologies.  - Proficiency with Microsoft Office applications (Word, Excel, PowerPoint) - Experience working within a distributed virtual team environment, with proficiency in remote collaboration tools and practices - Strong interpersonal skills with a willingness to foster strong relationships with coworkers and vendors - Highly organized with strong attention to detail - Outstanding verbal and written communication skills with the ability to dynamically tailor communications effectively to different levels of the organization - Ability to obtain and hold a Public Trust clearance – US Citizenship is required Desired Qualifications: - Experience supporting NOAA, NASA, or national space mission programs, particularly ground systems. - Prior experience with CM / DM processes for new programs or large system deployments. - Active Public Trust clearance. Peraton Overview Peraton is a next-generation national security company that drives missions of consequence spanning the globe and extending to the farthest reaches of the galaxy. As the world’s leading mission capability integrator and transformative enterprise IT provider, we deliver trusted, highly differentiated solutions and technologies to protect our nation and allies. Peraton operates at the critical nexus between traditional and nontraditional threats across all domains: land, sea, space, air, and cyberspace. The company serves as a valued partner to essential government agencies and supports every branch of the U.S. armed forces. Each day, our employees do the can’t be done by solving the most daunting challenges facing our customers. Visit peraton.com to learn how we’re keeping people around the world safe and secure. Target Salary Range $80,000 - $128,000. This represents the typical salary range for this position. Salary is determined by various factors, including but not limited to, the scope and responsibilities of the position, the individual’s experience, education, knowledge, skills, and competencies, as well as geographic location and business and contract considerations. Depending on the position, employees may be eligible for overtime, shift differential, and a discretionary bonus in addition to base pay. EEO EEO: Equal opportunity employer, including disability and protected veterans, or other characteristics protected by law.

United States
$80K - $128K / year