Job Closed

This listing is no longer active.

Carnegie Mellon University logo
Carnegie Mellon University

We learn, we make, we solve, we create, and we give it to the world.

Data Pipeline Engineer – Computer Science

Data EngineerData EngineerPart TimeRemoteJuniorTeam 5,001-10,000Since 1900H1B SponsorCompany SiteLinkedIn

Location

United States

Posted

26 days ago

Salary

0

Seniority

Junior

Bachelor Degree1 yr expEnglishAirflowApacheCloudLinuxPostgreSQLPythonSQL

Job Description

Data Pipeline Engineer – Computer Science

Carnegie Mellon University

• Monitor and maintain the health and efficiency of data pipelines • Troubleshoot and perform root cause analysis for data discrepancies and pipeline issues • Communicate with data providers to understand data discrepancies and manage changes in data delivery • Implement fixes and enhancements to improve data quality and pipeline performance • Collaborate with data scientists and analysts to understand data needs and implement effective data solutions • Develop strategies for data validation and quality assurance • Optimize data flow and collection to improve system efficiency • Document and manage data pipeline architectures, including maintenance and update protocols • Use tools such as SQL, version control and CI/CD, containerization, task schedulers, python frameworks, and cloud services for data pipeline management • Ensure compliance with data governance and security standards

Job Requirements

  • Bachelor’s Degree required
  • Minimum one year of research computing experience required
  • Basic Linux use and administration: system layout, file permissions, shell, utilities (syslog, cron), diagnostic tools (ps, htop, grep, lsof)
  • Experience in Apache Airflow, preferably version 3.0
  • Basic database use, especially in Postgres
  • Rough script programming (Python, bash)
  • Team software development (git/GitHub, Jira, code reviews, agile methodologies)
  • Data analysis: diagnosing and fixing runtime errors and logic bugs; performing basic growth projections to predict future problems; communicating results

Benefits

  • comprehensive medical, prescription, dental, and vision insurance
  • generous retirement savings program with employer contributions
  • tuition benefits
  • ample paid time off and observed holidays
  • life and accidental death and disability insurance
  • free Pittsburgh Regional Transit bus pass
  • access to our Family Concierge Team to help navigate childcare needs
  • fitness center access

Related Categories

Related Job Pages

More Data Engineer Jobs

Microsoft logo

Data Engineer II

Microsoft

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to any characteristic protected by applicable local laws, regulations, and ordinances.

Data Engineer26 days ago
Full TimeRemoteTeam 10,001+H1B Sponsor

Role Description We are looking for a talented Data Engineer II to join our growing and fast-paced Insider Risk Engineering team. In this role, you will work on developing and managing data pipelines, joining and filtering data sets, and building advanced insider risk detections to proactively identify and address potential threats. - Design, build, and optimize data pipelines to ingest, process, and prepare data for use in insider risk detection models. - Join, filter, and integrate diverse data sources to create comprehensive datasets that enable effective and accurate insider risk detections. - Work with large datasets, applying advanced data transformation techniques to ensure data quality and accessibility for risk detection. - Develop, test, and deploy insider risk detection models based on data-driven insights to proactively identify anomalous or risky behavior patterns. - Collaborate with insider risk team members to define and refine detection use cases, ensuring they are accurate, scalable, and aligned with business needs. - Share knowledge and actively contribute ideas in team technical discussions. - Maintain and monitor insider risk engineering systems to ensure reliable operation, security, and compliance with internal engineering standards and policies. - Join on-call rotations, lead incident response, and drive thorough root-cause analysis. - Document data processes, detection workflows, and system configurations to support future development and maintenance. - Use development and coding best practices (e.g., reusable, modular). - Own end-to-end quality for the code you deliver, including testing and DevOps automation. Qualifications - Master's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 1+ year(s) experience in business analytics, data science, software development, data modeling, or data engineering OR Bachelor's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 2+ years experience in business analytics, data science, software development, data modeling, or data engineering OR equivalent experience. - Preferred Qualifications: - Bachelor’s degree in Computer Science, Data Science, Engineering, or a related field, or equivalent work experience. - 4+ years of experience in data engineering, data science, or a hybrid data engineering/data science role. - Proficiency in query languages (e.g., SQL, KQL). - Experience with object-oriented programming languages (e.g., Python, C#, Java, or C++). - Experience with security product usage such as Insider Risk Management or Sentinel. - Demonstrated ability to build and manage data systems in the cloud. - Experience with big data systems and tools, such as PySpark, Databricks, or Azure Synapse. - Familiar with data engineering best practices like layered data architecture, data modeling, and developing reliable and scalable data pipelines. - Strong understanding of engineering and security compliance standards, with experience in regulated environments. - Excellent problem-solving skills and attention to detail, with a proactive approach to identifying and mitigating risks. - Experienced working within agile frameworks such as Scrum and Kanban. Requirements - Candidates must be able to meet Microsoft, customer and/or government security screening requirements are required for this role. - Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter. - Citizenship & Citizenship Verification: This role will require access to information that is controlled for export under export control regulations. - This position requires verification of citizenship due to citizenship-based legal restrictions. Benefits - Data Engineering IC3 - The typical base pay range for this role across the U.S. is USD $102,100 - $202,200 per year. - There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $133,800 - $219,200 per year. - Certain roles may be eligible for benefits and other compensation. - Find additional benefits and pay information here .

United States
$102.1K - $219.2K / year
Job Closed
Distro logo

Senior Data Engineer

Distro

Distro is a marketplace to find, hire, and pay technical talent in over 200 countries. Join now for free.

Data Engineer26 days ago
Full TimeRemoteTeam 1-10Since 2021H1B Sponsor

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python. • Process and integrate data from multiple formats and sources (JSON, CSV, XML). • Build and manage data transformations and orchestration workflows using dbt and orchestration tools such as Airflow, Prefect, or Dagster. • Enforce data governance, quality, and security standards. • Extend, maintain, and optimize the Elastic Hierarchy data framework. • Collaborate with analytics, ML, and product teams to deliver reliable, business-ready datasets. • Support data operations in SaaS and hybrid on-premise environments.

Mexico
$4.2K - $5.2K / month
Job Closed
Volga Partners logo

English Language Data Evaluator

Volga Partners

Smart and Reliable Technology Solutions

Data Engineer26 days ago
Full TimeRemoteTeam 1,001-5,000Since 2020H1B Sponsor

Role Description Candidates MUST BE located in USA due to client's technical requirements. - Remote position - Must-Haves: Full fluency in English language. - Full Time Remote position - Hours: 9am to 3pm PST Mon-Fri - MUST BE AVAILABLE PACIFIC TIME ZONE - This position is expected to end towards the end of June, with potential extension as per client's needs Responsibilities - Data Evaluation & Annotation: Label and categorize text and images, ensuring accuracy and quality - Platform Management: Work within the client’s universal language platform to manage and evaluate language-based data - Trend Analysis: Track and assess data, identifying insights to enhance project quality - Quality Assurance: Ensure accuracy by proofreading and maintaining high-quality outputs - Process Improvement: Provide insights and recommendations to improve data relevance and performance - Adaptability & Collaboration: Work in a fast-paced environment with shifting priorities while maintaining communication with your team Qualifications - Fluency in English – Must have full professional proficiency in both written and spoken communication - Strong analytical skills – Ability to evaluate data logically and provide sound reasoning - Attention to detail – A sharp eye for spotting inconsistencies and ensuring high data accuracy - Time management skills – Ability to manage multiple tasks efficiently - Critical thinking & problem-solving – Ability to make data-driven decisions - Tech-savvy mindset – Comfortable navigating platforms and tools - Adaptability & teamwork – Willingness to collaborate and adjust to changing project needs Requirements - Work Authorization: Must have valid authorization to work in the U.S. - Background Check: Required for this role - Equipment: Must have a personal laptop or computer Benefits - 401(k) - Paid time off - Fully Remote Company Description Join Volga Partners, a dynamic U.S.-based company at the forefront of Artificial Intelligence (AI) and Machine Learning (ML), proudly partnering with leading technology giants and global multinational corporations. Why Join Us? - Fully remote role – Work from the comfort of your home - Entry to mid-level opportunity – Gain hands-on experience in data evaluation and AI - Flexible training hours – Get the support you need to succeed - This current opening is for full time. - Work with cutting-edge technology – Contribute to improving AI and language models

United States
$15 - $18 / hour
Enroute logo

Data Engineer – Database Migration

Enroute

We deliver IT services and solutions provided by a team of passionate problem solving individuals highly skilled.

Data Engineer26 days ago
Full TimeRemoteTeam 51-200H1B Sponsor

• Analyze existing Oracle schemas, tables, views, stored procedures, functions, indexes, and related database objects • Support the migration of database structures and data from Oracle to Azure SQL • Identify compatibility gaps between Oracle and Azure SQL and propose remediation approaches • Convert or refactor Oracle SQL, PL/SQL, stored procedures, functions, and views into Azure SQL-compatible equivalents • Support source-to-target mapping, data profiling, and migration analysis • Perform data validation, reconciliation, and quality checks between Oracle and Azure SQL • Optimize migrated database objects for performance, scalability, and maintainability • Collaborate with ETL / ADF engineers to ensure pipelines align with the target data model • Document migration findings, conversion decisions, validation results, and technical recommendations • Support testing, defect resolution, production migration, and post-migration stabilization activities • Troubleshoot performance, compatibility, and data quality issues throughout the migration lifecycle

Mexico
Job Closed