Brillio logo
Brillio

Turning technological disruptions into the advantages. Let's create something Brillian(t) together!

Lead Data Engineer

Data EngineerData EngineerFull TimeRemoteSeniorTeam 1,001-5,000H1B SponsorCompany SiteLinkedIn

Location

Washington

Posted

13 hours ago

Salary

$120K - $125K / year

Seniority

Senior

Bachelor Degree5 yrs expEnglishAzureCloudETLSparkSQL

Job Description

Lead Data Engineer

Brillio

• Building state of the art pipelines and data models that are at the heart of the studio decision-making process. • Working on large-scale lakehouse and warehouse analytics systems that process data feeds in real-time and batch processes.

Job Requirements

  • 5+ years’ experience with SQL required.
  • 5+ years’ experience designing and implementing scalable ETL processes including data movement and quality tools.
  • 3+ years’ experience with modern Big Data Analytics using Data Lake, Spark and formats like Parquet.
  • 2+ years’ experience building cloud hosted data systems. Azure highly preferred.
  • Preferred:
  • Building data pipelines in Azure Databricks/Fabric/Spark.
  • Working with data in delta lake format and from Azure Data Explorer/Kusto
  • Applying AI/ML to data engineering use cases (feature engineering, feature stores, model training/serving datasets, and model monitoring data pipelines).
  • Experience preparing and governing datasets for modern AI applications (LLM/RAG, experimentation/A-B testing, and privacy-aware data access).

Related Categories

Related Job Pages

More Data Engineer Jobs

Role Description We're seeking an experienced Data Engineer/ Sr Data Engineer / Lead Data Engineer with expertise in data engineering across major data platforms. The ideal candidate will have a strong background in Python, SQL, ETL, and data modeling, with experience in tools like Teradata, Informatica, Hadoop, Spark, PySpark, ADF, Snowflake, and Big Data. Cloud knowledge (AWS, Azure, or GCP) is a plus. The role requires a willingness to transition and upskill into Databricks & AI/ML projects. - Design, develop, and maintain large-scale data systems - Develop and implement ETL processes using various tools and technologies - Collaborate with cross-functional teams to design and implement data models - Work with big data tools like Hadoop, Spark, PySpark, and Kafka - Develop scalable and efficient data pipelines - Troubleshoot data-related issues and optimize data systems - Transition and upskill into Databricks & AI/ML projects Qualifications - Relevant years of experience in data engineering - Strong proficiency in Python, SQL, ETL, and data modeling - Experience with one or more of the following: - Teradata - Informatica - Hadoop - Spark - PySpark - ADF - Snowflake - Big Data - Scala - Kafka - Cloud knowledge (AWS, Azure, or GCP) is a plus - Willingness to learn and adapt to new technologies, specifically Databricks & AI/ML Requirements - Experience with Databricks - Knowledge of AI/ML concepts and tools - Certification in relevant technologies Benefits - Competitive salary and benefits - Opportunity to work on cutting-edge projects - Collaborative and dynamic work environment - Professional growth and development opportunities - Remote work opportunities & flexible hours

India

Data Engineer

Encora Digital

Encora, a leader in digital engineering, drives innovation by crafting cutting-edge, cloud-first, data-first, and AI-first solutions that redefine industries. S

Data Engineer16 hours ago

Role Description Location: Brazil Job Mode: Full-time Work Mode: Work from home Essential Skills: - Desire to work at high level with stakeholders to devise, understand and communicate clearly requirements and architecture design for data platforms in an efficient manner. - Large Experience with architecture, governance, security, design, business mapping and understanding, performance and tuning for data lake, data warehouse and other data storage systems and transformation. - Experience with talking with customers and stakeholders and extracting business and technical requirements in high and low level. - Strong knowledge of Data frameworks: Hadoop (YARN, HDFS), Hive, Spark, Kafka, Pentaho, Airflow, AWS data tools. - Experience manipulating data using SQL, NoSQL and unstructured data sources, including metadata. - Strong knowledge of ETL frameworks and workflow processes (data ingestion, clean up and preparation). Highly Desirable Skills: - Experience with Python and Java for system administration. - Large experience with data modeling and data design. - System administration experience on Linux platforms. - Financial and/or banking marketing knowledge is a plus. - Experience with AWS platform. Additional Skills: - Machine learning algorithms and workflow. - Pipeline and workflow orchestration (Oozie, Luigi). - Experience with Kubernetes. Company Description Encora is the preferred digital engineering and modernization partner of some of the world’s leading enterprises and digital native companies. With over 9,000 experts in 47+ offices and innovation labs worldwide, Encora’s technology practices include: - Product Engineering & Development - Cloud Services - Quality Engineering - DevSecOps - Data & Analytics - Digital Experience - Cybersecurity - AI & LLM Engineering At Encora, we hire professionals based solely on their skills and qualifications, and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.

Brazil
Oregon Health & Science University logo

Clinical Data Engineer

Oregon Health & Science University

We are Oregon's only public academic health center. In addition to caring for patients, we lead groundbreaking research. We also train the next generation of health care professionals. As Portland's largest employer, we give you opportunities to learn and advance in a system of hospitals and clinics across Oregon and Southwest Washington. All are welcome. OHSU welcomes people of all ages, ethnicities, genders, national origins, religions and sexual orientations. We are striving to build an anti-racist, multicultural institution and encourage people with diverse backgrounds to apply. To request reasonable accommodation, contact askhr@ohsu.edu.

Data Engineer17 hours ago
OtherRemoteTeam 11-50

Role Description The Clinical Data Engineer sits on the clinical data team within the broader Clinical Business Intelligence unit, alongside analysts, engineers, administrators, and architects. This position builds new clinical data warehousing solutions, data transformations, and data integration assets, and supports the changes, enhancements, and maintenance of existing assets in support of OHSU clinical data initiatives. You will work closely with cross-functional teams — clinical and operational stakeholders, data architects, and IT specialists — to develop robust data pipelines, implement data quality controls, and deliver trustworthy data that supports clinical decision-making. Development happens primarily in the Epic Caboodle Console, Microsoft SQL Server tools, and Microsoft Fabric Data Engineering tools, with Azure DevOps for version control, code management, and deployment. Duties may extend to other cloud data engineering tools, such as Apache Airflow, as needed. - Design and develop ETL pipelines that extract, transform, and load clinical data from a variety of sources into structures suitable for analysis. - Implement data quality controls that validate, monitor, and maintain the accuracy, completeness, and consistency of clinical data across the pipeline lifecycle. - Contribute to the growth of the OHSU Caboodle Data Warehouse by designing, developing, testing, and implementing custom clinical data models. - Partner with BI architects, developers, analysts, and customers (practice managers, data scientists, quality analysts) to build and publish data models, ETL processes, Lakehouses, Warehouses, Notebooks, and metadata using the Epic Caboodle Console and third-party ETL tools. - Develop data feeds using SSIS or similar tools, ensuring appropriate security review and transport consistent with information privacy and security requirements, business associate agreements, and data use agreements. - Document warehouse content in the Caboodle Console and the Analytics Marketplace so users can determine what data exists, how it is defined, and how it traces back to Epic Clarity. - Troubleshoot ETL failures, data anomalies, and warehouse issues surfaced by automated monitoring, other developers, and end users; resolve or escalate through established processes and communicate status to affected groups. - Recommend improvements to ETL processes, tool sets, data models, and monitoring techniques that increase reliability and efficiency. - Deploy warehouse content using approved Azure DevOps systems and processes, and follow approved SDLC practices throughout. - Manage assigned projects by building timelines, identifying risks and milestones, and reporting status. - Respond to and track issues in Jira Service Desk, gathering information from customers and triaging to resolution. Qualifications - Bachelor’s degree in computer science, a related field, or a clinical field and six years of work-related experience in the information technology field or a combination of clinical or operational healthcare environments; - OR Associate’s degree in computer science, a related field, or a clinical field and seven years of work-related experience in the information technology field or a combination of clinical or operational healthcare environments; - OR Eight years work related experience in the information technology field or a combination of clinical or operational healthcare environments; - OR Equivalent combination of education and experience where one year of experience will be substituted for an Associate’s degree and two years of experience will be substituted for a Bachelor’s degree. Requirements - Minimum of two (2) years of experience as an Application Engineer or Developer (or equivalent classification) developing data warehouse objects and data integration ETL solutions. - Minimum of three (3) years SQL Server Experience, including SSIS and T-SQL, coding, performance tuning, and system optimization. - Minimum of five (5) years with Microsoft SQL Server T-SQL. - Minimum of two (2) years of experience in a medallion architecture data warehouse environment. - One year of experience with Microsoft Fabric using OneLake and Data Engineering tools. - One year of experience with Python or PySpark. Skills and Abilities - Knowledge of data warehousing architecture and dimensional modeling concepts. - Knowledge of data validation and testing methodologies for ETL processes. - Familiarity with data governance and cataloging practices. - Proven communication, analytical, and problem-solving skills. - Ability to manage competing priorities and communicate progress on an ongoing basis with excellent attention to detail. - Ability to accurately document system technical artifacts at a level of detail sufficient for ongoing production support. Certifications - Epic Clarity Data Model Certifications and Epic Caboodle Developer Certification within 6 months of hire. - Microsoft DP-700 Fabric Data Engineer certification within 9 months of hire. Preferred Qualifications - Experience with the Epic Clarity and Caboodle data models. - Experience planning and managing small projects. - Experience with HIPAA and PHI compliance. - Microsoft DP-700 Fabric Data Engineer Certification. Benefits - Healthcare for full-time employees covered 100% and 88% for dependents. - $50K of term life insurance provided at no cost to the employee. - Two separate above market pension plans to choose from. - Vacation - up to 200 hours per year dependent on length of service. - Sick Leave - up to 96 hours per year. - 9 paid holidays per year. - Substantial Tri-Met and C-Tran discounts. - Employee Assistance Program. - Childcare service discounts. - Tuition reimbursement. - Employee discounts to local and national businesses. Why apply to OHSU? We are Oregon's only public academic health center. In addition to caring for patients, we lead groundbreaking research. We also train the next generation of health care professionals. As Portland's largest employer, we give you opportunities to learn and advance in a system of hospitals and clinics across Oregon and Southwest Washington. All are welcome. OHSU welcomes people of all ages, ethnicities, genders, national origins, religions and sexual orientations. We are striving to build an anti-racist, multicultural institution and encourage people with diverse backgrounds to apply. To request reasonable accommodation, contact askhr@ohsu.edu.

United States
$107.4K - $162.6K / year
Full TimeRemoteTeam 201-500

Role Description The Senior Data Engineer designs, builds, and maintains the data architecture and pipelines that support SBA OIG's Technology Solutions Division (TSD) in its loan fraud detection and investigative mission. The role works within SBA's Microsoft Azure cloud environment to migrate, transform, and structure data assets so that TSD's Data Analytics team can perform agile inquiries and systematic machine learning in support of audits and investigations. - Provide highly skilled and authoritative expertise on data engineering methods and best practices, including code-first development approaches and modern pipeline design patterns. - Design, implement, and maintain an efficient, secure, stable, and flexible data architecture, with all assets managed via source control. - Design, implement, and maintain ELT/ETL pipelines for processing source data in Azure Synapse and Azure Machine Learning. - Review, maintain, and improve existing architecture and pipelines, including periodic audits addressing bottlenecks, deprecated dependencies, and architecture drift. - Establish quality controls for pipeline maintenance, and introduce error handling, logging mechanisms, and validation checks. - Incorporate source control for all pipelines and data analytics codebases to enable iterative code development while maintaining data architecture stability. - Optimize the ingestion, processing, and storage of a wide variety of datasets and data types, including modern columnar formats such as Parquet. - Develop self-service capabilities for SBA OIG analysts to query and export data for investigations and audits. - Coordinate with the Senior Data Scientists to ensure the architecture supports machine learning algorithms and data pipelines in Azure Machine Learning. - Develop standard operating protocols governing the authoring, development, validation, publishing, execution, and monitoring of all data pipelines and assets in the Azure environment. - Provide detailed documentation of the data architecture, including data dictionaries, entity-relationship diagrams, and pipeline process maps. - Maintain and expand the environment with additional datasets and services upon request, following a defined intake and testing process prior to production deployment. - Stay current with emerging AI tools relevant to data engineering and contribute to exploratory efforts evaluating automation and large language model-assisted capabilities. Qualifications - Five (5) years of hands-on experience maintaining SQL databases and conducting advanced operations in SQL and T-SQL. - Five (5) years of hands-on experience designing, implementing, and maintaining ELT/ETL processes in cloud-based data analytics environments. - Three (3) years of hands-on experience working in Azure Synapse and Azure Machine Learning within the modern data stack; DP-203 or an equivalent certification is preferred. - Three (3) years of hands-on experience manipulating data in Python, with required proficiency in Pandas; PySpark or Polars experience is preferred, along with experience developing reusable, modular code. - Preferred: implementing pipelines and infrastructure using code-first approaches, including Python SDK, CLI, REST APIs, or infrastructure-as-code tooling. - Preferred: implementing source control and CI/CD workflows. - Preferred: demonstrated familiarity with AI coding assistants and large language model integration patterns. Requirements - A Bachelor's degree in data engineering, computer science, data science, machine learning, mathematics, or a related field satisfies the education standard set forth in PWS Section 5.2.1. - In the absence of a bachelor's degree, five (5) years of applied work experience in data engineering, computer science, data science, machine learning, mathematics, or a related field will satisfy this requirement. - Microsoft Certified: Azure Data Engineer Associate, or an equivalent Azure data engineering certification, is preferred consistent with the Azure Synapse and Azure Machine Learning experience. - Certifications supporting source control, CI/CD, or infrastructure-as-code practices are preferred but not required. Benefits - Health, dental, and vision insurance - 401(k) retirement plan - Paid time off (PTO) and holidays - Group Term Life and Accidental Death and Dismemberment Insurance - Voluntary Term Life Insurance - Short and Long-term disability insurance

United States