CDW Corporation is a leading multi-brand provider of information technology solutions to business, government, education and healthcare customers in the United States, the United Kingdom and Canada. A Fortune 500 company and member of the S&P 500 Index, CDW helps its customers to navigate an increasingly complex IT market and maximize return on their technology investments. For more information about CDW, please visit www.CDW.com. Our broad array of products and services range from hardware and software to integrated IT solutions such as security, cloud, hybrid infrastructure and digital experience.
Senior Data Engineer
Location
Illinois
Posted
20 days ago
Salary
0
Seniority
Senior
Job Description
Senior Data Engineer
CDW
• Build and maintain scalable data pipelines, ETL and ELT processes, and data models within the Databricks platform. • Design, develop, and deploy data and AI solutions using Databricks, Spark, Delta Lake, and related technologies. • Develop batch and streaming pipelines using tools such as Databricks Workflows and Azure Data Factory. • Design logical data flow diagrams and normalized schemas, implementing Lakehouse patterns such as the Medallion Architecture (Bronze, Silver, Gold layers). • Ensure data quality, integrity, security, and governance throughout the data lifecycle, including use of Unity Catalog. • Optimize Spark jobs and data transformations through effective partitioning, caching, and join strategies. • Monitor pipeline execution, identify failures, and troubleshoot complex data processing issues. • Collaborate with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver solutions. • Support documentation of data processes, standards, and data flows.
Job Requirements
- 5 Years of experience designing, developing, and deploying data solutions on the Databricks platform.
- Proficiency in Python, including PySpark, and SQL.
- Hands-on experience with Spark, Delta Lake, and Lakehouse architectures.
- Experience implementing data quality, governance, and security practices across data pipelines.
- Familiarity with machine learning concepts, tools, and libraries such as TensorFlow, PyTorch, Scikit-learn, and MLflow is a plus.
- Experience configuring and integrating external AI models and working with AI governance and monitoring tools is a plus.
- Experience with asynchronous programming patterns in Python for building scalable data or AI workloads is a plus.
- Strong problem-solving, collaboration, and communication skills.
Benefits
- Health insurance
- 401(k) matching
- Paid time off
- Professional development
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Lead/SME
Cormac CorporationAt CORMAC, we leverage the power of data management and analytics to enable our customers to achieve their strategic goals. With over 20 years of experience in health information technology (HIT), human-centered design principles, and Agile development methodologies, CORMAC delivers complex digital solutions to solve some of the most challenging problems facing public healthcare programs today.
Role Description CORMAC is seeking a Data Lead / SME to be responsible for providing deep functional and technical expertise on Centers for Medicare & Medicaid Services (CMS) data domains, business processes, and system relationships across the enterprise data platform. Serves as the primary expert for understanding how CMS data sources, programs, workflows, and reporting requirements connect across claims, enrollment, provider, beneficiary, and operational systems to support analytics, compliance, and decision-making. - Serve as the primary SME for CMS data domains, including claims, eligibility, enrollment, provider, beneficiary, financial, quality, and operational datasets. - Provide expertise on how CMS systems, programs, and data flows interact across enterprise platforms and downstream reporting environments. - Support data mapping, source-to-target analysis, and business rule validation for data ingestion, transformation, and reporting initiatives. - Work closely with data architects, system architects, business analysts, and engineering teams to ensure accurate interpretation and use of CMS data. - Define and validate business logic, data lineage, and cross-system relationships for enterprise analytics and reporting solutions. - Assist with metadata management, data catalog governance, and stewardship activities to improve data discovery and usability. - Support data quality assessments, reconciliation efforts, and root cause analysis for data discrepancies and reporting issues. - Provide guidance on CMS program requirements, policy impacts, and operational dependencies affecting data architecture and analytics. - Collaborate with stakeholders to translate complex business and policy requirements into actionable technical and reporting requirements. - Support user adoption of data platforms such as Databricks, reporting portals, and metadata catalog solutions. - Participate in governance reviews, audits, compliance activities, and documentation efforts related to federal healthcare data management. - Develop business process documentation, data dictionaries, source system documentation, and knowledge transfer materials. Qualifications - Bachelor’s degree in Health Informatics, Information Systems, Public Health, Business, or related field. - Must be U.S. Citizen. - Must be eligible to obtain a Public Trust (PT) Clearance. - 5+ years of experience working with CMS programs, operations, and enterprise healthcare data environments. - Prior experience supporting Centers for Medicare & Medicaid Services programs, federal healthcare contracts, or Medicare/Medicaid operations. - Deep understanding of claims, provider, enrollment, beneficiary, and healthcare financial data structures. - Experience working with federal healthcare systems, reporting environments, and regulatory requirements. - Strong understanding of data lineage, metadata, business rules, and enterprise data governance. - Ability to interpret policy, operational processes, and system dependencies across multiple CMS domains. - Experience supporting analytics platforms such as Databricks, data warehouses, and reporting tools. - Strong stakeholder engagement, communication, and cross-functional collaboration skills. - Ability to bridge business operations and technical implementation teams. Preferred Skills & Experience - Master’s degree in Health Informatics, Information Systems, Public Health, Business, or related field. - Familiarity with metadata catalog tools, enterprise data governance platforms, and reporting solutions such as Amazon QuickSight, Power BI, or Tableau. Location Leesburg, VA Work Arrangement 100% Remote Why CORMAC? At CORMAC, we leverage the power of data management and analytics to enable our customers to achieve their strategic goals. With over 20 years of experience in health information technology (HIT), human-centered design principles, and Agile development methodologies, CORMAC delivers complex digital solutions to solve some of the most challenging problems facing public healthcare programs today. As a US Federal Government contractor in the public healthcare sector, our work is impactful and cutting-edge while being performed in a supportive, collaborative, and welcoming environment. We offer flexible work schedules with remote, hybrid, or fully in-person workplace options to empower our employees to decide the workplace most suitable for them. At CORMAC, we have a highly diverse workforce and believe a work environment is a place where creativity, collaboration, enthusiasm, and innovation happen, regardless of location. Equal Employment Opportunity As an Equal Employment Opportunity employer, CORMAC provides equal employment opportunity to all employees and applicants without regard to an individual's protected status, including race/ethnicity, color, national origin, ancestry, religion, creed, age, gender, gender identity/expression, sexual orientation, marital status, parental status, including pregnancy, childbirth, or related conditions, disability, military service, veteran status, genetic information, or any other protected status.
• Design and implement a Data Lake and Lakehouse architecture from scratch, defining the layer structure (Bronze, Silver and Gold / Medallion Architecture), the logical and physical organization of data, and ensuring the solution’s scalability and resilience; • Assess and recommend the most appropriate technology stack for the client context (including Microsoft Fabric, Databricks and/or Snowflake), taking into account volume, complexity and governance requirements; • Define and implement the data governance framework — covering cataloging, classification and lineage — establishing a unified data catalog as the central management element of the platform; • Plan integration strategies to ingest data from unstructured sources (spreadsheets, SharePoint, legacy systems), defining ETL/ELT standards and loading strategies (full load or incremental as required); • Establish technical standards for data modeling, data contracts and development best practices to guide the engineering team; • Evaluate data volume and characteristics to recommend the most suitable processing platform; • Guide and direct the team in implementing the defined solutions, conducting technical reviews and ensuring adherence to architectural decisions; • Translate technical decisions into clear, accessible documentation.
Senior Data Engineer
Capital Bank, N.A.Capital Bank, N.A. is an Affirmative Action, E-Verify, and Equal Opportunity Employer.
Role Description As part of the Technology team, the Senior Data Engineer will act as a subject matter expert (SME) responsible for managing, designing, and optimizing enterprise data pipelines and database solutions that support all lines of business at Capital Bank, including our Commercial Bank, Consumer Card division, and Mortgage division. This role owns end-to-end delivery and operations across our primary technology stack (Azure, SQL Server, and Snowflake), ensuring scalable, secure, and high-performance data platforms, reliable integrations with enterprise systems, and strong data governance. The Senior Data Engineer will build and optimize ETL/ELT processes, data lake and data warehouse solutions, and data transfer/file processing workflows to enable trusted analytics and reporting across the bank. Qualifications - Bachelor’s degree or higher in Computer Science, Information Systems, or a related field. - 6+ years of experience in data engineering, ETL, database management with experience in cloud-based databases and financial services preferred. - Experience in Database Administration (DBA), managing and optimizing databases for performance and security. - 3+ years of experience designing and building data lakes and data warehouses using platforms like Azure Fabrik, Snowflake, Amazon Redshift, or Google BigQuery. - 2+ years of experience using data visualization tools like Power BI, Sisense, Google Looker, Tableau, or similar platforms. - Experience with managing data transfers and file processes, including SFTP, secure data pipelines, and real-time or batch data movement. - Experience in proactively identifying, addressing, and monitoring data quality issues, adhering to established data quality standards. - Excellent communication skills and the ability to collaborate effectively across teams and stakeholders, explain technical concepts to non-technical stakeholders. Requirements - Expertise in cloud-based database platforms, such as Azure SQL, Amazon RDS, Google Cloud Spanner, and Snowflake. - Strong knowledge of data lake and data warehouse architectures, including designing efficient schemas, partitioning strategies, and optimizing storage. - Proficiency with data integration tools and technologies, such as Apache Kafka, Apache Spark, Talend, or Informatica. - Hands-on experience building and maintaining ETL pipelines to support large-scale data environments. - Advanced SQL skills and familiarity with programming languages like Python or Java for data manipulation and automation. - Experience with data visualization platforms, including building and optimizing dashboards using Sisense, Power BI, Google Looker, Tableau, or similar tools. - Experience with CI/CD tools (e.g., GitLab, Azure DevOps, Jenkins) and data pipeline monitoring tools (e.g., Airflow, Apache NiFi, Azure Data Factory). - Strong understanding of database security best practices, including encryption, access controls, and compliance with regulatory standards. - Ability to manage data file transfers and processing workflows effectively. - Experience with database monitoring and performance tuning in cloud and hybrid environments. - Preferred experience in understanding data relationships, as well as developing and optimizing SQL queries, stored procedures, and database functions in both OLTP and OLAP systems. - Strong organizational and problem-solving skills in Agile or fast-paced environments. Benefits - Base Salary Range: $115,000 - $130,000 annually. - This role will include a yearly annual target bonus based on individual performance. - This is a 100% remote role. - Comprehensive benefits package including Medical, Dental, Vision, Company Paid Life Insurance, Disability Insurance, and more. - Company Contributions to your 401k - Regardless of your contribution. - Employee Perks: Paid Parental Leave, Employee Recognition Program, Leadership Program, Tuition Reimbursement Program, Employee Bank Checking Account, and much more! - Generous Paid Time Off and Paid Holidays - Including Paid Charity Hours to support volunteer opportunities.
Senior Data Engineer – Vaga Afirmativa para PCD
Jusbrasil💻 Descomplicamos o acesso à informação jurídica por meio da tecnologia
• Projetar e manter pipelines de ingestão e transformação de dados em larga escala (batch e streaming). • Garantir a qualidade, confiabilidade e integridade dos dados ao longo de todo o ciclo (testes, versionamento, monitoramento). • Implementar e evoluir arquitetura de dados em GCP (BigQuery, Cloud Storage, Pub/Sub, Composer). • Colaborar com equipes de Analytics, Produto e Engenharia para definir necessidades de dados e traduzir requisitos de negócio em soluções técnicas. • Participar da evolução das boas práticas de engenharia de dados e padronização interna de modelos, nomenclaturas e monitoramentos.


