Job Closed
This listing is no longer active.
We help clients turn data into decisions no matter where it lives-in apps, on-prem, in a hybrid model, or in the cloud.
Data Engineer, Databricks
Location
United States
Posted
63 days ago
Salary
0
Seniority
Senior
Job Description
Data Engineer, Databricks
Datavail
• Lead and contribute to end-to-end Databricks implementations for clients, including data migration, Lakehouse architecture, and pipeline development • Gather technical requirements, design solutions, and present recommendations to client stakeholders (technical and business) • Build scalable ETL/ELT pipelines using PySpark, Delta Lake, Delta Live Tables (DLT), and Databricks Workflows • Design and implement Databricks Genie • Design and implement semantic layers • Use Databricks AI features to accelerate development, debugging, and code optimization • Design and implement Lakebase architectures for operational and analytical workloads, including transactional data use cases • Develop solutions using SDLC best practices, including modular code design, testing, and documentation • Use Git based version control with proper branching strategies • Implement CI/CD pipelines for Databricks asset • Implement data quality checks, validations, and expectations within workflows • Design and implement Unity Catalog governance, security, and lineage solutions • Optimize Databricks workloads for performance, cost, and reliability (Photon, cluster policies, Liquid Clustering, Auto Loader, etc.) • Integrate Databricks with client ecosystems (Azure, AWS, GCP, Snowflake, Kafka, legacy systems, etc.) • Support client workshops, proof-of-concepts (POCs), and knowledge transfer sessions • Deliver projects following consulting methodologies while meeting quality, timeline, and budget expectations • Document architectures, runbooks, and best practices for client use • Participate in solutioning activities (scoping, estimation, technical demos) as needed
Job Requirements
- 3 -5 years of hands-on Databricks experience (or strong Spark experience with significant recent Databricks work)
- Proven experience delivering Databricks projects in a consulting or professional services environment (preferred) or equivalent client-facing project delivery
- Strong proficiency in PySpark, Spark SQL, Python, and SQL
- Deep experience with Delta Lake, Unity Catalog, Delta Live Tables, and Databricks Jobs
- Hands-on experience with Git version control, pull requests, code reviews, and collaborative development workflows
- Cloud platform experience (Azure Databricks, AWS, or GCP - at least one)
- Excellent client-facing and communication skills - able to explain complex concepts to both technical and non-technical audiences
- Solid understanding of data governance, security, and Lakehouse best practices
- Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience)
Benefits
- As a Databricks Data Engineer, you will work directly with clients across multiple industries to design, implement, and optimize Databricks-based data solutions
- You will be a key member of our Professional Services delivery teams, delivering high-quality projects on time and within scope while building strong client relationships
- This is a client-facing role that combines hands-on technical delivery with consulting best practices
- Collaborate with client data teams to ensure successful adoption and handover of solutions
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Guide clients on optimizing their data environment to work most effectively and efficiently for them. Clarify management objectives through data solutions. • Develop system engineering, integrations, and architectures based on client needs. • Implement and provide advice on data warehouse solutions, ETL pipelines, and business intelligence reporting tools. • Develop a data model around stated use cases to capture client’s KPIs and data transformations. Validation and testing of data models. • Teach technical data modeling concepts to a variety of audiences, including developers, data architects, business users, and IT professionals. • Support, maintain, and document clients’ data environments. • Work within a project management framework to meet objectives, understand scope, and impact of your work across an organization. • Work with the Sales and Marketing teams to develop Thought Leadership and support our sellers by talking to clients about DAS42’s capabilities.
• Build Pipelines: Design and maintain scalable data pipelines to ingest and enrich healthcare data using Databricks and Spark. • Optimize Data Workflows: Improve data intake processes and optimize SparkSQL/Python workloads for performance, scalability, and cost efficiency. • Design Data Models: Partner with senior engineers to develop data marts and semantic models that power analytics products and reporting. • Ensure Quality: Monitor pipeline health, troubleshoot failures, and implement data validation and quality controls. • Learn & Grow: Expand your knowledge of the full data lifecycle, cloud infrastructure (Azure/AWS), and healthcare data standards.
Lead Data Engineer – Platform
ValtechA pioneer in the fields of digital and technology, Valtech is a global business-transformation agency that was founded in 1993 to deliver "innovation with a pur
• Design, build, and maintain scalable data engineering frameworks and platform utilities used across engineering teams • Develop reusable patterns, templates, and abstractions to standardise and accelerate delivery • Define and evolve platform architecture decisions, ensuring scalability, maintainability and consistency • Design and implement CI/CD pipelines and automation frameworks to improve engineering velocity • Define and enforce engineering standards for testing, code quality, deployment and documentation • Identify and eliminate manual or repetitive processes through automation and tooling improvements • Integrate AI-assisted development tools into engineering workflows to improve productivity • Develop and maintain AI engineering assets such as coding guidelines, prompt frameworks and reusable agent configurations • Lead the development and operational support of core data transformation frameworks (including dbt Core at enterprise scale) • Investigate and resolve framework-level issues, including deployment failures, dependency conflicts and production incidents • Support onboarding and enablement of engineering teams adopting platform tooling • Act as the main technical point of contact for platform and framework-related queries • Partner with engineering teams to identify pain points and translate them into platform improvements • Ensure platform tooling meets security, compliance and operational requirements • Conduct and support code and design reviews across platform components • Monitor platform health, performance and adoption, iterating based on feedback and metrics • Contribute to documentation, developer guides and enablement materials to improve usability and adoption
Data Engineer I
ProvidenceAt Providence, our strength lies in Our Promise of “Know me, care for me, ease my way.” Working at our family of organizations means that regardless of your role, we’ll walk alongside you in your career, supporting you so you can support others. We provide best-in-class benefits and foster an inclusive workplace where diversity is valued, and everyone is essential, heard, and respected. Together, our 120,000 caregivers serve in over 50 hospitals, over 1,000 clinics and a full range of health and social services across Alaska, California, Montana, New Mexico, Oregon, Texas and Washington. As a comprehensive health care organization, we are serving more people, advancing best practices and continuing our more than 100-year tradition of serving the poor and vulnerable.
• Designs and builds modern data-centric software applications to support clinical and operational processes • Builds data pipelines and transformations, data enrichment processes, provisioning layers, and user interfaces • Works closely with the Product, Platform, and Architecture teams to deliver on joint efforts



