SugarCRM logo
SugarCRM

SugarCRM is the CRM platform that makes the hard things easier. With Sugar, you let the platform do the work.

Senior Data Engineer – Databricks

Data EngineerData EngineerFull TimeRemoteSeniorTeam 501-1,000Since 2004H1B SponsorCompany SiteLinkedIn

Location

United States

Posted

5 days ago

Salary

$155K - $185K / year

Seniority

Senior

Job Description

Senior Data Engineer – Databricks

SugarCRM

• Own Databricks production support for the Sugar Predict data platform, including monitoring, alerting, and incident response across all production data flows • Maintain and report on SLA performance metrics for data pipeline delivery, ensuring visibility into platform health and accountability across internal and external stakeholders • Identify and implement pipeline optimizations that reduce Databricks compute costs, improve throughput, and reduce processing windows while tracking impacts through measurable KPIs • Migrate legacy ETL/ELT pipelines to Databricks, building automation tooling to reduce manual intervention and ensure uninterrupted data delivery during transitions • Support new customers onboarding by provisioning, validating, and hardening tenant data pipelines that deliver reliable, isolated data from day one • Design and build high-performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale across both Azure and AWS environments • Own the Delta Lake architecture including schema design, partitioning strategies, data quality enforcement, and incremental processing patterns • Enforce data security best practices across Databricks environments, including role-based access control, secrets management, and compliance requirements for enterprise CRM and ERP data • Implement data quality monitoring and observability across pipeline health and ML model inputs, ensuring data integrity that directly supports Sugar Predict prediction accuracy • Apply and enforce multi-tenant data isolation patterns ensuring reliable, secure data delivery across Sugar Predict enterprise customers • Partner with the Enterprise Architecture team to ensure Sugar Predict data pipelines integrate seamlessly with the broader SugarAI product ecosystem • Support a globally distributed operation through on-call rotation and after-hours incident response, meeting SLAs across multiple time zones • Maintain technical documentation, runbooks, and architectural decision records, contributing to team knowledge sharing and operational readiness across on-call and incident response scenarios • Apply CI/CD best practices to data pipeline development, including version control, automated testing, and deployment tooling to ensure reliable and repeatable pipeline delivery

Job Requirements

  • 4+ years of data engineering experience
  • At least 2 years on Databricks or the Apache Spark ecosystem across Azure and/or AWS
  • Proficiency in PySpark, SQL, and Python with a strong track record building and operating production-grade pipelines under SLA constraints
  • Hands-on experience with Delta Lake including schema evolution, ACID transactions, optimize/vacuum lifecycle, and both incremental and streaming processing patterns
  • Hands-on experience with pipeline performance tuning and compute optimization in production Databricks environments
  • Solid working knowledge of PostgreSQL including query optimization, schema design, and use as a source or sink in production data pipelines
  • Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines, or similar) in production
  • Experience supporting large-scale multi-tenant architectures with a focus on tenant isolation, per-tenant performance, and data privacy, including navigating tools and platforms that default to single-tenant assumptions
  • Proven ability to work collaboratively across data science, product, and infrastructure teams, owning end-to-end delivery in a cross-functional environment
  • Strong understanding of data governance, security, and compliance principles, including access control, data privacy, and protection of sensitive enterprise data across multi-tenant environments

Benefits

  • Excellent healthcare package for you and your family
  • Savings and Investment – 401(k) match
  • Unlimited Paid Time Off
  • Paid Parental Leave
  • Online Legal Services (Rocket Lawyer)
  • Financial Planning Services (Origin)
  • Discounted Pet Insurance (Embrace Pet Insurance)
  • Corporate Benefit Program (Working Advantage). This benefit offers you exclusive travel and entertainment offers and special discounts that are not available to the general public
  • Health and Wellness Reimbursement Program
  • Travel Discounts
  • Educational Resources - Career & Personal Development Program
  • Employee Referral Bonus Program
  • We are a merit-based company - many opportunities to learn, excel and grow your career!

Related Categories

Related Job Pages

More Data Engineer Jobs

Full TimeRemoteTeam 1,001-5,000H1B No Sponsor

• Assist in building and maintaining ETL/ELT pipelines for healthcare datasets including claims, eligibility, provider, risk adjustment, HEDIS, EHR, and clinical data • Support the development of data models and data transformations aligned with healthcare standards (e.g., HL7, FHIR, X12) • Contribute to data quality checks, validation rules, and documentation for healthcare data assets • Work with analysts and business users to understand data requirements and translate them into technical tasks • Assist in ingestion and integration of new data sources from EMR systems, CMS feeds, and vendor partners • Develop SQL queries, transformations, functions, and stored procedures to support reporting and analytics workflows • Support data platform tools such as Azure Data Factory, Databricks, Python/Spark jobs, and version control workflows • Participate in issue resolution related to data pipeline failures or data quality errors • Maintain data dictionaries, mapping files, and documentation as part of data governance processes • Collaborate with senior engineers to implement best practices in security, compliance (HIPAA), and architecture

United States
Job Closed
Full TimeRemoteTeam 51-200H1B No Sponsor

Role Description The Data Engineering Practice Leader will serve as a senior pillar lead responsible for shaping data engineering strategy, growing a book of business, leading practice development, and overseeing delivery of enterprise-scale data transformation initiatives. This role sits within FormativGroup’s Data & Analytics practice and blends consulting leadership, executive advisory, technical oversight, commercial ownership, and team development. The position is approximately 25% billable delivery and 75% business development, practice ownership, and strategic leadership. As the Practice Leader, you will partner with executive client stakeholders, lead account growth, oversee solution architecture, guide delivery teams, and help define scalable, modern data solutions across Snowflake, Databricks, AWS, Azure, Microsoft Fabric, and related cloud data ecosystems. What You'll Work On - Serve as a senior pillar lead for the Data & Analytics - Data Engineering practice. - Own and grow a book of business through client relationship leadership, business development, and strategic account expansion. - Lead strategic data modernization discussions with executive and senior client stakeholders. - Shape solution strategy, future-state architecture, and transformation roadmaps for enterprise clients. - Oversee delivery of complex data engineering programs, ensuring alignment to business outcomes, architecture standards, and delivery quality. - Develop new data engineering offerings, accelerators, products, and reusable delivery assets. - Provide technical oversight across Snowflake, Databricks, AWS, Azure, Microsoft Fabric, and equivalent modern data platforms. - Guide solutioning for data lakes, data warehouses, lakehouses, advanced analytics enablement, and AI-ready data platforms. - Lead proposal strategy, pursuit support, client presentations, and commercial solution development. - Partner with internal leadership to define practice priorities, talent strategy, delivery methodology, and growth plans. - Manage, mentor, and develop senior managers, managers, architects, consultants, and cross-functional delivery teams. - Establish governance, data quality, metadata, lineage, security, privacy, and compliance standards across client programs. - Evaluate emerging data, analytics, and AI-enablement technologies and determine applicability to client and practice needs. - Monitor portfolio performance, delivery health, financial outcomes, and client satisfaction. - Represent FormativGroup as a trusted advisor in data engineering, analytics modernization, and cloud data transformation. Qualifications - 15+ years of experience in data engineering, analytics engineering, data architecture, cloud data platforms, or related technology consulting roles. - 10+ years of leadership experience within a traditional consulting firm, technology consulting firm, systems integrator, or professional services organization. - Bachelor’s degree in Data Analytics, Business Analytics, Information Systems, Computer Science, Statistics, Mathematics, Economics, or a related field. - Strong technical background with the ability to solution, oversee architecture, evaluate technical tradeoffs, and develop new data products or offerings. - Executive-level advisory experience across analytics strategy, data modernization, cloud transformation, and business value realization. - Strong understanding of data architecture, data modeling, ETL/ELT, pipeline design, data integration, governance, security, and compliance. - Experience leading large enterprise programs, multi-year engagements, or complex consulting portfolios. - Experience with proposal development, pursuit strategy, solution design, pricing support, and executive presentations. - Strong commercial mindset with the ability to balance client outcomes, delivery quality, and financial performance. - Exceptional communication, executive presence, facilitation, negotiation, and stakeholder management skills. - Experience leading account growth, business development, relationship leadership, and strategic account expansion. Benefits - Discretionary bonuses, commissions, or other incentive programs. - Comprehensive benefits package that includes medical, dental, vision, 401(k), paid time off, etc. Employment Eligibility Applicants must be authorized to work for ANY employer in the U.S. We are unable to sponsor or take over sponsorship of an employment visa currently. This is a remote role with approximately 50% travel. The preferred location is the Northeast U.S. To be considered for this position, candidates must reside in one of the following U.S. states or Washington, DC: AL, AR, AZ, CA, CO, CT, DE, FL, GA, IA, ID, IL, IN, KS, MA, MD, MI, MN, MO, NC, NH, NJ, NV, NY, OH, OK, OR, PA, TN, TX, VA, WI, or Washington, DC. Candidates residing outside these locations are not eligible for consideration currently. Compensation The estimated compensation range for this position is $185,000 — $265,000 USD. The actual compensation offered will be determined based on factors such as the candidate’s experience, skills, education, work location, and internal equity. Company Description FormativGroup operates within the critical middle layer of business technology, where applications and systems connect infrastructure to business processes. We are specialists who help the middle market take full advantage of their technology investments with deep, industry-centric expertise, all in one place, to unify fragmented systems. With deep technical expertise across cloud architecture, system integration, AI, and data strategy, we bridge the gap between business goals and modern platforms. FormativGroup is an equal opportunity employer providing opportunities to applicants and employees without regard to race, color, religion, age, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status.

United States
$185K - $265K / year
Job Closed
Zenni Optical logo

Manager, Data Engineering

Zenni Optical

Zenni Optical is an online retailer that specializes in prescription eyewear. As an employer, the company aims to cultivate a flexible work environment that pro

Data Engineer5 days ago

Role Description Zenni is experiencing rapid growth in the affordable eyewear space, and data is at the core of our success. We are seeking a Manager, Data Engineering with a strong technical background and deep business domain expertise to partner closely with our Engagement Marketing team. In this leadership role, you will bridge the gap between complex data architecture and strategic marketing initiatives. You will be responsible for translating large datasets into clear, actionable insights, empowering our cross-functional teams to make confident, informed decisions. By building robust data solutions and fostering strong collaboration across departments, you will directly impact our operational efficiency and drive revenue growth. If you are a collaborative leader who excels at leveraging data to solve real-world business challenges, this could be the next move for you! This is a remote role overseeing a team of 2 and reporting to the Sr. Director of Data Engineering. Responsibilities - Champion data literacy: Provide ongoing training and support to the Engagement Marketing team, ensuring complex data concepts are seamlessly communicated and understood. - Drive cross-functional alignment: Act as the strategic bridge between the Data and Engagement Marketing teams, streamlining communication to boost operational efficiency and collaboration. - Power marketing platforms: Integrate enriched, highly trusted data with key business platforms (e.g., Google, Tik Tok, Meta) to optimize ad delivery and AI performance. - Ensure data integrity: Own the lifecycle of data feeds, encompassing validation, monitoring, testing, and proactive alerting, to guarantee uncompromising data quality and reliability. - Define success metrics: Partner with functional teams to establish a shared, accurate understanding of the core metrics that drive organizational performance. - Shape the data roadmap: Lead the strategic planning for the data team, ensuring technical initiatives directly align with Zenni’s overarching growth objectives. - Innovate continuously: Stay at the forefront of modern data engineering technologies and industry trends to evolve our strategy and keep our technical capabilities competitive. Qualifications - Bachelor’s degree in Computer Science, Machine Learning, Engineering, or a related technical discipline. - 8+ years of dedicated experience in the data area, coupled with at least 3 years navigating the fast-paced e-commerce landscape. - Deep business domain knowledge across key marketing functions, including but not limited to Performance Marketing, Lifecycle, and Brand marketing. - Hands-on experience implementing marketing measurement methodologies, including Last-Touch, MTA, incrementality testing, MMM, and CVR measurement frameworks. - Extensive experience designing data architecture, performing complex data modeling, and building scalable ETL/ELT pipelines for centralized data warehouses. - Advanced fluency in SQL and Python, with the ability to dive deeply into a modern tech stack as a hands-on contributor. - Demonstrated success leading projects from start to finish, spanning requirement gathering, cross-functional stakeholder communication, architecture design, implementation, and QA. - Proactive leadership and cross-functional communication skills, with a proven ability to drive business impact. Preferred Qualifications - Hands-on expertise working within Databricks environments. - Proven track record of guiding and mentoring high-velocity data teams in a fast-paced setting. - Demonstrated success managing complex data integrations, advanced reporting, and platform optimization across diverse marketing channels. - An enthusiastic, proactive approach to tackling new technical challenges and driving continuous improvement. Benefits - Annual bonus program - 100% employer-paid medical, dental, and vision (dependent coverage capped at $300/month) - Life, AD&D, and disability coverage (with optional add-ons) - Company-matched RRSP contributions up to 4% - Comprehensive wellbeing programs (mental, physical, emotional support) - Lifestyle Spending Account (wellbeing + learning) - Work-from-home stipend (for eligible roles) - Generous Flexible Time Off in addition to our Week of Zen - the time between Christmas and New Year’s to relax and recuperate - Fully Paid Parental Leave - 16 weeks for birthing and non-birthing parents - Family forming & fertility support and services through Maven - Annual Zenni Gift Card - to use towards our products Compensation In addition to other forms of compensation like perks and benefits, the estimated range for this role is CAD 163,000 - 185,000. The final offer will be determined based on permissible, non-discriminatory factors such as skill set, experience, geographical location, market conditions, and other organizational needs. The Company reserves the right to change, modify, or revisit the salary range for various reasons including business needs.

Canada
C$163K - C$185K / year
Inspiren logo

Manager, Data/ML Platform

Inspiren

Inspiren offers the most complete and connected ecosystem in senior living. Founded by Michael Wang, a former Green Beret turned cardiothoracic nurse, Inspiren proves that compassionate care and technology can coexist - bringing peace of mind to residents, families, and staff. Our integrated solutions seamlessly fit into existing workflows, capturing everything happening within a community. Backed by nurse specialists and powerful analytics, we provide the data operators need to make informed clinical and operational decisions - driving efficiency, profitability, and better care outcomes.

Data Engineer5 days ago
Full TimeRemoteTeam 1-10Since 2016H1B No Sponsor

• Lead Data + ML Platform strategy and execution • Own technical direction for the data and ML platform, spanning data engineering, data platform, and ML Ops. • Invest in ML Ops capabilities that accelerate ML enablement: offline experimentation, model registry, shadow deployments and testing, drift/product monitoring, online and offline feature stores, and training data management. • Build measurement and experimentation frameworks for LLM and agentic workflows. • Invest in data platform and data engineering capabilities to handle large-scale streaming and batch data across modalities. • Modularize and scale our system, increase reliability, and modernize legacy infrastructure. • Make it easy for ML engineers to have ideas, test them, shadow deploy, gather data, test online, and ship. • Set clear success criteria for platform capabilities and hold the team accountable to outcomes. • Anticipate skill gaps and build a deliberate hiring plan; consistently attract and retain strong data and ML platform engineers. • Coach with depth: set explicit expectations, give direct feedback, and accelerate individual growth. • Build an environment of ownership, trust, and continuous learning. • Develop future leaders and create succession within the team. • Work closely with Product, Hardware, Data Science, and Analytics to shape problem definition and roadmap. • Translate ambiguous business needs into concrete technical direction. • Influence hardware and device design decisions where they intersect with data and ML needs. • Contribute to company strategy at the intersection of data, ML, devices, and care outcomes.

New York
$200K - $230K / year