Job Closed
This listing is no longer active.
We make sense of data to drive your business forward. #MakeSenseofData #DriveYourBusinessForward #PartnerYourWay
Lead Data Engineer – Databricks
Location
New Jersey
Posted
81 days ago
Salary
0
Seniority
Senior
Job Description
Lead Data Engineer – Databricks
EXL
• Lead design and optimization of complex data solutions • Oversee data engineering projects • Mentor junior engineers • Ensure solutions align with organizational goals • Provide strategic guidance.
Job Requirements
- Bachelor's/Master's in Engineering
- 5-8 years of experience
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Staff Software Engineer
MNTNMNTN provides advertising software for brands to reach their audience across Connected TV, web, and mobile. MNTN Performance TV has redefined what it means to advertise on television, transforming Connected TV into a direct-response, performance marketing channel. Our web retargeting has been leveraged by thousands of top brands for over a decade, driving billions of dollars in revenue. Our solutions give advertisers total transparency and complete control over their campaigns all with the fastest go-live in the industry. As a result, thousands of top brands have partnered with MNTN, including Tarte, Decked, and National University.
At MNTN, we put our people first, full stop. This allows our company culture to be defined by our team members and their shared values, like trust, ambition, quality, radical honesty, and compassionate leadership. It’s why we all really love working for the Hardest Working Software in Television™ (and also why we were named one of Ad Age’s Best Places To Work in 2025.) We pride ourselves on bringing unrivaled performance and simplicity to Connected TV advertising. Our self-serve technology makes running TV ads as easy as search and social, helping brands drive measurable conversions, revenue, site visits, and more. It’s what led MNTN to being named one of Fast Company's Most Innovative Companies in 2023. You can learn more about us and everything we do by visiting https://mountain.com/. We’re committed to innovation that empowers, not replaces. At MNTN, AI is a tool for growth, enhancing efficiency while keeping a people-first approach. Our goal is to streamline workflows and drive new solutions—without compromising the human element that makes our company great. So if wanting to do more, own more, and make a bigger impact comes naturally to you, then you may be the person we're looking for to join us in our next stage of growth. The Backend Reporting squad builds and operates the data foundation that powers customer-facing reporting and decision-making at MNTN. We own the full lifecycle of reporting data, from raw log-level ingestion to trusted, production-grade analytics systems that serve as the single source of truth for enterprise reporting. This includes core performance metrics, incrementality, invoicing data, medallion-layer transformations in BigQuery, SQLMesh models, Airflow orchestration, ClickHouse serving, and the systems that enforce data quality, freshness, performance, cost efficiency, and reporting SLAs. The team is accountable for reporting correctness end to end. We are hiring a Staff Software Engineer in Data Engineering to act as a senior technical leader on this platform. This is a hands-on role focused on architecting and evolving production-grade data systems with real SLAs and cost constraints. You will set technical standards, drive architectural decisions, and partner with Engineering and Product to ensure the reporting foundation scales without becoming brittle or inconsistent. What you’ll do - Own the technical architecture of the reporting data platform across ingestion, transformation, serving, and observability layers. - Design and evolve metric computation systems, including versioning strategies, backfills, schema evolution, and change management that preserve correctness and stakeholder trust. - Lead the design of transformations across the medallion architecture in BigQuery, ensuring performance, cost efficiency, and clarity of ownership. - Define and enforce engineering standards for SQLMesh models, orchestration patterns, testing coverage, and code review quality. - Build and mature automated data quality frameworks, reconciliation systems, and data contracts that prevent regressions and detect anomalies early. - Act as the technical lead during reporting incidents, driving root cause analysis and implementing durable systemic fixes rather than one-off patches. - Partner with UI, API, and Program teams to shape reporting requirements early and translate ambiguous business asks into clear, durable system design. - Continuously improve query performance, storage patterns, and workload optimization in BigQuery and ClickHouse to maintain predictable cost and latency. - Identify architectural risks and technical debt early and create pragmatic, staged plans to remediate them without stalling product velocity. - Mentor and level up Data Engineers and Analytics Engineers through design reviews, pairing, and by setting a high bar for production readiness. What success looks like - Faster, safer iteration on metrics and KPIs - Reporting metrics are consistent, versioned, and trusted across all downstream consumers. - Data incidents are rare, and when they occur, the systems in place make detection and remediation fast and methodical. - Schema changes and new metric launches happen with minimal regression risk. - Analytics workloads have predictable cost and performance characteristics. - Clear technical standards exist for modeling, orchestration, and testing, and the team consistently adheres to them. - The reporting platform evolves without accruing brittle logic or unowned complexity. What you’ll bring - 8+ years of experience in software or data engineering, including significant ownership of production analytics platforms. - Deep experience designing and operating large-scale data platforms in production environments with explicit SLAs for freshness, correctness, and performance. - Strong expertise in SQL-based analytics systems, including complex transformations, backfills, schema evolution, and metric governance. - Hands-on experience with modern analytics tooling such as SQLMesh or dbt-class frameworks, Airflow-class orchestration, and columnar data warehouses like BigQuery. - Demonstrated ability to design for operability, including observability, alerting, testing strategies, and incident management. - Strong systems thinking. You understand how upstream event design, transformation logic, serving layers, and cost models interact. - A track record of reducing operational toil through automation, standardization, and clear architectural patterns. - Ability to influence technical direction across teams without formal authority, using clear reasoning and strong design artifacts. - Comfort operating in ambiguous product spaces where requirements evolve and tradeoffs between speed, accuracy, and complexity must be made explicit. - A pragmatic mindset. You care about long-term durability, but you also know how to sequence work so the platform improves without blocking the business. MNTN Perks - 100% remote within the US - Flexible vacation policy - Annual vacation allowance for travel related expenses - Three-day weekend every month of the year - Competitive compensation - 100% healthcare coverage - 401k plan - Flexible Spending Account (FSA) for dependent, medical, and dental care - Access to coaching, therapy, and professional development About MNTN Our recruiters will always reach out using an email address ending with @mountain.com OR @mntn.com. If you’re contacted by someone without that address and they mention a Reference Code (which we never use), then that ain’t us folks. Tell those trolls to take a hike–you’re waiting to climb a MNTN. MNTN provides advertising software for brands to reach their audience across Connected TV, web, and mobile. MNTN Performance TV has redefined what it means to advertise on television, transforming Connected TV into a direct-response, performance marketing channel. Our web retargeting has been leveraged by thousands of top brands for over a decade, driving billions of dollars in revenue. Our solutions give advertisers total transparency and complete control over their campaigns all with the fastest go-live in the industry. As a result, thousands of top brands have partnered with MNTN, including Tarte, Decked, and National University. #LI-Remote
• Lead and manage ERP data migration projects across multiple client engagements. • Oversee data extraction, transformation, loading, validation, and reconciliation activities. • Develop migration plans, timelines, cutover strategies, and risk mitigation plans. • Ensure migration deliverables are completed accurately, efficiently, and on schedule. • Lead testing cycles, mock conversions, and go-live readiness reviews. • Establish and maintain migration standards, methodologies, and quality controls. • Develop validation and reconciliation frameworks to ensure data completeness and accuracy. • Create repeatable migration templates, mapping standards, and automation tools. • Define and monitor migration KPIs, quality metrics, and success criteria. • Partner with clients to identify and resolve data quality issues prior to conversion. • Serve as subject matter expert for ERP data migration strategy and execution. • Design and oversee data mapping and transformation requirements. • Support development of migration automation tools and ETL processes. • Collaborate with implementation teams to align migration activities with project milestones. • Maintain expertise across ERP platforms and evolving migration technologies. • Lead, coach, and develop a team of migration specialists and data analysts. • Establish delivery standards and promote continuous improvement. • Support hiring, onboarding, workforce planning, and capability development. • Create scalable processes that support growth of Accordant’s migration services practice. • Serve as the primary migration leader for client engagements. • Communicate migration status, risks, dependencies, and recommendations. • Facilitate data workshops, mapping sessions, validation reviews, and cutover planning meetings. • Build trusted relationships with client stakeholders and implementation teams. • Support pre-sales discussions by providing migration assessments, estimates, and recommendations.
• Design and build scalable ETL/ELT data pipelines using dbt, PySpark and other modern data transformation tools. • Develop and maintain data ingestion pipelines for generative AI workloads, including document processing, chunking and embedding workflows. • Orchestrate workflows using Airflow, Dagster or cloud-native orchestration tools. • Plan and execute data migration projects, including source analysis, schema mapping, validation and rollback strategies. • Implement Change Data Capture (CDC) solutions using industry-standard tools. • Build and maintain data quality frameworks with automated tests and validations. • Ensure data governance, security and compliance, including proper handling of PII (personally identifiable information) and enforcement of RBAC (role-based access control) policies. • Collaborate with AI Engineers and Full Stack Developers to support RAG pipelines and applications based on generative AI. • Apply event-driven architecture concepts to design scalable and reliable data processing solutions.
Databricks Pipeline Architect
ECS Tech IncAll candidates must meet the following criteria: Must be a US Citizen, no dual Citizenships. Must be able to secure a Public trust clearance. Must be able to work across multiple programs across the Federal and DOD space. The core values that ECS looks for in an engagement manager include: Teamwork, Respect, Accountability, Integrity, and Leadership.
Role Description ECS is seeking a Databricks Pipeline Architect to work remotely. This position is contingent upon additional funding. The role will support the design, implementation, and long-term operation of Databricks within a complex Government enterprise environment. Responsibilities include: - Lead data engineering efforts within Databricks, including pipeline development on AWS in an enterprise environment. - Provide unification for organization and enterprise data capabilities to break down operational silos, including mechanisms to standardize and migrate existing data to a unified data plane/warehouse. - Support the integration of Databricks with existing analytics platforms and metadata solutions. - Serve as a technical advisor to stakeholders and help upskill internal teams through guidance/documentation. - Assist and develop customer use case applications of Databricks. Qualifications - Must be a US Citizen. - Ability to successfully maintain a DHS Entry on Duty (EOD)/Public Trust. - Minimum 7 years of hands-on experience architecting, designing, and data engineering with Databricks and AWS in a government setting. - Strong understanding and experience in areas such as Apache Spark, Lakebase, Lakehouse architectures, Delta Lake, Delta Sharing, Medallion Architecture, Data & AI Governance, and use case acceleration. - Demonstrated hands-on experience with AWS Cloud Native data and ETL capabilities (S3, Lambdas, Glue, etc.). - Demonstrated experience leveraging data to provide search, enrichment, correlation, AI/ML Model Training, in alignment with government mission and business use cases. - Experience integrating Databricks with other analytics platforms. - Prior consulting or customer-facing experience in data architecture or modernization initiatives. - Strong Python skills. - Experience in Agile environments and managing cross-functional technical delivery. Requirements - Strong Databricks and AWS experience. - Familiarity with architecting solutions for scale and governance. - Comfortable integrating Databricks with other data platforms. Benefits - Salary Range: $150,000 - $180,000. - General Description of Benefits.




