Data Science, Digital Transformation and eCommerce Strategy from experienced eCommerce and AI/ML experts
Lead Databricks Architect
Location
United States
Posted
69 days ago
Salary
0
Seniority
Lead
Job Description
Lead Databricks Architect
Nimble Gravity
Do you want to design and scale enterprise data platforms that power analytics, AI, and real business impact? Ready to lead architectural decisions across complex, high-impact data environments? Then this role is for you. At Nimble Gravity, we turn complex data into actionable insight—and that starts with building scalable, high-performing data platforms. We’re seeking a Lead Databricks Architect who is passionate about designing and operating modern data ecosystems, driving platform excellence, and enabling organizations to unlock the full value of their data. If you thrive at the intersection of data architecture, platform ownership, and hands-on technical execution, and enjoy leading high-impact initiatives in complex environments, then you belong with us. Enterprise Data Architecture & Modernization - Act as the senior technical authority for enterprise data architecture across client engagements. - Design and evolve cloud-native, best-of-breed data platforms for our clients, supporting analytics, reporting, automation, and AI at enterprise scale, with a strong focus on Databricks as a core platform. - Define target-state architectures for data ingestion, transformation, storage, semantic layers, and consumption across federated domains. - Establish and enforce architectural standards for scalability, performance, security, resiliency, and cost efficiency, including platform-level governance and operational best practices. Modern Data Pipelines & Platforms - Architect and guide the implementation of modern data pipelines using tools such as Databricks and dbt. - Design ELT and streaming patterns that are testable, observable, and resilient. - Define data modeling standards (analytical, dimensional, and semantic) that enable trusted self-service analytics. - Partner with client data engineering teams to ensure consistent application of patterns and reuse of shared platform capabilities, including effective use and management of Databricks environments. AI-Ready & Semantic Data Foundations - Design AI-ready data foundations, including semantic layers, contextual metadata, and reusable data contracts. - Enable machine learning and generative AI use cases by ensuring data is well-modeled, discoverable, governed, and explainable. - Collaborate with client Data Science and Analytics teams to support feature stores, experimentation environments, and reusable data products. - Advance semantic modeling and context engineering to enable natural-language analytics and AI-driven insights. Data Governance, Quality & Trust - Partner with client Data Governance stakeholders to define standards for data quality, metadata management, lineage, and stewardship. - Ensure architectural designs support regulatory and compliance requirements (e.g., SOX, SEC, FINRA, GDPR, DORA). - Promote reuse, interoperability, and consistent definitions of critical enterprise data across client environments, including alignment with platform governance models. Databricks Platform Ownership & Administration - Own and administer Databricks workspaces across client environments, including configuration, access controls, and governance (Unity Catalog, IAM, RBAC). - Manage cluster configurations, job orchestration, and cost optimization to ensure efficient and reliable platform operations. - Monitor platform performance, troubleshoot issues, and ensure SLAs for data pipelines and workloads. - Establish best practices for CI/CD, environment management, and operational support within Databricks. - Partner with client teams to operationalize and scale Databricks as a core enterprise data platform. Qualifications Education & Certificates - Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or related fields. - Cloud or data architecture certifications (AWS, data management, analytics) preferred. Professional Experience - 10+ years of experience in data architecture, data engineering, or enterprise analytics. - Proven experience designing and operating modern, cloud-native data platforms. - Hands-on expertise with AWS-based data ecosystems and modern analytics stacks. - Strong experience building data platforms that support AI, machine learning, and advanced analytics. - Experience operating in federated data models and complex enterprise environments. - Strong hands-on experience with Databricks, including workspace setup, configuration, administration, and platform optimization; experience with AWS setup and administration is strongly preferred. Competencies & Attributes - Deep technical fluency combined with strong business acumen. - Ability to translate strategy into executable architectural designs. - Pragmatic, delivery-oriented mindset with strong attention to data quality and sustainability. - Collaborative, trusted partner to senior client stakeholders and technical teams. - Comfortable operating in high-visibility, transformational initiatives. Nimble Gravity is an Equal Opportunity Employer and considers applicants for employment without regard to race, color, religion, sex, orientation, national origin, age, disability, genetics or any other basis forbidden under federal, state, or local law. Nimble Gravity considers all qualified applicants. This position is open only to candidates who are legally authorized to work in the United States. We do not provide visa sponsorship for this role.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Engineering Architect – Manager
Naveera Technology LLCEngineering Production-Ready Data, AI & Cloud Platforms - Scalable, Secure, and Built for Enterprise Growth.
• Design and implement scalable Data Lakehouse architectures on AWS • Build and optimize data warehouse solutions using Amazon Redshift and Snowflake • Develop and manage ETL/ELT pipelines using PySpark and AWS Glue • Design Fact and Dimension data models for analytics workloads • Optimize SQL queries, data distribution, and partition strategies • Process and transform structured and semi-structured data using PySpark • Manage data transformation workflows using DBT • Ensure data quality, performance, and reliability across data pipelines • Work with BI tools such as Tableau or Power BI for reporting and analytics • Collaborate with data engineers, analysts, and business teams to deliver scalable data solutions
Senior Software Engineer, Data Platform Team
SysdigConfidently secure containers, Kubernetes and cloud services with #SecureDevOps.
• Own the design and development of features and components for the data platform, focusing on high-throughput data ingestion, transformation, and storage. You will report to the Director, Engineering. • Architect and implement robust, distributed, and scalable data processing pipelines in Go to ensure data quality and reliability. • Contribute to the technical strategy and roadmap for the data platform, anticipating future data needs for product features and internal analytics. • Mentor junior and mid-level engineers on the team, and conduct thorough code reviews to ensure quality and best practices. • Participate in an on-call rotation to address urgent operational issues impacting data services.
Sr. Distinguished, Data Engineer - Enterprise Data Storage and Consumption Platforms Data Engineer - Remote-Eligible
Capital OneAt Capital One, we think and work like a tech company, using our digital fluency to transform everything about the customer experience. We’re bending data to our will, and turning a stodgy industry on its head. That’s reflected in our ranking as the number one business technology innovator in the U.S. in the 2016 InformationWeek Elite 100.
Sr. Distinguished, Data Engineer - Enterprise Data Storage and Consumption Platforms Data Engineer - Remote-Eligible Distinguished Data Engineers are individual contributors who strive to be diverse in thought so we visualize the problem space. At Capital One, we believe diversity of thought strengthens our ability to influence, collaborate and provide the most innovative solutions across organizational boundaries. Distinguished Engineers will significantly impact our trajectory and devise clear roadmaps to deliver next generation technology solutions. About the Team: You will join the Data Storage & Consumption Platform team within Enterprise Data, serving as a Senior Director-level Individual Contributor (IC). This role demands a visionary and hands-on leader to drive the technical strategy for consumption platforms leveraged by all lines of business. Your focus is on maturing platform scalability, Operational Excellence, Performance, and Resiliency. We require deep expertise in AWS infrastructure ,Lakehouse architecture, Snowflake, Databricks platform and BI tools to mature these enterprise-scale systems and execute technical strategy. Our Sr. Distinguished Engineers Are: - Deep technical experts and thought leaders that help accelerate adoption of the very best engineering practices, while maintaining knowledge on industry innovations, trends and practices - Visionaries, collaborating on Capital One's toughest issues, to deliver on business needs that directly impact the lives of our customers and associates - Role models and mentors, helping to coach and strengthen the technical expertise and know-how of our engineering and product community - Evangelists, both internally and externally, helping to elevate the Distinguished Engineering community and establish themselves as a go-to resource on given technologies and technology-enabled capabilities Responsibilities: - Build awareness, increase knowledge and drive adoption of modern technologies, sharing consumer and engineering benefits to gain buy-in - Identify, benchmark and contribute towards opportunities to improve system availability, security, reliability, cost optimization and performance - Strike the right balance between lending expertise and providing an inclusive environment where others' ideas can be heard and championed; leverage expertise to grow skills in the broader Capital One team - Promote a culture of engineering excellence, using opportunities to reuse and innersource solutions where possible - Effectively communicate with and influence key stakeholders across the enterprise, at all levels of the organization - Operate as a trusted advisor for a specific technology, platform or capability domain, helping to shape use cases and implementation in an unified manner - Lead the way in creating next-generation talent for Tech, mentoring internal talent and actively recruiting external talent to bolster Capital One's Tech talent Basic Qualifications: - Bachelor's Degree - At least 7 years of experience in software engineering or data engineering - At least 5 years of people management experience - At least 3 years of experience with Big Data and data streaming - At least 3 years of experience with cloud-based data platforms (Amazon Web Services, Microsoft Azure, Google Cloud Platform) - At least 9 years of experience in data engineering writing production quality code with a focus on distributed systems Preferred Qualifications: - Masters' Degree - 10+ years of experience in data engineering - 3+ years of experience implementing big data frameworks and processing solutions - 3+ years of data modeling experience - 2+ years of experience with ontology standards for defining a domain - 2+ years of experience using Python, SQL or Scala - 1+ year of experience deploying machine learning models - 5+ years of experience with Agile engineering practices - 8+ years in-depth experience with the Hadoop stack (MapReduce, Pig, Hive, Hbase ) - 8+ years experience with NoSQL implementation (Mongo, Cassandra) - 8+ years experience developing Java based software solutions - 8+ years experience in at least one scripting language - 8+ years experience developing software solutions to solve complex business problems - 8+ years experience with Relational Database Systems and SQL - 8+ years experience designing, developing, and implementing ETL - 8+ years experience with UNIX/Linux including basic commands and shell scripting - 8+ years of experience providing technical leadership - 8+ years of experience leading the full life-cycle of IT development and platform support Capital One is open to hiring a Remote Employee for this opportunity. At this time, Capital One will not sponsor a new applicant for employment authorization, or offer any immigration related support for this position (i.e. H1B, F-1 OPT, F-1 STEM OPT, F-1 CPT, J-1, TN, E-2, E-3, L-1 and O-1, or any EADs or other forms of work authorization that require immigration support from an employer). The minimum and maximum full-time annual salaries for this role are listed below, by location. Please note that this salary information is solely for candidates hired to perform work within one of these locations, and refers to the amount Capital One is willing to pay at the time of this posting. Salaries for part-time roles will be prorated based upon the agreed upon number of hours to be regularly worked. Remote (Regardless of Location): $286,200 - $326,700 for Sr. Dir, Data Engineering Richmond, VA: $286,200 - $326,700 for Sr. Dir, Data Engineering Candidates hired to work in other locations will be subject to the pay range associated with that location, and the actual annualized salary amount offered to any candidate at the time of hire will be reflected solely in the candidate's offer letter. This role is also eligible to earn performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI). Incentives could be discretionary or non discretionary depending on the plan. Capital One offers a comprehensive, competitive, and inclusive set of health, financial and other benefits that support your total well-being. Learn more at the Capital One Careers website . Eligibility varies based on full or part-time status, exempt or non-exempt status, and management level. This role is expected to accept applications for a minimum of 5 business days. No agencies please. Capital One is an equal opportunity employer (EOE, including disability/vet) committed to non-discrimination in compliance with applicable federal, state, and local laws. Capital One promotes a drug-free workplace. Capital One will consider for employment qualified applicants with a criminal history in a manner consistent with the requirements of applicable laws regarding criminal background inquiries, including, to the extent applicable, Article 23-A of the New York Correction Law; San Francisco, California Police Code Article 49, Sections 4901-4920; New York City's Fair Chance Act; Philadelphia's Fair Criminal Records Screening Act; and other applicable federal, state, and local laws and regulations regarding criminal background inquiries. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation, please contact Capital One Recruiting at 1-800-304-9102 or via email at RecruitingAccommodation@capitalone.com . All information you provide will be kept confidential and will be used only to the extent required to provide needed reasonable accommodations. For technical support or questions about Capital One's recruiting process, please send an email to Careers@capitalone.com Capital One does not provide, endorse nor guarantee and is not liable for third-party products, services, educational tools or other information available through this site. Capital One Financial is made up of several different entities. Please note that any position posted in Canada is for Capital One Canada, any position posted in the United Kingdom is for Capital One Europe and any position posted in the Philippines is for Capital One Philippines Service Corp. (COPSSC).
• Contribute to the design and development of our Data platform. • Work closely with AI / ML Engineers to build intelligence and automation into all Canoe systems. • Design and implement data models, data pipelines, and data architectures in support of new Canoe product offerings. • Participate in cross-functional teams as Data Engineering’s representative. • Build observable systems and be an organization leader in operational excellence.




