Solve for today, evolve for tomorrow.
Lead Consultant, Data Engineer
Location
Virginia
Posted
43 days ago
Salary
0
Seniority
Senior
Job Description
Lead Consultant, Data Engineer
Excella
• Lead the design, development, and optimization of scalable data solutions. • Define data architecture strategies and ensure best practices are followed. • Oversee and guide the creation and automation of data pipelines and platforms. • Establish and enforce data quality and governance frameworks. • Collaborate with Architects, Product Owner, Data Scientists, and DevOps to align data solutions with business needs. • Research and evaluate emerging data technologies and methodologies. • Ensure seamless integration of data management solutions into client environments. • Develop risk mitigation strategies and implement data recovery plans. • Lead the development of data repositories, including data warehouses, data lakes, and operational data stores. • Mentor and support the development of junior and senior team members.
Job Requirements
- 6+ years of experience in data engineering and related technical domains.
- Designed and built scalable, extensible data pipelines for production environments with strong expertise in SQL, Python, and PySpark.
- 2+ years of hands-on experience with Databricks, including development, data processing, and pipeline optimization in a cloud-based environment.
- Hands-on experience with big data and distributed systems, Parquet, Delta Lake, HDInsight.
- Extensive work with cloud-native data platforms and services, such as Azure Data Factory, Synapse, Azure Storage Accounts, Streaming Analytics, Azure Fabric.
- Deep understanding of data lake architecture, secure and performant data store design, and analytical platform patterns.
- Proficient in orchestration tools and CI/CD workflows, infrastructure-as-code (e.g., Terraform, Azure DevOps Pipelines), and observability best practices.
- Advanced Git and Azure DevOps Repos expertise and strong familiarity with version control, branching strategies, and collaborative development.
- Leadership experience mentoring engineers and guiding cross-functional technical teams.
- Excellent communicator, capable of translating complex technical ideas for both technical and non-technical audiences.
- Agile practitioner with experience leading and delivering within Scrum/Kanban frameworks, and a continuous learning mindset.
- Technically savvy, entrepreneurial spirit who thrives in environments that reward self-initiative and resourcefulness.
Benefits
- We offer top of industry medical, dental, and vision benefits with multiple options to choose from such as an employer-contributed health savings account, infertility coverage, and orthodontia so you can select the plan that works best for you.
- Regardless of what stage of life you’re in, Excella wants to support you. We provide 8 weeks of Parental Leave, discounted pet insurance, and a Care.com membership with 3 back-up emergency child or elder care days annually – all available to you on your first day.
- Starting day one, every employee is bonus eligible and receives 15 days of paid vacation, 6 federal holidays, and 4 floating holidays.
- Doing your best work means having the best tools! Excella’s TechEleX program provides you with multiple options to suit your technology needs. Choose between a variety of Mac or PC devices, and to ensure your hardware remains current, at the end of a 3-year period Excella will replace your existing computer with a new model from the program. Plus, we’ll even give you the original device to keep for your personal use!
- With Excella’s Annual Internet Reimbursement benefit, all employees receive an additional $25 per month to help offset the cost of internet access. This initiative reflects our commitment to supporting you in staying connected and productive, no matter where you work.
- We'll invest in your career by providing 3 days of paid professional development every year, including an allowance for registration fees to attend classes, conferences, or obtain professional certifications.
- We encourage mindfulness and overall well-being through employee wellness events, a HeadSpace membership, as well as access to TalkSpace and mental health coverage through our medical plans.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Staff Data Engineer
NateraFounded in 2004 and led by CEO Steve Chapman, Natera is a company in the biotechnology market that offers genetic testing and diagnostics on a global scale. Ope
• Design, develop, and maintain robust, scalable, and secure data pipelines for ingesting, transforming, and validating clinical real-world data (RWD). • Implement software solutions using programming languages (e.g., Python, SQL) and cloud technologies (e.g., AWS). • Collaborate with software/quality engineers/analysts and product managers. • Communicate testing progress, risks, and quality metrics. • Analyze and fix defects identified within applications, services, or data pipelines. • Write SQL, Python, ELT code for querying clinical and genomic data. • Create data marts and perform data transformation and de-identification tasks. • Develop scalable data pipelines and processing workflows using AWS services. • Experience with Apache Airflow for workflow orchestration. • Apply version control (Git), CI/CD pipelines, and agile methodologies. • Support staff recruitment and onboarding, providing mentorship.
• Develop, maintain and evolve data ingestion and transformation pipelines following a medallion architecture; • Build and monitor integrations between the data lake and external applications (Octadesk, Salesforce and others); • Implement observability mechanisms for proactive failure detection, performance monitoring and cost control; • Collaborate with product, BI and engineering teams to understand needs and design scalable data solutions; • Document processes, pipelines and technical decisions to facilitate maintenance and onboarding of new team members.
Senior Data Engineer, Databricks
Codvo.aiBuilding Advance AI & Cloud Native Software Using The "Virtual Silicon Valley" Model. Let’s Talk AI, Cloud and Outcomes.
• Design, develop, and maintain scalable data pipelines using Databricks and Python • Build and optimize ETL workflows for structured and unstructured data • Work with Databricks Lakehouse architecture and implement best practices • Manage and implement data governance using Unity Catalog • Integrate AWS data services such as S3, IAM, VPC, and other relevant services • Collaborate with cross-functional teams to deliver high-quality data solutions • Build dashboards and support data visualization requirements • Ensure data quality, reliability, and performance optimization
• Build data pipeline, ELT and infrastructure solutions • Create automated monitoring, auditing and alerting processes • Work with business and application teams to implement data strategies • Design, develop, implement, and optimize ETL processes • Improve operations, monitoring and performance of data warehouse • Influence and communicate with stakeholders • Redesign and optimize existing data processes using AI • Live by and champion company values
