Lead Data Engineer

Data EngineerData EngineerFull TimeRemoteSeniorTeam 10,001+H1B SponsorCompany SiteLinkedIn

Location

Florida

Posted

2 days ago

Salary

$127K - $236.5K / year

Seniority

Senior

Bachelor Degree9 yrs expExperience acceptedEnglishAzureETLJavaScriptPandasPySparkPythonSDLCSparkSQLSSISTypeScript

Job Description

Lead Data Engineer

L3Harris Technologies

• Provide architectural oversight for the company’s data platform and related initiatives, ensuring alignment with company standards and best practices • Assist development initiatives with design reviews, offering guidance on architecture and data modeling • Lead design reviews for proposed solutions, including data sourcing, data pipelines, data models, and ontology resources, providing actionable feedback to enhance design scalability • Assist and develop architectural artifacts, perform requirements decomposition, and develop other documentation to support the software development lifecycle (SDLC) • Support the SDLC process by participating in data architecture review sessions, reviewing proposed solutions for compliance with architectural standards • Oversee and provide guidance for data pipelines, ensuring scalability, reusability, and performance • Utilize Palantir Foundry to perform data integration and ontology management to support big data analytics at scale • Collaborate with cross-functional teams to define data governance, metadata, and ontology standards, ensuring consistency and clarity across the data platform and enterprise • Remain current with emerging data technologies and frameworks to recommend improvements to the data platform architecture.

Job Requirements

  • Bachelor’s Degree and minimum 9 years prior Palantir experience or Graduate Degree and a minimum of 7 years of prior Palantir experience
  • In lieu of degree, minimum 13 years of prior Palantir experience
  • Minimum of 4 years of experience with Data Pipeline development or ETL tools such as Palantir Foundry, Azure Data Factory, SSIS, or Python
  • Minimum of 4 years of experience in Data Integration
  • Preferred Additional Skills: Experience supporting or implementing solutions that leverage Generative AI capabilities, including Retrieval-Augmented Generation (RAG), semantic search, and LLM integration
  • Experience with Palantir Foundry tools such as Ontology Manager, Compass, Code Repository, Solution Designer, OSDK, AI FDE, and AIP Analyst
  • Understanding of BI (Business Intelligence) & DW (Data Warehouse) development methodologies
  • Experience with Python, Pandas, Databricks, JavaScript, Typescript or other scripting languages
  • Experience with AI tools such as OpenAI, Palantir AIP, Snowflake Cortex, or similar
  • Hands on Experience with design, development of Data Pipelines in Palantir Foundry Pipeline Builder or Code Repository, or similar technologies leveraging PySpark and Spark SQL
  • Familiarity with SDLC processes and architecture review or data governance board processes.

Benefits

  • Health insurance
  • Disability insurance
  • 401(k) match
  • Flexible spending accounts
  • Employee Assistance Program (EAP)
  • Education assistance
  • Parental leave
  • Paid time off
  • Company-paid holidays

Related Categories

Related Job Pages

More Data Engineer Jobs

Appfire logo

Senior Data Engineer - Data Insights

Appfire

Appfire is a computer software company that has been recognized as a “global authority” on Atlassian products and software. As an employer, the company aims

Data Engineer2 days ago

Role Description Data is the fuel behind Appfire's global footprint. We’ve grown fast, and now our focus has shifted toward building the most mature, high-performing data platform in our industry. We are looking for a Senior Data Engineer to take ownership of our core infrastructure, champion engineering excellence, and help shape the future of our data ecosystem. Leveraging your expert Python, SQL, and cloud infrastructure skills, you will develop pragmatic architectures that ensure stability and performance across the organization. As a key member of our central data insights team, you will evolve our custom data lake ("firelake" built on AWS and Snowflake). To accelerate our platform optimization efforts, you will ruthlessly eliminate technical debt, resolve edge cases, and introduce modern, AI-driven development practices. This isn't just about maintaining pipelines; it's about architecting a highly secure, resilient, and scalable foundation that empowers company-wide decision-making. If you love solving complex data puzzles, driving overarching data strategy, and seeing the direct impact of your work across an entire organization, let's talk. Qualifications - 5+ years in a data or software engineering role with a deep understanding of the full data lifecycle, modern data warehousing, and agile software development best practices. - 5+ years of experience autonomously building scalable data products, pipelines, and solutions that support company-wide systems and overarching business goals. - Exceptional personal organization and multitasking skills, with the ability to work with minimal supervision while driving 3-5 concurrent cross-departmental projects. - 5+ years of advanced SQL optimization and complex ETL/ELT pipeline development, with extensive, hands-on experience in Snowflake. - 5+ years of advanced Python programming. - 5+ years of experience designing and maintaining AWS cloud environments. Strong, hands-on proficiency with Terraform and Docker is required. - Deep operational experience with Airflow for orchestration, dbt (3+ years) for transformation, and familiarity with managing Fivetran pipelines. - Proven ability to design pragmatic, cost-effective architectures from scratch that prioritize security, scalability, and high performance without over-engineering. - Incredibly strong troubleshooting skills with a ruthless dedication to reducing technical debt, optimizing CI/CD pipelines, and enforcing strict version control and testing standards. - Solid understanding of modern data governance principles, including automated data classification, PII masking, and row/column-level access controls. - Strong written and verbal skills, with a proven ability to translate and simplify complex technical architectures to both engineering peers and non-technical business stakeholders. - A team-oriented mindset with a passion for coaching and providing constructive feedback to junior team members. Requirements - Own the complete solution development lifecycle, from driving initial requirements to building Proof of Concepts (POCs) and Minimum Viable Products (MVPs), and continuously iterating to improve and scale production solutions. - Work closely with internal and external partners to stand up and optimize robust data delivery solutions. Build, manage, and scale complex data pipelines (ETL and reverse ETL) utilizing expert-level Python and Airflow. - Develop standard integration patterns as well as custom data pipelines that interface with atypical APIs to efficiently extend our data delivery capabilities. - Architect and maintain highly efficient, cost-effective scalable data solutions and cloud infrastructure from scratch. - Be the key driver in the evolution of our custom data lake ("firelake" built on AWS/Snowflake), serving as the ultimate subject matter expert on our tech stack and data infrastructure. - Manage Fivetran and custom pipeline operations, proactively handle schema drift, and implement rigorous data quality checks to ensure we have ingested a true and accurate representation of all source systems. - Spearhead AI adoption in our daily engineering workflows to automate Pull Requests, enforce new "definition of done" standards, and accelerate CI/CD, test coverage, and release management. - Drive the integration of AI capabilities directly into our data solutions by interfacing with LLMs and developing intelligent assistants and agents to solve complex business problems. - Stay on the forefront of the rapidly evolving data landscape and drive proof-of-concepts for new tools to ensure we leverage best-in-class technologies to scale quickly. - Implement advanced PII management by leveraging automated data classification techniques, applying masking policies, and enforcing sophisticated row-level and column-level security practices. - Develop and deploy robust monitoring solutions and governance strategies to ensure accurate, secure data is available on time and to the correct audience. - Autonomously lead 3-5 concurrent cross-departmental data projects, driving everything from requirements gathering and UAT to production deployment. - Partner seamlessly with business, analytics, and engineering teams, translating complex technical architectures into relatable concepts to influence stakeholders and align goals. - Champion best practices, systematically reduce technical debt, write clear documentation, and provide both technical and soft-skill mentorship to junior team members. Benefits - Work in a product-first SaaS B2B environment focused on building and scaling solutions for large enterprise customers. - Deliver meaningful, high-impact work that directly influences product direction and customer success. - Every Appfire team member is eligible for company equity, fostering a true sense of ownership and connection to our growth. - Employees receive 25 days of paid time off per year. After 5 years of employment, employees receive 30 days per year. - Appfire provides comprehensive coverage through Empire Health, including medical, dental, and vision insurance. Additional protection includes life insurance, critical illness coverage, accidental death and dismemberment, and long-term disability insurance. - To promote health and well-being, Appfire offers a yearly sport allowance of CAD 600, which can be used to cover the cost of various fitness or outdoor activities from gym memberships to sports equipment. - Every employee receives 3 fully paid days each year to participate in Appfire Town, our Corporate Social Responsibility (CSR) program supporting local communities.

Worldwide

Data Engineer III

itD Tech

About itD: We are part of a new generation of consulting and software development company that blends diversity, innovation, and integrity with real business results. Our structure rejects any strong hierarchy, empowering us to deliver excellent results. We are a woman- and minority-led firm. Every day, we challenge ourselves to be considerate, fair and to re-think what great outcomes mean for our customers. This permeates down to how we approach every interaction, on every project, for every client. You’ll thrive here if you are a dynamic self-starter, a difference-maker or someone who wants to deliver great results, without constraints. The itD Digital Experience: Joining us means you’ll be part of our global community, you have a say about your own career journey, and you’ll get a chance to give back to causes that matter. You will experience working with Fortune 500 companies and high-performance teams across numerous industries. itD offers our employees excellent benefits such as medical, dental, vision, life insurance, paid holidays, 401K + matching.

Data Engineer2 days ago

Role Description itD is seeking a Data Engineer III to design, build, and maintain scalable data architectures and enterprise data solutions that support business operations and enable data-driven decision-making. The ideal candidate will bring strong expertise in data engineering, ETL development, and AI-assisted coding tools, with a proven track record of delivering reliable, high-performance data pipelines and scalable data management systems. Location: Remote (Must be available to work PST hours) Duration: 5 Months Pay Rate: Competitive hourly W2 rate, depending on experience. Please note that we are only considering direct W2 candidates at this time, as we are unable to offer sponsorship. Responsibilities - Design, develop, test, and maintain highly scalable enterprise data management systems. - Build, implement, automate, and optimize large-scale ETL processes and data pipelines. - Develop high-performance algorithms, prototypes, predictive models, and proof-of-concept solutions. - Ensure data architectures meet business requirements, performance expectations, and industry best practices. - Collaborate with data architects, modelers, and cross-functional IT teams to deliver data engineering initiatives. - Evaluate, maintain, and enhance existing data solutions to improve performance, reliability, and scalability. - Leverage AI coding tools to improve development efficiency and accelerate solution delivery. Internal Responsibilities - Attend regular internal practice community meetings. - Collaborate with your itD practice team on industry thought leadership. - Complete client case studies and learning material (blogs, media material). - Build out material to contribute to the Digital Transformation practice. - Attend internal itD networking events (in person and virtual). - Work with leadership on career fast-track opportunities. Qualifications - 6–7 years of professional experience in Data Engineering. - Experience designing and maintaining scalable data architectures and enterprise data management systems. - Strong experience developing and maintaining large-scale ETL processes and data pipelines. - Experience building algorithms, predictive models, and proof-of-concept solutions. - Strong understanding of AI coding tools and AI-assisted software development. - Ability to work independently with minimal supervision while managing multiple priorities. - Proficiency with Microsoft SharePoint. - Strong written, verbal, and presentation communication skills. Preferred Qualifications and Skills - Experience collaborating with data architects, data modelers, and cross-functional IT teams. - Experience working in large enterprise environments. - Self-driven with strong problem-solving and analytical skills. - Experience delivering scalable, high-performance data solutions. - Familiarity with modern AI-assisted development practices. Education - Bachelor's degree in Computer Science, Computer Engineering, or a related technical field required. Company Description About itD: We are part of a new generation of consulting and software development company that blends diversity, innovation, and integrity with real business results. Our structure rejects any strong hierarchy, empowering us to deliver excellent results. We are a woman- and minority-led firm. Every day, we challenge ourselves to be considerate, fair and to re-think what great outcomes mean for our customers. The itD Digital Experience: Joining us means you’ll be part of our global community, you have a say about your own career journey, and you’ll get a chance to give back to causes that matter. You will experience working with Fortune 500 companies and high-performance teams across numerous industries. itD offers our employees excellent benefits such as medical, dental, vision, life insurance, paid holidays, 401K + matching, networking & career learning and development programs. We are growing and we want to see you grow! Visit https://itdtech.com/careers to learn more about what working at itD can mean for you. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability or protected veteran status, or any other legally protected basis, in accordance with applicable law. itD is committed to working with and providing reasonable accommodation to individuals with disabilities. If, because of a medical condition or disability, you need a reasonable accommodation for any part of the application process, or to perform the essential functions of a position, please contact us at recruiting@itdtech.com and let us know the nature of your request and your contact information. Dynamic environment in a culture of respect, empowerment and recognition for a job well done, apply today!

PST (UTC-8)
PEN2 / hour

Data Engineer / Data Modeler

Tieto

Tieto Tech Consulting (MentorMate from 2001 to 2026) provides design-led, data-centric, and AI-powered digital engineering & consulting services to enterprises worldwide.

Data Engineer2 days ago

Role Description We are looking for a Data Engineer/ Data Modeler to support the design and implementation of a modern Azure-based data platform. The ideal candidate combines strong hands-on data engineering skills with the ability to work closely with business stakeholders, structure unclear requirements, and translate them into scalable data models. The role is especially suitable for someone with experience in: - Azure Synapse - Azure Data Factory - Python/ PySpark - SQL - SAP source systems - Data warehousing - Power BI reporting enablement Key Responsibilities - Analyze and clarify business requirements from scattered or incomplete documentation - Work directly with business departments to align requirements, data definitions, and reporting needs - Translate business requirements into conceptual, logical, and physical data models - Design and implement data models for: - Silver layer: normalized / 3NF model - Gold layer: dimensional / star schema model - Build and maintain medallion architecture in Azure Synapse, including silver and gold tables - Develop data transformations and pipelines using Azure Synapse, Azure Data Factory, SQL, Python/ PySpark - Integrate data mainly from SAP source systems - Support data cataloging, governance, and lineage through Microsoft Purview - Use Azure DevOps for repository management, CI/CD, and project tracking - Enable reporting and analytics through well-structured datasets for Power BI Qualifications - Strong experience with Azure cloud data platforms - Hands-on experience with: - Azure Synapse - Azure Data Factory - Python/ PySpark - SQL - Data Lake/ Lakehouse architectures - Solid understanding of data modeling: - 3NF / normalized models - Star schema/ dimensional modeling - Data warehouse design - Experience working with SAP source systems - Ability to translate business requirements into technical data models - Experience with Git-based development and CI/CD, ideally in Azure DevOps - Good understanding of reporting needs, ideally with Power BI - Strong communication skills and ability to work with both business and technical stakeholders Requirements - A significant advantage would be: - Experience with Microsoft Purview - Experience in enterprise reporting, forecasting, or analytics platforms - Background in SAP-heavy enterprise environments - Experience with agile delivery and tools such as Jira/ Azure DevOps - Microsoft data engineering certification is a plus Benefits - Diversity, equity and inclusion - Sustainability Additional Information At Tieto, we believe in the power of diversity, equity, and inclusion. We encourage applicants of all backgrounds, genders (m/f/d), and walks of life to join our team, as we believe that this fosters an inspiring workplace and fuels innovation. Our commitment to openness, trust, and diversity is at the heart of our mission to create digital futures that benefit businesses, societies, and humanity. Stay alert to recruitment scams: - Tieto only communicates via @tieto.com email addresses. - We never ask candidates to pay fees or share financial details during recruitment. - If you receive a suspicious message claiming to be from Tieto, don't respond — verify it via our official careers site. - For more information visit Important Advisory Regarding Recruitment Fraud | Tieto Blog Remote Type Remote Job Area Application and Product Development Business Unit Tech Consulting

Worldwide

Big Data Engineer

Bright Vision Technologies

Bright Vision Technologies is a forward-thinking software development company dedicated to building innovative solutions that help businesses automate and optimize their operations. We leverage cutting-edge technologies to create scalable, secure, and user-friendly applications.

Data Engineer2 days ago

Role Description We are seeking an experienced Big Data Engineer to design, build, and operate large-scale data processing pipelines and analytics platforms on Hadoop and related big-data ecosystems. In this role you will be responsible for ingesting, transforming, and analyzing massive volumes of structured and unstructured data to support enterprise analytics, machine learning, and reporting workloads. The ideal candidate will combine deep technical expertise across the Hadoop ecosystem with strong software engineering fundamentals and a clear understanding of how to deliver reliable, performant, and cost-effective data platforms in production environments. Key Responsibilities - Design, develop, and operate end-to-end big-data pipelines on Hadoop, ingesting data from a diverse mix of relational, file-based, streaming, and API-driven sources. - Build robust ETL/ELT workflows using Apache Spark, Hive, Pig, and Sqoop, with strong attention to data quality, idempotency, error handling, and recoverability. - Develop high-throughput streaming data pipelines using Kafka, Spark Streaming, or Flink, and integrate them with downstream analytical and operational systems. - Optimize Spark and MapReduce jobs through careful tuning of partitioning, memory, serialization, and skew handling to meet demanding SLAs at minimal cost. - Design and maintain data models and storage layouts on HDFS, Hive, HBase, and modern lakehouse formats (Parquet, ORC, Delta, Iceberg, Hudi) to balance flexibility and performance. - Implement data governance, lineage, and quality controls in collaboration with data governance and security teams. - Build robust monitoring, alerting, and logging strategies for big-data pipelines, including job-level SLAs and proactive failure detection. - Partner with data scientists and analysts to deliver curated, reliable, and well-documented datasets that accelerate their work. - Automate pipeline orchestration using Airflow, Oozie, or similar workflow engines, with clean dependency management and clear ownership boundaries. - Continuously evaluate and adopt new technologies in the big-data and cloud ecosystem (EMR, Databricks, Snowflake, BigQuery) where they offer meaningful improvements. - Lead performance reviews and architecture audits of existing pipelines, proposing concrete refactoring and optimization initiatives. - Document data architectures, schemas, pipeline behaviors, and operational runbooks in a way that makes the platform supportable as the team scales. - Mentor junior engineers and contribute to the team’s engineering standards and best practices. Qualifications - Bachelor’s degree in Computer Science, Engineering, or a related technical discipline. - Five or more years of professional experience designing and operating big-data pipelines on Hadoop. - Strong hands-on expertise with Apache Spark (Scala, Python, or Java) in production environments. - Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem. - Hands-on experience with streaming data platforms such as Kafka, Spark Streaming, or Flink. - Strong SQL skills and experience working with both relational and NoSQL data stores. - Experience with workflow orchestration tools such as Airflow or Oozie. - Solid understanding of distributed systems concepts, including partitioning, replication, and fault tolerance. - Strong scripting skills in Python or Shell. - Excellent troubleshooting, debugging, and documentation skills. Preferred Qualifications - Experience operating Hadoop on cloud platforms such as AWS EMR, Azure HDInsight, or Databricks. - Familiarity with modern lakehouse formats (Delta, Iceberg, Hudi). - Exposure to data governance tooling such as Apache Atlas or Collibra. - Experience with Kubernetes-based data platforms (Spark-on-K8s, Trino). - Hands-on experience with CI/CD and infrastructure-as-code in data engineering workflows. How to Apply Would you like to know more about this opportunity? For immediate consideration, please send your resume to [email protected] Equal Employment Opportunity (EEO) Statement Bright Vision Technologies (BV Teck) is committed to equal employment opportunity (EEO) for all employees and applicants without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, veteran status, or any other protected status as defined by applicable federal, state, or local laws. This commitment extends to all aspects of employment, including recruitment, hiring, training, compensation, promotion, transfer, leaves of absence, termination, layoffs, and recall. BV Teck expressly prohibits any form of workplace harassment or discrimination. Any improper interference with employees' ability to perform their job duties may result in disciplinary action up to and including termination of employment.

United States
$100K - $150K / year