Global Workforce Solutions With A Human Touch. #Spearheadingfuture
Data Engineer
Location
United States
Posted
14 days ago
Salary
$76 - $83 / hour
Seniority
Mid Level
Job Description
Data Engineer
LanceSoft, Inc.
Role Description Data Analyst - Finance Data Platform - Purpose: Support data validation, data reconciliation, and reports development for the Finance mainframe application modernization and data platform migration to modern GCP-based data platform. - Outcome: Identify data variance between legacy and modern platforms for modern Finance applications and 2 quarter parallel operation. Collaborate with product and business teams to identify root causes for the variances and resolve/document the variance to assist on Go-Live decision. - Core Skill: Hands-on SQL experience with big data platforms/databases like Google BigQuery, Hadoop or similar data platform. Qualifications - Senior level (5+ years) hands-on SQL experience with big data platforms/databases like Google BigQuery, Hadoop or similar data platform. - Senior experience with ETL and data pipeline tools such as Dataproc, Airflow, Dataform, dbt, or similar tools. - Good experience with Python/Pyspark coding for data movement and analysis. - Experience with data visualization and reporting tools such as Power BI, Looker, and LookML for dashboard development, KPI reconciliation, semantic layer validation, and data analysis. - Strong experience in data analysis, data validation, source-to-target reconciliation, duplicate detection, reject/error validation, audit/control total validation, and business rule validation. - Ability to analyze mapping documents, reporting requirements, transformation logic, data lineage, and data quality rules. - Experience creating reconciliation reports, validation summaries, exception reports, defect logs, and business sign-off documentation. - Strong communication and collaboration skills with the ability to work closely with engineering, product, finance, business, and legacy system teams. Requirements - Develop queries and dataset required to validate legacy vs modern applications datasets to identify data variances. - Develop reports using end user visualization tools to present the variances and trends between sources. - Document data variances and identified root causes in collaboration with Product Manager/Engineers/Business Teams and get business sign off for known variances. - Build dashboards for data variance trend for the stakeholders and assist on determining tolerance thresholds for Go Live decisions. - 5+ years working in data analytics role. - 1-3 years of experience working in agile team. Benefits - Four options of medical Insurance - Dental and Vision Insurance - 401k Contributions - Critical Illness Insurance - Voluntary Permanent Life Insurance - Accident Insurance - Other Employee Perks
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Scientist Architect
Navtech, Inc.NAVTECH INC 1600 Golf Road. Suite 1200, Rolling Meadows, IL 60008 Ph: (224) 348-1340 Email: alex@navtechusa.com Website: www.navtechusa.com E-Verified Company
Role Description Designs and develops scalable solutions using AI tools and machine-learning models. - Performs research and testing to develop machine learning algorithms and predictive models. - Utilizes big data computation and storage tools to create prototypes and datasets. - Conducts model training and evaluation. - Integrates, tests, tunes, and monitors solutions. - Proficient with multiple AI tools such as Python, Java, or R and machine learning frameworks like Spark, TensorFlow, or scikit-learn. Requires a master's degree in computer science, mathematics, engineering or equivalent. Typically reports to a manager or head of a unit/department. - P05-Expert: Works autonomously. Goals are generally communicated in "solution" or project goal terms. - May provide a leadership role for the work group through knowledge in the area of specialization. - Works on advanced, complex technical projects or business issues requiring state of the art technical or industry knowledge. - Typically requires 10+ years of related experience. Qualifications - Master's degree in computer science, mathematics, engineering or equivalent. - 10+ years of related experience. Requirements - Proficiency in AI tools such as Python, Java, or R. - Experience with machine learning frameworks like Spark, TensorFlow, or scikit-learn. - Ability to work autonomously and lead a work group. - Expertise in advanced technical projects or business issues.
• Own the database consisting of Postgres and Supabase, managing schema design, modeling, and scaling performance. • Ensure data quality end-to-end via validation gates, deduplication, and entity resolution. • Oversee the communications layer, linking raw emails and call transcripts to the right people and companies. • Develop ingestion and enrichment pipelines for funding rounds and company research at scale. • Manage the knowledge graph comprising companies, investors, funding rounds, and news as entities and relationships. • Create a unified data layer that serves every campaign and product feature.
Role Description Forge requires a Mid Data Engineer to support legacy-to-modern data transformation in a secure AWS environment for a DoW customer. The role will develop batch and event-driven pipelines, automate data quality and testing, integrate with application services, and provide observable, recoverable, high-quality data flows across mission and external interfaces. Key Responsibilities - Build secure Python and AWS ETL/ELT pipelines for ingestion, transformation, reconciliation, and delivery. - Develop and evolve relational data models, schemas, indexes, constraints, views, and access patterns for MariaDB, PostgreSQL, or comparable platforms. - Develop data workflows and interfaces using Python on AWS Lambda and PySpark for event-driven, batch, and distributed transformation workloads. - Create automated data-quality checks for accuracy, completeness, consistency, timeliness, uniqueness, and business-rule conformance. - Implement source-to-target mapping, lineage, auditability, restartability, exception handling, and controlled replay. - Develop parity tests that compare legacy and modern processing outcomes and document the disposition of intentional differences. - Tune SQL and pipeline performance for high-volume batch and near-real-time workloads while protecting transactional integrity. - Implement monitoring, logging, alerting, and operational dashboards for pipeline health, latency, failures, and data quality. - Automate CI/CD, version-controlled data changes, deployments, rollback, and operational recovery controls. - Collaborate with architects, mission SMEs, Appian developers, testers, security personnel, and interface partners. Qualifications - Ability to think strategically, act tactically, and demonstrate strong analytical and critical-thinking skills. - Build strong cross-group working relationships and demonstrate exceptional organizational skills and attention to detail. - Thrive and succeed in an entrepreneurial environment and not be hindered by ambiguity or competing priorities. - Self-managing candidates who enjoy working collaboratively in a fast-paced environment and with dynamic teams. Requirements - U.S. Citizen (Authorization to Work in the U.S. will not suffice); previous professional experience supporting the U.S. Federal Government, either as a federal employee or contractor, is required. - 4+ years of professional experience in data engineering, database development, or data-platform delivery. - Bachelor's degree in Computer Science, Information Systems, Data Engineering, or equivalent, OR 4 additional years of relevant professional experience in lieu of a degree. - Advanced Python software-engineering skills and experience building AWS Lambda functions and PySpark data-transformation pipelines. - Experience building, testing, and operating production ETL/ELT pipelines with automated data-quality controls. - Experience with data modeling, schema migration, source-to-target mapping, lineage, reconciliation, and data-quality automation. - Experience integrating data platforms with REST APIs, application services, file exchanges, and event-driven interfaces. - Experience with Git, CI/CD, automated testing, logging, monitoring, performance tuning, and production support. - Active CompTIA Security+ or equivalent DoW-approved baseline cybersecurity certification, or ability to obtain within the first 30 days of starting. - Active Tier 2 background investigation or higher, completed or favorably adjudicated within the previous 18 months. Highly Desired Qualifications - Experience using Palantir Foundry for data integration, transformation, lineage, governance, and operational workflows. - Active Secret security clearance preferred. - Previous professional experience supporting a DoW organization, mission, or customer is strongly preferred; experience in modernizing COBOL flat files or legacy relational data into a modern relational architecture. - Experience integrating with Appian, Python microservices, financial transactions, logistics workflows, or high-volume external interfaces. - Experience serving as a technical team lead, mentoring junior engineers, or assisting teammates across delivery tasks. - Experience with BI, analytics, archival, records-retention, or NARA-aligned data lifecycle requirements. Benefits - Complete Flextime - 401k With Employer Matching - Healthcare, Including Medical, Dental, and Vision - Health Savings Account (HSA) And Pre-Tax Premium Options - Supplementary healthcare and family support - Extended Short-Term Disability and Long-Term Disability - Healthcare Insurance Deductible Paydown - Health and Wellness Programs - Tuition Reimbursement, Student Loan Repayment, and Education & Training Stipends - Cell Phone / Internet Stipends - College Saving Plans with Employer Contributions - Alternative Work Locations and Tele-Commuting - Employee Referral Awards - Retention, Signing & Performance Bonuses - Commuter Benefits - Paid Sabbatical
Senior Data Engineer, Microsoft Fabric Engineer
Weekday (YC W21)We are a Y-Combinator-backed startup building your AI-powered Recruiter Agent
• Design, develop, and maintain scalable data pipelines using Microsoft Azure Fabric, Databricks, and Azure data services. • Build and optimize robust ETL/ELT processes for structured, semi-structured, and unstructured data across enterprise environments. • Collaborate with business stakeholders to understand data requirements and translate them into scalable technical solutions. • Develop and maintain reliable data integration workflows that ensure data accuracy, consistency, and availability. • Optimize data processing pipelines for performance, scalability, cost efficiency, and operational reliability. • Leverage AI-assisted development tools and modern engineering practices to accelerate solution delivery and improve code quality. • Monitor, troubleshoot, and resolve production data pipeline issues while proactively identifying opportunities for automation and optimization. • Work closely with architects, analysts, developers, and cross-functional teams throughout the project lifecycle to deliver high-quality data solutions. • Implement best practices for data engineering, governance, documentation, testing, and operational support. • Take ownership of project deliverables by ensuring quality, meeting timelines, communicating risks proactively, and continuously improving engineering processes.


