AgileEngine logo
AgileEngine

AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.

Data Engineer

Data EngineerData EngineerFull TimeRemoteMid LevelTeam 1,001-5,000

Location

Worldwide

Posted

2 days ago

Salary

0

Seniority

Mid Level

Job Description

Data Engineer

AgileEngine

Role Description We are looking for a Data Engineer to support a strategic data platform migration, moving tables, views, and stored procedures from Amazon Redshift to Databricks. You will build and support Databricks pipelines and jobs, work with AWS services including S3, Glue, and Athena, and apply data governance practices including data lineage, cataloging, and role-based access controls throughout the migration. This is a 5–6 month project engagement. What You Will Do - Migrate tables, views, and stored procedures from Amazon Redshift to Databricks. - Build and support Databricks pipelines and jobs. - Work with AWS services, including Amazon S3, AWS Glue, and Amazon Athena. - Support data migration activities within a modern data lake architecture. - Apply data governance and data management practices throughout the migration. - Collaborate with project stakeholders to support migration activities. Qualifications - 3–5 years of experience as a Data Engineer. - Hands-on experience with Databricks. - Experience with Amazon Redshift. - Strong SQL skills. - Experience with AWS services, including Amazon S3, AWS Glue, and Amazon Athena. - Experience building and supporting Databricks pipelines and jobs. - Understanding of data lake architectures. - Familiarity with data governance and data management practices, including data lineage, data catalogs, and role-based access controls (RBAC). - Upper-intermediate English level. Nice to Haves - Experience with Databricks Lakebridge transpiling capabilities. Benefits - Growth without limits: build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget. - Competitive compensation: get recognition that reflects your skills and impact, with regular performance and compensation reviews. - Flexibility: work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm. - Meaningful, modern projects: build impactful products using modern technologies alongside global teams and leading brands. - Collaborative culture: join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized. - Well-being & support: access local well-being programs and people-focused support tailored to your location.

Related Categories

Related Job Pages

More Data Engineer Jobs

Medisolv, Inc. logo

Data Engineer

Medisolv, Inc.

Purpose-Built Quality Management Software for Hospitals and Physicians

Data Engineer2 days ago
Full TimeRemoteTeam 51-200Since 1999

• Work closely with Medisolv clients to extract, transform and load clinical, financial and administrative healthcare data and health plan data. • Assist in loading and troubleshooting data issues from EHR vendor products (including but not limited to: Epic, Meditech, Cerner and EClinicalWorks). • Support CMS quality measures (HEDIS/Stars) and fuel downstream AI workflows. • Integrate with the data management team and become familiar with workflows and tools in Azure and AWS environments (Snowflake, Databricks, Data Factory, Glue, Lambda, SQL Server). • Review existing data pipelines, data models, and transformation scripts to support troubleshooting. • Investigate technical and data-related issues affecting measure results and mapping (e.g., SNOMED, ICD-10, RXNORM). • Apply SQL querying skills to investigate data anomalies within Databricks and Snowflake environments. • Troubleshoot data pipelines to support internal operations and client deliverables. • Perform data analysis to identify trends, discrepancies, and opportunities for improvement. • Collaborate with cross-functional teams to gather requirements and enhance data quality and reporting outputs. • Develop and deliver dashboards, reports, and visualizations that support business and client decisions. • Monitor data quality, troubleshoot issues, and drive improvements in data integrity and consistency. • Partner with stakeholders to translate data into actionable insights aligned with business goals. • Lead enhancements to data architecture, reporting capabilities, and data processes. • Support advanced analytics initiatives, including predictive insights and AI-driven solutions. • Document processes and establish best practices to ensure scalability, reliability, and consistency as the organization grows.

Tennessee
Stellus Rx logo

Senior Data Engineer

Stellus Rx

Trusted, pharmacist-led health support in every moment that matters.

Data Engineer2 days ago
Full TimeRemoteTeam 201-500Since 2022

• Develop, construct, and maintain large-scale data processing systems that collect data from a variety of structured and unstructured sources — using AI code generation tools to accelerate pipeline authoring, reduce boilerplate, and improve code quality. • Build and optimize ELT pipelines using AI-assisted tooling to identify bottlenecks, suggest optimizations, and automate routine pipeline maintenance tasks. • Identify, design, and implement internal process improvements: use AI to automate manual processes, optimize data delivery, and re-design infrastructure for greater scalability — replacing manual analysis with AI-driven discovery of improvement opportunities. • Build the infrastructure required for optimal extraction, transformation, and loading of data from various sources; use AI to accelerate infrastructure-as-code authoring and configuration. • Prepare data for data scientist exploration and discovery using AI-assisted data profiling and quality assessment tools — surfacing anomalies, schema drift, and data gaps faster than manual inspection allows. • Perform data wrangling and munging for downstream analytics and machine learning; leverage AI tools to generate and validate transformation logic against business rules. • Assemble large, complex datasets that meet functional and non-functional business requirements; use AI to rapidly evaluate dimensional modeling approaches and ontology alignment strategies. • Enable large-scale machine learning by designing and maintaining annotated datasets, elastic search approaches, and scalable data lake structures that support AI/ML workloads. • Create and maintain analytics pipelines that generate data and insight to power business decision-making; use AI-assisted analysis to proactively surface trends, anomalies, and opportunities within pipeline outputs. • Collaborate with data scientists, analysts, and business stakeholders on requirements for dimensional modeling, distributed ETL pipelines, and cross-repository data migration. • Evaluate, compare, and improve design patterns, data lifecycle approaches, and data ontology alignment — using AI to model trade-offs and accelerate proof-of-concept validation. • Work with data and analytics experts to continuously improve the functionality, reliability, and intelligence of data systems. • Perform root cause analysis on internal and external data and processes using AI-assisted investigation tools — replacing slow, manual log and lineage review with faster, AI-accelerated diagnostics. • Develop and maintain data quality frameworks; use AI to automate anomaly detection, schema validation, and data contract enforcement across pipelines. • Develop a strong understanding of company domains, strategic direction, and user needs to ensure data systems are aligned to business outcomes, not just technical requirements.

Peru
Stellus Rx logo

Senior Data Architect

Stellus Rx

Trusted, pharmacist-led health support in every moment that matters.

Data Engineer2 days ago
Full TimeRemoteTeam 201-500Since 2022

• Define and maintain enterprise data architecture standards across structured, semi-structured, and unstructured data domains. • Use AI-assisted modeling tools to accelerate data model design and validate designs against business requirements. • Design and govern the organization's cloud data lake, data warehouse, and lakehouse architectures on AWS. • Establish data ontology, taxonomy, and semantic layer standards that enable AI systems to reason over organizational data accurately. • Evaluate emerging data architecture patterns and build a roadmap for their adoption across Stellus Rx. • Design scalable data models and ELT/ETL pipeline architectures that support analytics and AI/ML workloads. • Define standards for data partitioning, indexing, caching, and storage optimization. • Partner with Data Engineers to translate architectural blueprints into production-ready pipelines. • Define and enforce data governance frameworks, data quality standards, and data contracts across the enterprise. • Develop and maintain a master data management (MDM) strategy.

Peru
interVal logo

Data Engineer

interVal

The Visibility Engine to Grow AUM in Less Time

Data Engineer2 days ago
Full TimeRemoteTeam 11-50Since 2019

• Design, develop, and maintain scalable data pipelines for ingestion, transformation, and delivery of large datasets across diverse industries. • Implement and ensure data privacy and security best practices, supporting data sovereignty and compliance with regulatory requirements. • Collaborate closely with AI/ML engineers, Data Scientists, and Platform engineers to enable advanced analytics and AI capabilities while retaining strict data control. • Optimize data platforms and systems for performance, reliability, and cost efficiency. • Build tools and frameworks for secure, privacy-preserving data processing and orchestration. • Develop and maintain documentation, data models, and technical workflows. • Partner with cross-functional teams to launch new data-driven product features and solutions.

United States