Proficiency with creating CI/CD pipelines and stages with Harness using YAML templates. Kong plugin development using Lua or Go. Experience with Package Managers like LuaRocks. Experience with containerization (Docker), and orchestration tools (Kubernetes). Experience with cloud platforms such as AWS, Google Cloud, or Azure. Familiarity with monitoring and logging tools like Datadog and OpenTelemetry.
GCP Data Engineer
Location
United States
Posted
9 days ago
Salary
0
Seniority
Mid Level
Job Description
GCP Data Engineer
Texas State Library and Archives Commision
Role Description We are looking for a Senior Data Engineer with strong hands-on experience in GCP, Cloud Composer, BigQuery, and Python. This is a fully remote US role. - Build and maintain scalable data pipelines using Cloud Composer (Airflow) - Develop high-performance data solutions in BigQuery - Write optimized Python code for data processing and automation - Support ETL/ELT workflows and ensure data quality - Collaborate with cross-functional teams to deliver cloud data solutions Qualifications - 7+ years of data engineering experience - Strong expertise in GCP, Composer, BigQuery, Python - Experience building large-scale data pipelines Requirements - Nice to Have: Snowflake experience - Nice to Have: Matillion experience - Location: Remote, USA (must be authorized to work in the U.S.)
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Create and maintain data pipelines for key data and analytics capabilities in the enterprise • Collaborate within an agile, multi-disciplinary team to develop optimal data integration and transformation solutions • Document and analyze data requirements (functional and non-functional) to develop scalable, automated, fault-tolerant data pipeline solutions for business and technology initiatives • Profile data to assess the accuracy and completeness of data sources and work with business partners to mitigate issues • Build and maintain data pipelines using appropriate tools and practices in development, test, and production environments • Design with modularity to leverage reuse of code wherever possible • Create data mappings, programs, routines, and SQL to acquire data from legacy, web, cloud, and purchased package environments into the analytics environment • Use a mix of ELT, ETL, data virtualization, and other methods to optimize the balance of minimal data movement against performance • Maintain metadata management processes and documentation • Monitor data quality to detect emerging issues and consult with the team to create transformation rules to cleanse against defined rules and standards • Participate in code reviews and unit testing to optimize performance and minimize issues.
Senior Data Engineer, Microsoft Fabric
VigilWe’re a tech consultancy, expert in software engineering and cloud transformation.
• Design, build and maintain enterprise data solutions using Microsoft Fabric • Develop scalable data ingestion pipelines from APIs, databases and file-based sources • Build and maintain Lakehouse architectures using the Medallion (Bronze, Silver and Gold) approach • Develop robust data models and semantic layers to support business reporting • Create and maintain Power BI semantic models and dashboards • Implement data quality, validation and reconciliation frameworks • Design secure, governed data solutions including Row-Level Security and access controls • Build and maintain PySpark notebooks for data transformation • Develop SQL and Python-based ETL/ELT processes • Configure and maintain Fabric workspaces, OneLake environments and deployment pipelines • Work closely with solution architects to implement scalable platform designs • Produce technical documentation and engineering standards • Collaborate with UK-based stakeholders and delivery teams • Support future client projects across the TXP Data & AI Practice
Data Engineer – Data Conversions, Junior-Mid
EnrouteWe deliver IT services and solutions provided by a team of passionate problem solving individuals highly skilled.
• Play a critical role in migrating insurance policy data from administration systems to a modern policy administration platform. • Perform analysis-first role with a data engineering component. • Deep-dive into data, trace and reconcile information across multiple data layers (databases and files). • Build mappings and transformations that move legacy data into the target format. • Resolve data quality issues to deliver validated loads. • Utilize strong, hands-on command of SQL for analysis, tracing, and transforming data. • Use AI tooling to accelerate work processes. • Receive mentorship from an experienced conversion manager. • Work through entire data migration lifecycle, from initial discovery through production cutover. • Support iterative delivery cycles with effective communication.
Arquitecto de Datos
Sofka TechnologiesTo transform people’s lives being the most trusted technology partner
• Definir la estrategia analítica y el modelo funcional de la solución. • Diseñar y construir el pipeline de ingesta y transformación sobre AWS. • Configurar la seguridad de la infraestructura (IAM). • Diseñar el modelo dimensional (star schema). • Actuar como autoridad técnica frente al cliente.



