MAGIC serves as the centralized ERP system for over 120 State agencies, supporting: Financial Management Procurement Grants Management Reporting These modules have been in production since 2015. The State is also currently implementing SAP SuccessFactors for: Human Capital Management (HCM) Payroll Processing
Big Data/ Hadoop admin
Location
United States
Posted
1 day ago
Salary
0
Seniority
Mid Level
Job Description
Big Data/ Hadoop admin
Ruri Software Technologies LLC
Role Description We are seeking a highly experienced Senior Big Data / AWS Administrator with 10+ years of hands-on experience in managing large-scale distributed systems. The ideal candidate will have deep expertise in AWS cloud infrastructure, AWS MSK, Hadoop ecosystem, and Kafka administration, ensuring high availability, scalability, and performance of enterprise data platforms. W2 POSITION Key Responsibilities - Cloud & Infrastructure (AWS) - Design, deploy, and manage scalable, secure, and highly available infrastructure on AWS MSK. - Administer core AWS services: EC2, S3, EMR, EKS, IAM, VPC, CloudWatch, Lambda. - Implement automated infrastructure provisioning using Infrastructure as Code (Terraform/CloudFormation). - Optimize cost, performance, and reliability of cloud environments. - Develop disaster recovery and backup strategies. - Big Data Platform Administration (Hadoop) - Install, configure, tune, and maintain Hadoop ecosystem components (HDFS, YARN, Hive, HBase, Spark). - Monitor cluster health, performance, and capacity planning. - Troubleshoot cluster issues and ensure high availability using HA configurations (NameNode HA, etc.). - Manage data ingestion and processing workflows. - Streaming Platform Administration (Kafka) - Install, configure, and manage Apache Kafka clusters. - Ensure high throughput, low latency, and fault-tolerant streaming systems. - Manage topics, partitions, brokers, and retention policies. - Perform Kafka cluster scaling, upgrades, and performance tuning. - Implement monitoring using tools like Prometheus, Grafana, or Confluent Control Center. - System Operations & Monitoring - Implement monitoring, logging, and alerting for all systems. - Conduct root cause analysis and performance tuning. - Ensure system security, access controls, and compliance standards. - Maintain detailed documentation and operational runbooks. - Automation & DevOps - Build CI/CD pipelines for infrastructure and platform deployments. - Automate routine administrative tasks using scripting (Python, Shell, etc.). - Collaborate with DevOps and engineering teams for platform improvements. Qualifications - Minimum 10+ years of experience in system administration, with at least 5+ years in big data and cloud environments. - Proven experience managing large-scale Hadoop clusters and Kafka environments. - Strong hands-on experience with AWS cloud platform. - Agentic AI experience. Technical Skills - AWS services: AWS MSK, AWS Administration, EC2, S3, EMR, IAM, VPC, CloudWatch, Lambda. - Big Data: Administration, Hadoop, HDFS, Hive, Spark, Kafka Administration.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Principal Engineer - Data Platform
ServiceNowAs the AI platform for business transformation, we're putting AI to work across organizations — freeing people for work that matters. Making old tech work with new tech. Reaching across departments, from the front office to the back office and every office in between. Our ambition? To become the AI defining enterprise software company of the 21st century (or "AI DESCO21C," as we like to call it). With more than 8,400+ customers, we serve approximately 90% of the Fortune 500®, and we're proud to be a Fortune 100 Best Companies to Work For® and World's Most Admired Companies™. Explore your future career with us, visit www.careers.servicenow.com From Fortune. ©2026 Fortune Media IP Limited. All rights reserved. Used under license.
Role Description As Principal Engineer for Data Platform Modernization, you will be the foundational architect for the GCS Data Warehouse and everything upstream of the query layer: - Design the GCS Data Warehouse, the modern lakehouse foundation (Trino, Iceberg, dbt, a modern catalog) that replaces the existing Cloudera-based platform. - Lead the one-year program to move GCS data off Cloudera (Impala, Hive, HDFS, Hive Metastore) so the organization can decommission it rather than renew. - Design one portable architecture that deploys independently into commercial and regulated environments. - Own the ingestion architecture: change data capture from primary source systems, transactional PostgreSQL databases, landed into Iceberg. - Own the streaming layer that carries changes, assessing Kafka and deciding whether to keep or replace it. - Define the data and schema translation approach: Hive and Impala schemas onto Iceberg tables. - Set the correctness bar: reconcile new outputs against the source platform as ground truth. - Design data governance and security on the lakehouse. - Help design the platform's operational model: SLOs, observability, runbooks, and on-call approach. - Establish engineering standards for reliability, determinism, observability, and production readiness. - Lead through influence, aligning engineers to the target architecture. - Navigate enterprise constraints, security, compliance, and approval processes. - Drive the responsible use of AI and ML tooling to accelerate migration, translation, and validation work. Qualifications - 15+ years of experience in software or data engineering. - Proven experience leading a large data platform migration or modernization off a legacy Hadoop or Cloudera stack. - Experience architecting for regulated or government cloud environments. - Deep expertise across the modern data stack (Trino/Presto, dbt, Apache Iceberg). - Hands-on experience designing streaming ingestion and change data capture. - Working knowledge of streaming platforms (Apache Kafka or comparable). - Proven track record as the lead architect or top technical authority for a platform. - Strong systems and backend engineering depth. - Demonstrated ability to operate at high velocity in greenfield environments. - Strong knowledge of data structures, algorithms, data modeling, design patterns, and performance optimization. - Full professional proficiency in English. Requirements - Experience leveraging or critically thinking about how to integrate AI into work processes. - Deep understanding of software quality principles including reliability, determinism, observability, security, and production readiness. - Ability to troubleshoot and reason about complex distributed systems. - Comfort with development tools such as IDEs, debuggers, profilers, source control, and Unix-based systems. Benefits - Base pay of $221,200 - $387,100, plus equity (when applicable), variable/incentive compensation, and benefits. - Health plans, including flexible spending accounts. - 401(k) Plan with company match. - ESPP and matching donations. - Flexible time away plan and family leave programs.
• Design and build ETL/ELT data pipelines to ingest, process, and transform large datasets from multiple sources • Implement best practices for performance tuning, partitioning, and clustering to optimize data queries and reduce costs • Establish and enforce data quality standards, data governance frameworks, and security policies for data storage and access • Develop and optimize data models and schemas to support analytics, reporting, and machine learning requirements • Collaborate with data scientists and analysts to design data solutions that integrate with BI tools and machine learning models • Create comprehensive documentation for data pipelines, workflows, and processes • Share best practices and mentor junior data engineers
Role Description Buscamos uma pessoa Engenheira de Dados Sênior para atuar no desenvolvimento de projetos de dados em ambiente Azure e Databricks. Procuramos um perfil técnico, proativo e hands-on, capaz de compreender as necessidades do negócio, transformá-las em soluções técnicas, apoio no desenho da arquitetura e acompanhar a implementação junto à squad. - Levantar requisitos e traduzir necessidades de negócio em soluções técnicas; - Participar da Elaboração de desenhos de arquitetura de dados; - Desenvolver pipelines e soluções utilizando Azure e Databricks; - Implementar arquitetura medalhão, com camadas Bronze, Silver e Gold; - Desenvolver e otimizar rotinas de processamento em Python; - Trabalhar com modelagem dimensional e bancos de dados; - Participar das cerimônias da squad e realizar code reviews; - Apoiar tecnicamente profissionais menos experientes; - Atuar diretamente no desenvolvimento das demandas mais complexas. Qualifications - Experiência sólida em Engenharia de Dados; - Domínio de Databricks Moderno e Azure; - Experiência avançada com Python; - Conhecimento em arquitetura medalhão e modelagem dimensional; - Experiência forte com MongoDB; - Conhecimento em arquitetura de dados e levantamento de requisitos; - Boa comunicação com áreas técnicas e de negócio; - Autonomia, capacidade analítica e perfil orientado à solução de problemas. Requirements - Experiência com recursos recentes da plataforma Databricks; - Conhecimento em Lakeflow Connect, Lakeflow Jobs e Lakeflow Spark Declarative Pipelines; - Vivência com Unity Catalog, Serverless Compute e governança de dados; - Conhecimento em Lakebase e processamento de dados em tempo real; - Familiaridade com Genie Code e recursos de engenharia de dados assistidos por IA. Benefits - Pessoa proativa, comunicativa, colaborativa, hands-on e capaz de simplificar problemas complexos de negócio.
Lead Data Engineer
Fortune Brands InnovationsFortune Brands Innovations is a leading innovation company that creates smarter, safer, and more beautiful homes. With a rich history of delivering high-quality solutions, the comp
Company Description Fortune Brands Innovations, Inc. is an industry-leading home, security and digital products company. We’re focused on exciting opportunities within the home, security and commercial building markets. Our driving purpose is to elevate every life by transforming spaces into havens. We believe our work and our brands can have incredibly positive impacts for not just our business and shareholders, but for people and the planet, too. At Fortune Brands, we’re building something big. We’re advancing exciting innovations in all of our products and processes. We’re delivering trust, dependability, sustainability, and style. To make it all happen, we’ve transformed our workplace into an environment where smart, ambitious people have the support to reach their fullest potential. When you join Fortune Brands, you become part of a high-performing team empowered to think big, learn fast and make bold decisions. We support an inclusive culture where everyone is encouraged be their authentic selves, and where our differences and unique perspectives are a key strength. Explore life at Fortune Brands here. Job Description Lead Data Engineer, Fortune Brands Innovations Group, Inc., Deerfield, IL. Administer a cloud data warehouse platform (e.g., Snowflake or similar), including roles/permissions, performance tuning, security, & cost governance. Utilize Git for version control & collaboration throughout the development lifecycle. Help design scalable, reliable, & secure data architectures & models. Must have a Bachelor's Degree in Computer Science, Information Systems, or related & 5 years of experience in a data engineering or data operations position. Qualifications Of experience required, must also have 5 years of experience in each of the following: - Developing & maintaining ETL/ELT pipelines - Utilizing professional SQL development - Professional Python development with code-quality/testing practices - Using version control within the development lifecycle - Working with a cloud data warehouse platform (e.g., Snowflake, Amazon Redshift, or Google BigQuery) including performance optimization & role-based access/security - Building transformations & models using a data transformation framework (e.g., dbt or similar) - Implementing CI/CD for data code using common tools - Monitoring & optimizing cloud data platform costs - Migrating & loading data from APIs, SaaS applications, & ERP systems (e.g., Oracle or SAP) - Working with ETL tools & relational database platforms; & - Unix-based command-line tools. Experience may be gained concurrently. 100% remote. Apply online at: www.fbin.com/careers Additional Information Fortune Brands believes in fair and equitable pay. A reasonable estimate of the base salary range for this role is $160,555 to $193,000 per year. An employee’s actual salary within the range will be based on factors including, but not limited to, relevant education, training, experience, professional achievement, business need, and location. In addition to base salary, employees will participate in either an annual bonus plan based on company and individual performance, or a role-based sales incentive plan. At Fortune Brands, we support the overall health and wellness of our associates by offering comprehensive, competitive benefits that prioritize all aspects of wellbeing and provide flexibility for our teammates’ unique needs. This includes robust health plans, a market-leading 401(k) program with a company contribution, product discounts, flexible time off benefits, adoption benefits, and more. We offer numerous ERGs (Employee Resource Groups) to foster a sense of belonging for all associates. Fortune Brands is built on industry-leading brands and innovation within the high-growth categories of water, outdoors and security. The Company makes innovative products for residential and commercial environments, with a growing focus on digital solutions and products that add luxury, contribute to safety and enhance sustainability. To learn more, visit our website at fbin.com. Equal Employment Opportunity Fortune Brands is an equal opportunity employer. Fortune Brands evaluates qualified applicants without regard to race, color, religion, sex, gender identity or expression, national origin, ancestry, age, disability/handicap status, marital status, protected veteran status, sexual orientation, genetic history or information, or any other legally protected characteristic. Reasonable Accommodations Fortune Brands is committed to working with and providing reasonable accommodations to individuals with disabilities. If, because of a medical condition or disability, you need a reasonable accommodation for any part of the application or interview process, please contact us at fbinleave.fbhs@fbin.com and let us know the nature of your request along with your contact information. Important Notice: Protect Yourself from Fraudulent Job Postings To protect yourself from fraudulent job postings or recruitment scams, please note that Fortune Brands job postings are exclusively hosted on our website at fbin.com/careers via our SmartRecruiters platform. Fortune Brands will never request banking information or sensitive personal details until an offer of employment has been accepted and the onboarding process begins.


