Hadoop Developer
Location
United States
Posted
11 days ago
Salary
0
Seniority
Senior
Job Description
Hadoop Developer
Bright Vision Technologies
• Design, develop, and operate end-to-end big-data pipelines on Hadoop, ingesting data from various sources. • Build robust ETL/ELT workflows using Apache Spark, Hive, Pig, and Sqoop. • Develop high-throughput streaming data pipelines using Kafka, Spark Streaming, or Flink. • Optimize Spark and MapReduce jobs to meet SLAs at minimal cost. • Design and maintain data models and storage layouts on HDFS and related formats. • Implement data governance and quality controls. • Build robust monitoring and logging strategies for big-data pipelines. • Partner with data scientists to deliver reliable datasets. • Automate pipeline orchestration using Airflow or Oozie. • Continuously evaluate and adopt new technologies in the big-data ecosystem. • Mentor junior engineers and contribute to the team’s engineering standards.
Job Requirements
- 5+ years of professional experience designing and operating big-data pipelines on Hadoop.
- Strong hands-on expertise with Apache Spark (Scala, Python, or Java) in production environments.
- Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
- Hands-on experience with streaming data platforms such as Kafka, Spark Streaming, or Flink.
- Strong SQL skills and experience working with both relational and NoSQL data stores.
- Experience with workflow orchestration tools such as Airflow or Oozie.
- Solid understanding of distributed systems concepts, including partitioning, replication, and fault tolerance.
- Strong scripting skills in Python or Shell.
- Excellent troubleshooting, debugging, and documentation skills.
Benefits
- Comprehensive benefits
- Competitive compensation packages
- Supportive work-life balance
Related Guides
Related Job Pages
More Software Engineer Jobs
• Build and maintain enterprise-grade digital experiences on AEM Sites, AEM Assets, and AEM as a Cloud Service • Focus on hands-on development of components, templates, workflows, integrations, and front-end experiences using AEM’s modern stack • Design and develop AEM Sites components, templates, and content fragments following modern AEM patterns • Build OSGi services, Sling models, servlets, and Java backend logic within AEM • Develop front-end experiences using HTL, JavaScript, and modern build tools • Implement headless and hybrid delivery patterns using AEM Content Fragments, GraphQL APIs, and edge delivery • Migrate or build implementations on AEM as a Cloud Service with cloud-native CI/CD pipelines • Develop AEM Assets workflows including DAM customizations and metadata schema design • Integrate AEM with Adobe Experience Cloud (Target, Analytics, Campaign) and external systems • Implement personalization, A/B testing, and targeted content delivery using AEM and Adobe Target • Optimize page performance, caching, and Dispatcher configuration for high-traffic sites • Build CI/CD pipelines using Cloud Manager, GitHub Actions, or Jenkins for AEM deployments • Participate in code reviews, contribute to standards, and document technical decisions • Troubleshoot production issues spanning author, publish, and Dispatcher tiers of the AEM stack • Collaborate with content authors, designers, and product owners on authoring experience improvements • Stay current with AEM releases, Cloud Service evolution, and Adobe ecosystem developments
• Deliver configuration, integration, and customization across the Guidewire InsuranceSuite • Focus on hands-on Gosu development, integration design, product model configuration, and Guidewire Cloud upgrades • Work closely with cross-functional partners to translate ambiguous requirements into well-engineered solutions • Raise the bar through code review, design review, and mentorship of more junior engineers
• Assist in customizing and extending Windchill using Java, JSP, Info*Engine, and Windchill APIs • Develop and maintain Workflows, Lifecycle templates, and UI customizations • Support Windchill configuration activities, including object initialization rules (OIR), type/attribute creation, and preference management • Assist in Windchill upgrades, patches, and deployments • Work with senior team members to debug issues in Method Server logs, workflow logs, and application logs • Support Data loading/migration activities using LoadFiles or Windchill Bulk Migrator (WBM) • Participate in integration tasks between Windchill and external systems (ERP, CAD applications, SSO, Directory Server, etc.) • Provide day to day support for end user issues, enhancements, and performance troubleshooting • Follow established development processes, documentation standards, and code review practices
Enterprise SWE - Systems Specialist
MercorCincinnatus is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for full-time and long-term contingent roles. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives. Roles hired through Cincinnatus are not project-based or freelance engagements. They are structured, role-based positions that typically involve full-time or fixed-term commitments, close collaboration with a client's internal teams, and integration into standard enterprise workflows. Cincinnatus is a legal entity separate from Mercor. While opportunities may be discovered through Mercor's platform, employment, onboarding, payroll, and benefits for these roles are administered by Cincinnatus. Equal Employment Opportunity Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic. Cincinnatus is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans throughout the job application process.
Role Description - Develop and complete coding tasks in various enterprise languages. - Evaluate and improve systems and build toolchains using C and Rust. - Enhance enterprise JVM and build ecosystems with C#, Kotlin, Scala, and Swift. - Optimize production web and ops scripting with Ruby, PHP, and Perl. - Advance scientific and quant computing using R. - Work independently and asynchronously to meet project deadlines. Qualifications - Strong professional experience in at least one listed domain. - Deep understanding of language internals, tooling, and real-world production constraints. - Ability to reason about edge cases, performance, and system design trade-offs. - Detail-oriented with strong written feedback skills. Company Description Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
