Job Closed

This listing is no longer active.

Ad Hoc LLC logo
Ad Hoc LLC

Digital-first government for the common good.

Data Engineer II, PySpark/Databricks

Data EngineerData EngineerFull TimeRemoteLeadTeam 501-1,000Since 2014H1B No SponsorCompany SiteLinkedIn

Location

United States

Posted

50 days ago

Salary

$90K - $110K / year

Seniority

Lead

Bachelor Degree8 yrs expExperience acceptedEnglishApacheAWSETLPySparkPythonSpark

Job Description

Data Engineer II, PySpark/Databricks

Ad Hoc LLC

• Build and maintain PySpark data pipelines in the Databricks environment • Optimize Spark jobs performance and resource usage, identifying and addressing bottlenecks and inefficiencies in backend systems • Design, develop, and maintain high-quality backend software components and services, ensuring functionality, performance, and scalability • Research and build proof of concepts in the data space • Write clean, well-structured, and maintainable code, adhering to established coding standards and best practices • Perform thorough code reviews, providing constructive feedback to peers and identifying potential risks or areas for improvement • Debug and resolve defects, proactively identifying and addressing potential issues before they impact users • Create and maintain comprehensive technical documentation • Actively participate in Agile ceremonies, such as stand-ups, sprint planning, and retrospectives, ensuring effective communication and collaboration across the team • Assist in the estimation, prioritization, and planning of development tasks, ensuring projects are delivered on time and within budget • Continuously evaluate and recommend new dataframe related technologies, frameworks, and tools, helping to drive innovation and keep the team up-to-date with industry trends • Engage in ongoing professional development to stay current with industry best practices, and share knowledge and insights with the team as appropriate • Assist in the implementation and maintenance of security, compliance, and governance policies within the Databricks and AWS environment to ensure adherence to industry standards and regulatory requirements.

Job Requirements

  • Bachelor's degree and 8 years of experience
  • Strong experience with Python / Apache Spark
  • Solid understanding of data modeling, ETL process, and distributed computing
  • Bachelor's degree in Computer Science, Computer Engineering or related field
  • Strong understanding of software design patterns, data structures, and algorithms
  • Experience with Agile development methodologies
  • Ability to work independently as well as in a team
  • Strong problem-solving and analytical skills
  • Strong verbal and written communication skills
  • Related experience in analytic programming, data extraction, querying databases/data warehouses and data analysis.

Benefits

  • Company-subsidized health, dental, and vision insurance
  • Flexible PTO
  • 401K with employer match
  • Paid parental leave after one year of service
  • Employee Assistance Program

Related Categories

Related Job Pages

More Data Engineer Jobs

Data Science Associate

Havas

Havas, formerly known as Euro RSCG 4D Impact, is a marketing and advertising agency that aims to establish innovative and meaningful connections between brands

Data Engineer50 days ago

Data Science Associate Location: Peru Full time job requisition id JR0094496 Job Description: Agency : Havas Media Group Job Description : Data Science Associate - Marketing Analytics & Advanced Measurement (Global CoE) Location: Perú Fully Remote. CSA, the data consultancy within Havas Media Group, is seeking a Data Science Associate with a focus on Marketing to join our growing team. The Data Science Associate will have a passion for leveraging data-driven insights, implementing innovative marketing strategies, and optimizing customer experiences. They will play a pivotal role in providing state-of-the-art, privacy-first solutions for our clients, working with both event-level and aggregated data to create actionable insights and drive marketing performance. Key Responsibilities - Perform data wrangling tasks, including data cleaning, transformation, and integration from multiple sources to ensure high-quality and accurate datasets for analysis. - Continuously research and learn about new technologies and statistical methodologies to stay current in the rapidly evolving field of data science and marketing analytics. - Assist in applying advanced statistical techniques and machine learning algorithms to analyze event-level and aggregated data, identify patterns, and provide actionable insights to drive marketing performance. - Support the implementation and evaluation of marketing models, such as customer segmentation, lift analysis, attribution modeling, and marketing mix models. Requirements: - Bachelor's or Master's degree in Data Science, Economy, Statistics, Computer Science, or a related field. - Proficiency in statistical programming languages and packages such as Python, Pandas, and sci-kit-learn. Knowledge of SQL is desirable. - Knowledge of statistics (e.g., correlation and regression, statistical significance tests, t-tests, z-score) and Machine Learning (Clustering, ensembles, SVM). - Proficiency in MS Office tools such as Excel. - Advanced English language skills (mandatory). - Ability to work in a team, even remotely with global teams. What We Offer: Global Exposure: Work with international teams and global brands. Long-Term Contract: Stability and the opportunity to grow within a global network. Remote Work Model: Virtual work environment with a healthy work-life balance. Health & Wellness: EPS 100% health insurance and wellness initiatives. Culture & Community: Inclusive, collaborative, and purpose-driven workplace. Competitive Compensation: Full time contract monthly salary + food benefit card If you are a Data Scientist with a passion for marketing analytics and are eager to make a significant impact on our clients' success, we would love to hear from you! Apply now to join the CSA team within Havas Media Group and take your career to new heights. Contract Type : Permanent Here at Havas across the group we pride ourselves on being committed to offering equal opportunities to all potential employees and have zero tolerance for discrimination. We are an equal opportunity employer and welcome applicants irrespective of age, sex, race, ethnicity, disability and other factors that have no bearing on an individual's ability to perform their job.

Peru
Rackspace Technology logo

Intern I - IN

Rackspace Technology

Where enterprise AI runs and outcomes scale

Data Engineer50 days ago
Full TimeRemoteTeam 5,001-10,000Since 1998H1B No Sponsor

Job Title: Intern-Software Engineer Location: Remote Duration: 6 Months (Paid Internship of INR 20,000 (monthly) for final year BE/ BTECH student ) Department: AI Transformation About the Role We are looking for a passionate and driven Software Engineer Intern to join our AI Industrialization team. This is an excellent opportunity to work in a collaborative, fast-paced environment, gaining hands-on experience in building innovative software solutions using AI. Key Responsibilities • Build and maintain full stack features for AI-powered internal applications • Integrate LLM capabilities into web applications using LangChain and AWS Bedrock • Write clean, well-tested Python backend services and APIs • Work with the team to deploy services on AWS and Kubernetes • Collaborate with senior engineers and the tech lead to break down requirements and deliver features • Learn and apply best practices in AI application development, prompt engineering, and cloud infrastructure Qualifications • Final year students of B.E. / B. Tech / MCA/ M. Tech (Computer Science) • Eagerness to learn, adapt, and work on new challenges. • A proactive and positive attitude toward learning and problem-solving. • Strong communication skills and ability to collaborate effectively in a team. • Passion for technology and software development. Must-Have Skills • Python — solid working proficiency for backend development and scripting • Full Stack Development — experience building features with a frontend framework (React or similar) and a backend API • AWS — foundational knowledge of core AWS services (S3, Lambda, EC2 or ECS) • LLM / AI APIs — some hands-on experience calling or integrating LLM APIs (OpenAI, Bedrock, or similar) • REST APIs — ability to build and consume RESTful APIs • Git & CI/CD — comfortable with version control and basic deployment pipelines Good-to-Have Skills • Exposure to LangChain or other LLM orchestration frameworks • Familiarity with Kubernetes or containerized deployments (Docker) • Basic understanding of RAG concepts or vector databases • Experience with AWS Bedrock or other managed AI services • Familiarity with agile development practices • Exposure to prompt engineering basics Why Join Us? • Hands-on experience with real-world projects and cutting-edge technology. • Mentorship and learning opportunities from experienced software engineers. • A collaborative and innovative work environment. • Potential for future full-time employment opportunity About Rackspace Technology We are the multicloud solutions experts. We combine our expertise with the world’s leading technologies — across applications, data and security — to deliver end-to-end solutions. We have a proven record of advising customers based on their business challenges, designing solutions that scale, building and managing those solutions, and optimizing returns into the future. Named a best place to work, year after year according to Fortune, Forbes and Glassdoor, we attract and develop world-class talent. Join us on our mission to embrace technology, empower customers and deliver the future. More on Rackspace Technology Though we’re all different, Rackers thrive through our connection to a central goal: to be a valued member of a winning team on an inspiring mission. We bring our whole selves to work every day. And we embrace the notion that unique perspectives fuel innovation and enable us to best serve our customers and communities around the globe. We welcome you to apply today and want you to know that we are committed to offering equal employment opportunity without regard to age, color, disability, gender reassignment or identity or expression, genetic information, marital or civil partner status, pregnancy or maternity status, military or veteran status, nationality, ethnic or national origin, race, religion or belief, sexual orientation, or any legally protected characteristic. If you have a disability or special need that requires accommodation, please let us know.

India
₹20K / month
Job Closed
Shell logo

Data Engineer

Shell

Shell is a diversified energy company in India with 13,000 employees, and presence in Integrated Gas, Downstream, Power, Renewable and Upstream. Our global strategy, Powering Progress, is designed to generate value for our shareholders, customers, and the wider society, focusing on creating more value with less emissions. We aim to profitably transform Shell into a net-zero emissions energy business by 2050. Our Lubricants business serves over 50,000 consumers through a strong network of over 200 distributors. Shell owns and operates a LNG re-gasification terminal at Hazira, Surat, with a capacity of 5 MTPA. Our three capability centres across Bangalore and Chennai serve as a technology and innovation powerhouse for Shell globally. We are committed to positively contributing to the communities in which we operate through programmes on STEM Education, Skilling, and Livelihood.

Data Engineer50 days ago
Full TimeRemoteTeam 10,001+H1B Sponsor

Texas, United States of America Job Family Group: Information Technology (IT) Worker Type: Regular Posting Start Date: April 13, 2026 Business Unit: Projects and Technology Experience Level: Experienced Professionals Job Description: What’s the role The Data Engineer is the one who designs and builds data foundations and end-to-end solutions for the Shell Business to maximize value from data. The role helps create a data-driven thinking within the organization, not just within IT teams, but also in the wider Business stakeholder community. A Data Engineer is expected to be a subject matter expert, who designs & builds data solutions and mentor junior engineers. They are also the key drivers to convert Vison and Data Strategy for IT solutions and deliver. Accountabilities: - Lead individual Data and Analytics product development in consultation with stakeholders, users, technical and functional leads being responsible to provide end-to-end solutions across data modeling and visualizations to the reporting needs of DE professionals. - Drive requirement gathering and continuous improvement discussions with customers that will drive a single source of truth data across DS. - Lead Process improvements and effective collaboration with different teams including DevOps IT Engineering and continue to learn new technologies and practices recommended by Shell – Agile, Power Apps, ADF etc. - Continuous improvements, working in a natural team environment, agile framework; being a team player is the success factors of this role. - Responsible for run and maintain of complex data model, building and publishing reports to track DS business metrices. - Effectively communicate your strong business acumen and build the business goal using Power BI and ADF What you bring - Must have legal authorization to work in the US on a full-time basis for anyone other than current employer - Bachelor’s degree in any IT discipline - Minimum 3 years of experience in the IT industry - Experience in handling large and complex data sets and deriving trend analysis, correlation, data discrepancy and data quality ensuring constancy of data across all platforms - Must bring Power BI solutions experience to address reporting needs for DS professionals which should enable tracking leading indicators for each of the business units - Experienced in working source control technologies (such as GITHUB, Azure DevOps), build and release pipelines - Lead ETL [Extract, Transfer & Load] activities. Expertise in Data stage, a solid understanding of Data warehousing concepts with significant experience with ETL Business Intelligence projects - Provide Power BI solutions to report requirements for RE professionals which should enable tracking leading indicators for each of the business units - Experience in writing complex SQL queries to perform data analysis. Hands-on experience in Azure Data Factory, Azure SQL, Azure Analytic Servicers, Azure Logic Apps, Power Apps - Experience in SharePoint end-to-end development and creating workflow solution - Good hands-on experience with Azure storage services such as Blob, Data Lake, Table Storage - Experience supporting and working with cross-functional teams in a dynamic environment - Good project management, requirement gathering and organizational skills - Self-motivated, Innovative and team player - Excellent communication and interpersonal skills Note: This role is a virtual position based in the US What we offer The base salary range for this position is $103,000 - $155,000 per year. Individual pay will be based on various factors, such as relevant work experience, education, certifications, skill level, seniority, and internal equity. For regular full-time or regular part-time employees of the Company (participating companies as listed in the Summary Plan Description), insurance coverage options include medical, dental, vision coverage, life Insurance, Business Travel Accident Insurance, and Occupational Accidental Death Benefit programs. Employees also participate in a company pension plan and a 401(k) plan. Paid leave includes up to 6 weeks of paid vacation time, up to 11 paid holidays, and parental leave offering 16 weeks of paid leave for birthing parents, and 8 weeks of paid leave for non-birthing parents. Additionally, employees are eligible for short-term disability leave for up to 26 weeks at 100% or 50% of base pay as well as Long-Term Disability insurance. Shell also offers other compensation such financial reimbursement for adoption, wellness, education, and personal learning expenses, and some roles are eligible for discretionary long-term incentives.  For interns, eligible benefits include medical, dental, and vision coverage, life insurance, Business Travel Accident Insurance, and Occupational Accidental Death Benefit programs; participation in a 401(k) plan; and paid leave for up to 11 paid holidays.  Additional information on Shell’s US benefit programs can be found at https://www.shell.us/careers/about-careers-at-shell/rewards-and-benefits.html. You bring your skills and experience to Shell and in return you work with talented, committed people on one of the most important challenges facing our planet. You’ll have the opportunity to develop the skills you need to grow in an environment where we value honesty, integrity, and respect for one another. You’ll be able to balance your priorities as you become the best version of yourself. - Progress as a person as we work on the energy transition together - Continuously grow the transferable skills you need to get ahead - Work at the forefront of technology, trends, and practices - Collaborate with experienced colleagues with unique expertise - Achieve your balance in a values-led culture that encourages you to be the best version of yourself - Benefit from flexible working hours, and the possibility of remote/mobile working - Perform at your best with a competitive starting salary and annual performance related salary increase – our pay and benefits packages are considered to be among the best in the world. - Take advantage of paid parental leave, including for non-birthing parents - Join an organization working to become one of the most diverse and inclusive in the world. We strongly encourage applicants of all genders, ages, ethnicities, cultures, abilities, sexual orientation, and life experiences to apply - Grow as you progress through diverse career opportunities in national and international teams - Gain access to a wide range of training and development programs We'd like you to know that Shell has a bold goal: to become one of the world’s most diverse and inclusive companies. You can get to know more about how we're working towards that goal, click here. Shell in The United States Shell has been in the US for more than 100 years, leading the sector in energy, petrochemicals and refined products. Today, we provide millions of Americans with the energy needed to heat and cool their homes and power the economy. We operate in all 50 states, from our Deepwater platforms in the Gulf of America to the Pennsylvania Chemicals complex and our miles of pipelines throughout the US. We reach our customers through our 13,000 branded retail stations, and we are number 1 in gasoline sales. We also own the #1 brand of motor oil in the U.S. – Pennzoil® – made from natural gas. We are excited to play a key role in the move to net carbon emissions while providing the oil and gas needed by society for many decades to come. - DISCLAIMER: Please note: We occasionally amend or withdraw Shell jobs and reserve the right to do so at any time, including prior to the advertised closing date. Before applying, you are advised to read our data protection policy. This policy describes the processing that may be associated with your personal data and informs you that your personal data may be transferred to Shell/Shell Group companies around the world. The Shell Group and its approved recruitment consultants will never ask you for a fee to process or consider your application for a career with Shell. Anyone who demands such a fee is not an authorised Shell representative and you are strongly advised to refuse any such demand. Shell participates in E-Verify. All qualified applicants will receive consideration for employment without regard to race, color, sex, national origin, age, religion, disability, sexual orientation, gender identity, protected veteran status, citizenship, genetic information or other protected status under federal, state or local laws. Shell is an Equal Opportunity Employer - Minorities/Females/Veterans/Disability. As a US Federal Contractor, hiring selections are subject to periodic audit review and documentation of your selections should be maintained for a period of three calendar years. It is the policy of Shell in the U.S. (“Shell”) to provide equal opportunity to all individuals, employees and all qualified applicants for employment consistent with employment requirements and qualifications. Shell prohibits discrimination based on race, color, sex, national origin, age, religion, disability, sexual orientation, gender identity, veteran status, citizenship, genetic information, or other protected status under federal, state or local laws. All employees are expected to support this policy and contribute to an environment of equal opportunity. If you need an accommodation for a disability during the resourcing process, please speak with an HR representative.

United States
$103K - $155K / year
Job Closed
Weekday (YC W21) logo

Director - Data Engineering

Weekday (YC W21)

We are a Y-Combinator-backed startup building your AI-powered Recruiter Agent

Data Engineer50 days ago
Full TimeRemoteTeam 11-50Since 2021H1B No Sponsor

This role is for one of the Weekday's clients Salary range: Rs 6000000 - Rs 8500000 (ie INR 60 - 85 LPA) Min Experience: 12 years Location: Remote (India) JobType: full-time We are seeking an experienced and visionary Director – Data Engineering to lead and scale our data engineering function. This role is ideal for a seasoned professional who combines strong technical expertise with leadership capabilities to drive data strategy, architecture, and execution. You will play a critical role in building robust, scalable, and efficient data platforms that power analytics, machine learning, and business intelligence across the organization.

India
₹6,000K - ₹8,500K / year
Job Closed