Particle41 logo
Particle41

We provide world-class teams for App Development, DevOps & Data Science.

Data Engineer

Data EngineerData EngineerFull TimeRemoteSeniorTeam 51-200H1B No SponsorCompany SiteLinkedIn

Location

India

Posted

151 days ago

Salary

₹800K - ₹1,750K / year

Seniority

Senior

Job Description

Data Engineer

Particle41

• Design, develop, and maintain scalable ETL (Extract, Transform, Load) pipelines to process large volumes of data from diverse sources. • Build and optimize data storage solutions, such as data lakes and data warehouses, to ensure efficient data retrieval and processing. • Integrate structured and unstructured data from various internal and external systems to create a unified view for analysis. • Ensure data accuracy, consistency, and completeness through rigorous validation, cleansing, and transformation processes. • Maintain comprehensive documentation for data processes, tools, and systems while promoting best practices for efficient workflows. • Collaborate with product managers, and other stakeholders to gather requirements and translate them into technical solutions. • Participate in requirement analysis sessions to understand business needs and user requirements. • Provide technical insights and recommendations during the requirements-gathering process. • Participate in Agile development processes, including sprint planning, daily stand-ups, and sprint reviews. • Work closely with Agile teams to deliver software solutions on time and within scope. • Adapt to changing priorities and requirements in a fast-paced Agile environment. • Conduct thorough testing and debugging to ensure the reliability, security, and performance of applications. • Write unit tests and validate the functionality of developed features and individual elements. • Writing integration tests to ensure different elements within a given application function as intended and meet desired requirements. • Identify and resolve software defects, code smells, and performance bottlenecks. • Stay updated with the latest technologies and trends in full-stack development. • Propose innovative solutions to improve the performance, security, scalability, and maintainability of applications. • Continuously seek opportunities to optimize and refactor existing codebase for better efficiency. • Stay up to date with cloud platforms such as AWS, Azure, or Google Cloud Platform. • Collaborate effectively with cross-functional teams, including testers, and product managers. • Foster a collaborative and inclusive work environment where ideas are shared and valued.

Job Requirements

  • Bachelor's degree in computer science, Engineering, or related field.
  • Proven experience as a Data Engineer, with a minimum of 3 years of experience.
  • Proficiency in Python programming language.
  • Experience with database technologies such as SQL (e.g., MySQL, PostgreSQL) and NoSQL (e.g., MongoDB) databases.
  • Strong understanding of Programming Libraries/Frameworks and technologies such as Flask, API frameworks, datawarehousing/lakehouse, principles, database and ORM, data analysis databricks, panda's, Spark, Pyspark, Machine learning, OpenCV, scikit-learn.
  • Utilities & Tools: logging, requests, subprocess, regex, pytest
  • ELK stack, Redis, distributed task queues.
  • Strong understanding of data warehousing/lakehousing principles and concurrent/parallel processing concepts.
  • Familiarity with at least one cloud data engineering stack (Azure, AWS, or GCP) and the ability to quickly learn and adapt to new ETL/ELT tools across various cloud providers.
  • Familiarity with version control systems like Git and collaborative development workflows.
  • Competence in working on Linux OS and creating shell scripts.
  • Solid understanding of software engineering principles, design patterns, and best practices.
  • Excellent problem-solving and analytical skills, with a keen attention to detail.
  • Effective communication skills, both written and verbal, and the ability to collaborate in a team environment.
  • Adaptability and willingness to learn new technologies and tools as needed.

Related Categories

Related Job Pages

More Data Engineer Jobs

Supermom Business logo

Data Engineer, ETL 工程师

Supermom Business

Help brands win mind and market shares with extraordinary speed

Data Engineer151 days ago
ContractRemoteTeam 51-200H1B No Sponsor

1. Responsible for data cleaning (ETL) and data warehouse construction to support large-scale AI models. 2. Responsible for training and fine-tuning large AI models to meet the requirements of specific business scenarios. 3. Responsible for developing supporting tools, such as dashboards and general business logic, to ensure the practicality of AI model applications. 4. Must have hands-on development experience and be able to lead a team or independently complete projects related to data collection and development.

Malaysia
Job Closed
Scratch Financial logo

Senior Data Engineer, Engineering & Operations

Scratch Financial

Scratch Financial is the world's simplest patient financing solution.

Data Engineer151 days ago
OtherRemoteTeam 11-50Since 1912H1B Sponsor

• Define partner onboarding and clean room architecture patterns across Snowflake, LiveRamp, and Databricks that are secure, scalable, and repeatable • Configure and manage partner-specific clean room environments; deploy and manage Python-based libraries within the platform ecosystem • Establish and maintain MLOps practices, including model serving, monitoring, and pipeline orchestration for AI/ML features deployed within the platform ecosystem • Own design and enforcement of granular RBAC policies and least-privilege service accounts • Serve as the technical lead for onboarding new partners, implementing privacy-preserving controls (e.g., aggregation thresholds and anonymization techniques) • Design, build, and operate scalable ELT pipelines using Snowpark and/or PySpark and advanced SQL to provision Gold datasets • Implement and evolve identity resolution logic mapping internal data to 3P identifiers (including LUIDs, RampIDs, TransUnion IDs), ensuring privacy-safe practices • Design and operate scalable data architectures across Snowflake and Databricks supporting batch and near real-time processing patterns • Build robust automated checks (e.g., Great Expectations or custom SQL assertions) and define quality standards to detect schema drift, null rate spikes, and volume anomalies • Lead performance optimization across platforms (query tuning, caching, incremental processing) and define and implement query tagging and chargeback models for accurate cost attribution • Establish monitoring, alerting, runbooks, and standard operating procedures to improve platform reliability and reduce incident time-to-resolution • Validate that output data adheres to privacy and business requirements, and define test strategies for partner-facing releases • Serve as the escalation point for diagnosing connection failures, data discrepancies, or latency issues with partner technical teams • Design and build internal AI agents (using frameworks like LangChain, Snowflake Cortex) and mentor other engineers through code reviews, design discussions, and operational best practices

New York
$140K - $180K / year
Job Closed
TWO95 International, Inc logo

Senior Data Engineer

TWO95 International, Inc

Recruitment and Staffing Soultion

Data Engineer151 days ago
Full TimeRemoteTeam 51-200Since 1993H1B No Sponsor

• Title: Senior Data Engineer • Location: Remote • Duration: C2H • Salary: 1-2Lkhs Per Month

India
₹100K - ₹200K / month
Henry Schein One logo

Manager, Data Engineering

Henry Schein One

Dentrix Enterprise. Dentrix. Dentrix Ascend. Jarvis Analytics. Lighthouse 360.

Data Engineer151 days ago
OtherRemoteTeam 1,001-5,000Since 2018

• Lead teams of software engineers to improve the lives of Henry Schein One users and their patients through the creation of market-leading solutions • Help recruit, develop, and retain engineers to create a high-performing team • Oversee team leaders and support them in fulfilling their mission and delivery commitments • Enable engineering teams to practice continuous delivery at scale by identifying and removing workflow roadblocks • Inspire, motivate and coach to enable continuous development of team leaders and their teams • Ensure products are secure, compliant and reliable for Henry Schein One customers • Collaborate with senior leaders on the vision and strategy for workstreams and products • Partner closely with Product leadership to influence and understand the strategic commitments of your teams • Build strong cross-functional relationships and influence others to pursue the best possible user experience • Enable process and practice maturity to deliver on global growth opportunities • Positively and collaboratively influence decision-making to improve practices across your teams, department, and stakeholder network • Travel typically up to 10% of time • Office environment with no special physical demands required

Alaska + 11 moreAll locations: Alaska | District of Columbia | Hawaii | Louisiana | Nebraska | North Dakota | Rhode Island | South Dakota | Vermont | Virginia | Washington | West Virginia
$145K - $165K / year
Job Closed