Boutique Recruitment Agency sourcing Leaders for growth businesses.
Lead Data Engineer
Location
United States
Posted
83 days ago
Salary
$180K - $200K / year
Seniority
Lead
Job Description
Lead Data Engineer
Serv Recruitment Agency
Join the Team Serv, a global executive recruitment partner, is hiring on behalf of our client Mercator.ai for a Lead Data Engineer (Data Pipelines). MercatorAI is building scalable data infrastructure to power high-quality, data-driven decision making at scale. As an early-stage company, the team is focused on creating robust, future-ready systems that can handle complex data ingestion, transformation, and delivery across a growing national footprint. This role is responsible for owning and evolving the company’s data pipeline architecture, ensuring systems are scalable, reliable, and built to support long-term growth. Position Overview The Lead Data Engineer is the technical owner of MercatorAI’s data infrastructure and pipeline systems. This individual will lead the design, development, and scaling of distributed data pipelines while remaining hands-on in implementation. This is a builder role suited for someone who enjoys working directly in the code while also setting architectural direction. You will be responsible for scaling existing systems, improving performance and reliability, and integrating modern tools including AI-assisted workflows to enhance engineering output. This role requires strong technical depth, ownership, and the ability to operate effectively in a fast-paced, early-stage environment. Position Responsibilities - Lead the architecture and evolution of scalable, distributed data pipelines, ensuring high availability and performance at scale - Design and implement robust data models to support reporting and advanced data applications - Build and maintain distributed web scraping systems using tools such as Playwright, Selenium, and BeautifulSoup - Develop systems capable of handling anti-scraping measures, proxy rotation, and high-volume data extraction - Integrate AI and LLMs into engineering workflows for code generation, automation, and optimization - Apply prompt engineering techniques to improve data processing, documentation, and troubleshooting - Identify and implement system and process improvements to optimize performance and efficiency - Manage and scale cloud-based data infrastructure, including data warehouses, object storage, and search systems - Deploy and maintain containerized workloads using Kubernetes - Implement data quality monitoring and governance processes to ensure accuracy and reliability - Mentor junior engineers through code reviews, documentation, and knowledge sharing - Communicate technical concepts clearly and provide business context for engineering decisions Qualifications Required Experience & Skills - 5+ years of experience in Data Engineering with a track record of scaling systems - Expert proficiency in Python and advanced SQL, including performance tuning and optimization - Strong experience with workflow orchestration tools such as Airflow or Prefect and transformation tools such as dbt - Proven experience building resilient web scraping systems using Playwright, Selenium, and BeautifulSoup - Deep understanding of relational and NoSQL databases including Postgres, MongoDB, and ElasticSearch - Experience working with large-scale data systems such as BigQuery - Strong proficiency with CI/CD pipelines, Git, and Docker - Experience designing and maintaining distributed systems with high availability and fault tolerance Nice-to-Have Experience - Experience with GCP or AWS and Kubernetes for infrastructure management - Familiarity with LLMs such as ChatGPT, Claude, or Gemini for engineering workflows - Experience with prompt optimization and AI-assisted development Dream Teammate Our ideal teammate: - Is highly proactive and takes full ownership of systems and outcomes - Balances hands-on execution with long-term architectural thinking - Communicates complex ideas clearly and effectively - Thrives in fast-paced, early-stage environments - Is detail-oriented and committed to building high-quality, scalable systems - Actively supports and elevates teammates through knowledge sharing and mentorship - Brings curiosity and continuously looks for ways to improve systems and processes Compensation: $180-$200K base Location: Remote (Eastern timezone preferred)
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Collection Engineer-North America
NulogyLeveraged by the world’s leading brands to optimize and accelerate global supply chain performance.
COMPANY OVERVIEW Nulogy is a technology company driving toward more agility within the supply chain. Nulogy’s software platform enables Consumer Packaged Goods (CPG) companies and their contracted (external) network of manufacturers and packagers to reduce waste, reduce costs, and better serve their customers. Nulogy’s software is currently being leveraged within the world’s leading CPG organizations (including brands like Colgate-Palmolive, L’Oréal, and P&G) and has become synonymous with best of breed supply chain solutions. As Nulogy has grown rapidly over the last decade, the company is proud to be recognized with a variety of awards including; Canada’s Top Small & Medium Employers, and Food Logistics Top 100 Software Provider and Canada's Most Admired Corporate Culture. Nulogy has also received multiple awards for company culture including most recently Canada’s Top Small & Medium Employers for 2024. By joining Nulogy, not only will you be joining a fantastic culture of capable and motivated individuals, you'll also be joining a team that is solving challenging problems everyday with the potential to make a global impact. YOUR MISSION Nulogy is seeking a Data Collection Engineer to join our implementation team. If you have deep experience in manufacturing automation, controls, or industrial networking and are excited about bridging shop floor technology with production monitoring software, this role is for you. KEY RESPONSIBILITIES Sensor & Hardware Strategy - Evaluate production processes and recommend appropriate sensors (photoeyes, encoders, proximity sensors, counters, etc.) - Determine best-fit hardware based on product type, line speed, and manufacturing environment - Configure and deploy sensor hardware to ensure accurate production data capture Industrial Networking & IT Collaboration - Quickly assess plant network architecture (VLANs, firewalls, segmentation, security policies) - Provide exact technical requirements and steps for secure machine connectivity - Configure gateways and edge devices for data collection - Partner with plant IT and maintenance teams to accelerate deployment Manufacturing Process Expertise - Identify key data points required for OEE and performance monitoring - Understand how discrete vs. high-speed packaging vs. co-manufacturing environments impact data strategy - Translate operational goals into scalable data collection architecture - Help customers understand and leverage their production data Customer Enablement - Conduct Site Surveys - Lead technical training sessions - Ensure customers are comfortable operating and maintaining the system - Support issue resolution post-implementation Machine Connectivity & PLC Integration - Program and modify PLCs (Allen-Bradley, Siemens, etc.) to enable data extraction - Develop and validate tag mappings between PLCs and the Smart Factory platform - Connect equipment using industrial protocols (EtherNet/IP, Modbus, Profinet, OPC-UA) - Work with unfamiliar equipment to determine optimal integration methods - Troubleshoot communication issues during implementation and post-go-live ABOUT YOU Required Qualifications - Experience configuring industrial gateways and edge devices - Strong networking knowledge (routing, switching, VLANs, firewalls, TCP/IP, DNS, DHCP) - 5+ years of experience in manufacturing automation, controls engineering, or industrial data collection - Proven experience integrating equipment into production monitoring or MES systems - Deep understanding of industrial communication protocols (EtherNet/IP, Modbus TCP, Profinet, OPC-UA) - Practical knowledge of plant network architecture and IT coordination - Experience selecting and deploying industrial sensors - Strong troubleshooting and analytical skills - Driver’s license and passport - Ability to Travel, up to 50% Preferred Qualifications - Experience with PLC programming (Rockwell/Allen-Bradley preferred; Siemens a plus) - Experience implementing OEE or production monitoring systems - Ability to document and standardize connectivity best practices - Comfort leading customer-facing technical sessions - Experience working in 3PL, co-packing, co-manufacturing, or high-speed manufacturing environments - Bilingual (English/Spanish) is a plus WHY NULOGY ● Competitive benefits package: Unlimited Paid Vacation policy, Parental Leave: 100% top-up for 15 weeks for any parent or biological or adopted children, RRSP Matching Program; $3k annual education budget, Group benefit plan including health, dental, and vision that start on day one; ● We’re solving problems that no one has been able to solve before. This is an opportunity to join an organization that is transforming the way that supply chains are structured. FULLY REMOTE Everyone is set up with the tools and resources required to stay connected and help employees do their best work. Nulogy embraces diversity, and we recognize the need for teams that represent a variety of backgrounds, perspectives, and skills. The more inclusive we are, the better our work will be. We encourage everyone to apply.
• Support and evolve a Lakehouse data platform on AWS based on the Medallion model (raw, trusted, and refined), with a focus on data originating from SAP systems; • Responsible for ensuring the reliability, scalability, and operational efficiency of data pipelines, driving continuous improvements in performance, security, governance, and cost optimization; • Technical leadership in the design and implementation of ingestion and processing pipelines using services such as AWS Glue, EMR, and Lambda, ensuring adherence to cloud-native architecture patterns and data engineering best practices; • Lead modernization and migration initiatives for legacy workloads to AWS, focusing on resilience, scalability, and reduced operational complexity; • Work across multiple projects with strong interaction with business, architecture, security, and operations teams, contributing to strategic alignment, requirements definition, and the evolution of the corporate data platform.
• Design and implement scalable data pipelines (ETL/ELT); • Lead the migration from Azure to AWS; • Work with Lakehouse architectures (Delta Lake / Iceberg); • Integrate tools such as ADF, Synapse, Glue, and Databricks; • Ensure data governance, quality, and orchestration; • Support solutions that underpin Machine Learning models.
• Monitor and troubleshoot data pipeline and warehouse quality and performance issues, ensuring smooth operation and proactively addressing potential bottlenecks • Design, build, and maintain scalable ETL/ELT pipelines to transfer data to and from clients from diverse sources, such as APIs, cloud storage, and databases • Collaborate with client engineering teams to understand their data structures, identify source systems, and contribute to technical integration requirements • Stay updated with industry trends and advancements in data engineering and bring innovative ideas to continuously improve our data processes • Ensure data security, privacy, and compliance by implementing appropriate Azure policies, procedures, and encryption methods • Monitor and optimize the performance, scalability, and cost-efficiency of data storage and processing solutions on Azure
