PortPro logo
PortPro

Meet PortPro's drayOS 2. The latest and most premier operating system for drayage carriers.

Director of Software Engineering – Node.js, Web Scraping Expert

Backend EngineerSoftware EngineerFull TimeRemoteLeadTeam 51-200H1B No SponsorCompany SiteLinkedIn

Location

California

Posted

111 days ago

Salary

0

Seniority

Lead

Job Description

Director of Software Engineering – Node.js, Web Scraping Expert

PortPro

• Architect, develop, and maintain scalable and distributed web scraping systems using Node.js. • Design and implement data extraction pipelines to process large volumes of structured and unstructured data. • Develop solutions to bypass anti-bot mechanisms, including CAPTCHA handling, session management, fingerprinting, and IP rotation. • Optimize scraping processes for performance, reliability, and efficiency while managing proxy services (residential, datacenter, rotating). • Oversee data storage and processing strategies, ensuring high availability and consistency. • Collaborate with Product, DevOps, and Data Science teams to integrate extracted data into analytics and business applications. • Implement best practices for microservices, API integrations, and real-time data streaming. • Lead the transition to cloud-native, containerized, and serverless architectures for web scraping. • Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.). • Optimize cloud resources (AWS, GCP, or Azure) to support high-throughput scraping. • Manage real-time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks. • Work closely with DevOps teams to optimize CI/CD pipelines, automated deployments, and system scalability. • Lead, mentor, and grow a high-performance engineering team. • Define and execute the technology roadmap, aligning with business objectives. • Foster a culture of continuous learning, collaboration, and innovation. • Implement agile development methodologies (Scrum, Kanban) to optimize project execution. • Ensure code quality, security, and best practices across all engineering efforts.

Job Requirements

  • 10+ years of experience in software engineering, with at least 5+ years in web scraping and large-scale data extraction.
  • Strong hands-on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation.
  • Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques.
  • Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs).
  • Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.).
  • Familiarity with data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.).
  • Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).

Related Job Pages

More Backend Engineer Jobs

Demand Chain logo

Lead Technical Architect

Demand Chain

Preferred Salesforce Partner for over 20 Years

Backend Engineer111 days ago
Full TimeRemoteTeam 11-50Since 2001H1B No Sponsor

• Provide technical leadership and vision for Salesforce initiatives, ensuring alignment with overall IT and business strategies. • Lead technical design sessions architect and document technical solutions aligned with client business objectives identify gaps between a customer’s current and desired end states. • Proven track record of successfully leading large scale Salesforce initiatives from concept to delivery. • Collaborate with business stakeholders, project managers, and development teams to gather requirements and translate them into technical solutions. • Design scalable and flexible Salesforce solutions that meet business requirements, ensure security, and adhere to best practices for platform optimization. • Stay updated with the latest Salesforce features, capabilities, and best practices, and recommend their adoption where applicable. • Collaborate with key stakeholders including business leaders IT teams and external partners to understand requirements prioritize initiatives and ensure alignment with organizational goals. • Develop architectural blueprints and technical designs for Salesforce implementations, including integrations with other systems. • Ensure adherence to best practices in design, development, and deployment of Salesforce solutions Communicate architectural decisions, plans, goals, and strategies effectively to both technical and non-technical stakeholders. • Conduct regular code reviews and ensure adherence to coding standards and architectural guidelines.

United States
Full TimeRemoteTeam 501-1,000Since 2000H1B Sponsor

• Partner with portfolio company technology and product teams to identify and prioritize agentic AI use cases aligned with their product roadmaps and business objectives. • Lead end-to-end delivery of AI agent implementations using Vista’s Agentic Factory methodology, targeting MVP delivery in under a quarter. • Run rapid proof-of-value sprints that demonstrate measurable business impact, then guide portfolio companies from prototype through production deployment. • Lead AI maturity assessments across the portfolio using Vista’s scoring rubrics, identifying gaps and creating tailored improvement roadmaps. • Conduct architectural reviews and provide recommendations for optimizing AI platform performance, scalability, and security. • Design scalable, resilient, and secure AI platforms leveraging public cloud AI services (AWS, Azure, GCP) while maintaining vendor independence through thoughtful abstraction layers.

California + 3 moreAll locations: California | Illinois | New York | Texas
$200K - $315K / year
YDUQS logo

Backend Development Specialist

YDUQS

A educação do Século XXI

Backend Engineer111 days ago
Full TimeRemoteTeam 10,001+Since 2019H1B No Sponsor

• Develop and maintain scalable, highly available microservices using Node.js and AWS; • Ensure efficient integration between internal and external systems using SNS, SQS, Lambda and other relevant technologies; • Work with Oracle, PostgreSQL, MySQL, MongoDB and DynamoDB databases; • Design, implement and manage scalable, resilient solutions in production environments.

Brazil
Job Closed
Backend Engineer111 days ago
Full TimeRemoteTeam 10,001+Since 2019H1B No Sponsor

• Desenvolver e manter microsserviços escaláveis e de alta disponibilidade utilizando Node.js e AWS; • Garantir a integração eficiente entre sistemas internos e externos, usando SNS, SQS, Lambda e outras tecnologias relevantes; • Trabalhar com bancos de Oracle, PostgreSQL, MySQL, MongoDB e DynamoDB; • Projetar, implementar e gerenciar soluções escaláveis e resilientes em ambientes de produção.

Brazil
Job Closed