Meet PortPro's drayOS 2. The latest and most premier operating system for drayage carriers.
Director of Software Engineering – Node.js, Web Scraping Expert
Location
California
Posted
111 days ago
Salary
0
Seniority
Lead
Job Description
Director of Software Engineering – Node.js, Web Scraping Expert
PortPro
• Architect, develop, and maintain scalable and distributed web scraping systems using Node.js. • Design and implement data extraction pipelines to process large volumes of structured and unstructured data. • Develop solutions to bypass anti-bot mechanisms, including CAPTCHA handling, session management, fingerprinting, and IP rotation. • Optimize scraping processes for performance, reliability, and efficiency while managing proxy services (residential, datacenter, rotating). • Oversee data storage and processing strategies, ensuring high availability and consistency. • Collaborate with Product, DevOps, and Data Science teams to integrate extracted data into analytics and business applications. • Implement best practices for microservices, API integrations, and real-time data streaming. • Lead the transition to cloud-native, containerized, and serverless architectures for web scraping. • Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.). • Optimize cloud resources (AWS, GCP, or Azure) to support high-throughput scraping. • Manage real-time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks. • Work closely with DevOps teams to optimize CI/CD pipelines, automated deployments, and system scalability. • Lead, mentor, and grow a high-performance engineering team. • Define and execute the technology roadmap, aligning with business objectives. • Foster a culture of continuous learning, collaboration, and innovation. • Implement agile development methodologies (Scrum, Kanban) to optimize project execution. • Ensure code quality, security, and best practices across all engineering efforts.
Job Requirements
- 10+ years of experience in software engineering, with at least 5+ years in web scraping and large-scale data extraction.
- Strong hands-on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation.
- Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques.
- Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs).
- Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.).
- Familiarity with data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.).
- Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).
Related Guides
Related Job Pages
More Backend Engineer Jobs
• Provide technical leadership and vision for Salesforce initiatives, ensuring alignment with overall IT and business strategies. • Lead technical design sessions architect and document technical solutions aligned with client business objectives identify gaps between a customer’s current and desired end states. • Proven track record of successfully leading large scale Salesforce initiatives from concept to delivery. • Collaborate with business stakeholders, project managers, and development teams to gather requirements and translate them into technical solutions. • Design scalable and flexible Salesforce solutions that meet business requirements, ensure security, and adhere to best practices for platform optimization. • Stay updated with the latest Salesforce features, capabilities, and best practices, and recommend their adoption where applicable. • Collaborate with key stakeholders including business leaders IT teams and external partners to understand requirements prioritize initiatives and ensure alignment with organizational goals. • Develop architectural blueprints and technical designs for Salesforce implementations, including integrations with other systems. • Ensure adherence to best practices in design, development, and deployment of Salesforce solutions Communicate architectural decisions, plans, goals, and strategies effectively to both technical and non-technical stakeholders. • Conduct regular code reviews and ensure adherence to coding standards and architectural guidelines.
• Partner with portfolio company technology and product teams to identify and prioritize agentic AI use cases aligned with their product roadmaps and business objectives. • Lead end-to-end delivery of AI agent implementations using Vista’s Agentic Factory methodology, targeting MVP delivery in under a quarter. • Run rapid proof-of-value sprints that demonstrate measurable business impact, then guide portfolio companies from prototype through production deployment. • Lead AI maturity assessments across the portfolio using Vista’s scoring rubrics, identifying gaps and creating tailored improvement roadmaps. • Conduct architectural reviews and provide recommendations for optimizing AI platform performance, scalability, and security. • Design scalable, resilient, and secure AI platforms leveraging public cloud AI services (AWS, Azure, GCP) while maintaining vendor independence through thoughtful abstraction layers.
• Develop and maintain scalable, highly available microservices using Node.js and AWS; • Ensure efficient integration between internal and external systems using SNS, SQS, Lambda and other relevant technologies; • Work with Oracle, PostgreSQL, MySQL, MongoDB and DynamoDB databases; • Design, implement and manage scalable, resilient solutions in production environments.
• Desenvolver e manter microsserviços escaláveis e de alta disponibilidade utilizando Node.js e AWS; • Garantir a integração eficiente entre sistemas internos e externos, usando SNS, SQS, Lambda e outras tecnologias relevantes; • Trabalhar com bancos de Oracle, PostgreSQL, MySQL, MongoDB e DynamoDB; • Projetar, implementar e gerenciar soluções escaláveis e resilientes em ambientes de produção.



