• Operate and evolve the Data Lake: work with data in its Raw, Processed and Refined zones, including deduplication processes, cataloging and storage optimization (Parquet, Iceberg). • Operate and monitor streaming pipelines via Kafka: create topics, connectors, ACLs, credentials and monitor consumer lag for real-time streaming pipelines. • Contribute to the evolution of the Data Platform API: create and maintain modules of our platform that abstract ingestion processes for users and developers (which may be via files, CDC or data streaming), data processing on Big Data platforms (Spark / Redshift) and pipeline orchestration (Airflow). • Investigate and resolve incidents: interpret failures in pipelines, dataset loads, data duplication and cluster errors, and assist in incident resolution. • Use and contribute to Infrastructure as Code (IaC): use Terraform and operate AWS resources under team guidance. • Participate in modernization and integration initiatives with AI tools: use LLMs to build agents and create tools that integrate AI into our systems (MCP Servers). • Documentation and monitoring maintenance: contribute to technical documentation of our tools and architecture. Maintain our observability tools.

Role Description Mindrift is looking for highly skilled Python Data Scraping Engineers to join the Tendem project and drive specialized data scraping workflows within our hybrid AI + human system. In this role, as an AI Pilot – that’s how we refer to this role at Mindrift – you’ll collaborate with Tendem Agents that handle repetitive tasks, while you provide critical thinking, domain expertise, and quality control to deliver accurate and actionable results. This part-time remote opportunity is ideal for technical professionals with hands-on experience in web scraping, data extraction and processing. Key Responsibilities - Own end-to-end data extraction workflows across complex websites, ensuring complete coverage, accuracy, and reliable delivery of structured datasets. - Leverage internal tools (Apify, OpenRouter) alongside custom workflows to accelerate data collection, validation, and task execution while meeting defined requirements. - Ensure reliable extraction from dynamic and interactive web sources, adapting approaches as needed to handle JavaScript-rendered content and changing site behavior. - Enforce data quality standards through validation checks, cross-source consistency controls, adherence to formatting specifications, and systematic verification prior to delivery. - Scale scraping operations for large datasets using efficient batching or parallelization, monitor failures, and maintain stability against minor site structure changes. Qualifications - At least 3 years of relevant experience in data engineering, web scraping, automation, or software development (required). - Bachelor's or Master’s Degree in Engineering, Applied Mathematics, Computer Science, or related technical fields is a plus. - Strong experience in Python web scraping (BeautifulSoup, Selenium or similar), including dynamic content (JS, AJAX, infinite scroll) and APIs via proxies. - Proven ability to extract data from complex structures (hierarchies, archived pages, inconsistent HTML). - Solid background in data cleaning, normalization, and validation, delivering structured datasets (CSV, JSON, Google Sheets). - Hands-on experience with LLMs and AI frameworks to enhance automation and problem-solving. - Strong attention to detail and commitment to data accuracy. - Self-directed work ethic with ability to troubleshoot independently. - A link to GitHub is a plus. - English proficiency: Upper-intermediate (B2) or above (required). Compensation On this project, contributors can earn up to $37 per hour equivalent, depending on their level and pace of contribution. Compensation varies across projects depending on scope, complexity, and required expertise. Please note that other projects on the platform may offer different earning levels based on their requirements. Benefits - Work fully remote on your own schedule with just a laptop and stable internet connection. - Gain hands-on experience in a unique hybrid environment where human expertise and AI agents collaborate seamlessly — a distinctive skill set in a rapidly growing field. - Participate in performance-based bonus programs that reward high-quality work and consistent delivery.

• Act as a trusted technical advisor across client engagements. • Build strong relationships with senior stakeholders, delivery leads and engineering teams. • Translate complex technical concepts into clear business value and delivery outcomes. • Support clients through technical decision-making, platform strategy and roadmap planning. • Lead the design and implementation of enterprise-scale Databricks solutions and modern data platforms. • Provide architectural leadership across cloud-native data engineering initiatives. • Define engineering standards, best practices and scalable delivery approaches. • Drive platform optimisation, resilience and performance improvements. • Lead technical delivery across multiple client engagements within a consultancy environment. • Work closely with cross-functional teams including Data Engineers, Architects, Product Owners and client stakeholders. • Support project planning, technical governance and delivery execution. • Ensure high-quality, scalable and maintainable engineering solutions. • Mentor and guide engineering teams across modern data engineering practices. • Lead technical problem-solving and incident resolution where required. • Drive adoption of engineering best practices, automation and DevOps principles.

• Design, develop, and maintain robust, scalable ETL/ELT data pipelines using Python, SQL, and data processing frameworks including dbt, Matillion, and AWS services. • Implement data quality checks, monitoring, and alerting across all data pipelines to ensure data integrity and reliability. • Optimize existing pipelines for performance, cost-efficiency, and error handling. • Contribute to the design and maintenance of InStride’s data warehouse and data lake solutions, including schema design, data modeling, and indexing strategies in Amazon Redshift. • Ensure data security, HIPAA compliance, and proper handling of protected health information (PHI) within all data infrastructure. • Work closely with data analysts, data scientists, and business intelligence engineers to understand their data requirements and deliver reliable, high-quality data access. • Troubleshoot and resolve complex production data issues with urgency and root-cause rigor. • Develop and maintain clear documentation for data models, pipelines, and data sources to enable self-service analytics. • Participate in code reviews and contribute to technical discussions, bringing a constructive and detail-oriented perspective. • Stay current on emerging data technologies and tools, bringing relevant insights to the team.