Transforming Together. Transforming Experiences.
Data Engineer – Web Scraping, Contract
Location
Thailand
Posted
13 days ago
Salary
0
Seniority
Mid Level
Job Description
Data Engineer – Web Scraping, Contract
CXG
• Maintain and manage website scraping configurations using Python. • Monitor scraping configurations for errors and potential crashes. • Oversee retrieved data to detect potential issues and blockages. • Coordinate with stakeholders to understand scraping task requirements and report issues. • Prepare and share periodic reports on scraping activities with stakeholders. • Develop necessary pipelines to ingest data into the Datalake and perform required transformations.
Job Requirements
- Proven experience in data engineering with expertise in designing and implementing scalable data architectures.
- Strong experience with ETL processes, data modeling, and data warehousing (Airflow & DBT preferred).
- Expertise in database technologies, both relational (SQL) and NoSQL.
- Knowledge of cloud platforms, particularly Azure.
- Solid understanding of data security measures and compliance standards.
- Excellent Python experience for data engineering and automation.
- Strong collaboration skills to work closely with data scientists and analysts.
- Ability to optimize data pipelines for performance and efficiency.
- Ability to build, test, and maintain tasks and projects.
- Experience with version control systems, such as Git.
- Hands-on experience with Airflow and/or DBT.
- Experience with Terraform for infrastructure management.
- Minimum 2 years of experience in a similar role.
- Strong academic background in a relevant field.
- Fluent in English (French is a plus).
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Architect – Databricks, Azure
Interval GroupHigh quality consulting. On demand. Delivered by top professionals.
• Technical leadership and guidance of a team of infrastructure and data engineers • Strategic development and refinement of a forward-looking Data Mesh architecture • Managing data platform scaling and ensuring structural integrity within the cloud ecosystem • Close collaboration with cross-functional teams to support data migrations toward a modern Lakehouse environment
Data Architect
Accenture Federal ServicesWe believe in the power of change, harnessed in ways that matter for our country and communities.
• Use your expertise to drive data and AI architecture and technology decisions • Provide insights that influence the overall strategy and plan for a scalable data solution or product that meets business needs • Educate customers on how to craft highly scalable, flexible, resilient, secure, and cost-effective architectures • Link technology with measurable business value and outcome • Identify and investigate new tools and technologies • Translate client needs into digestible requirements
Data Engineer
decircleTalent Partner for decentralized organizations and projects that are building Web3.
• Dive deep into blockchain data (Ethereum and other L1/L2s) and enrich it with insights from various data providers. • Build pipelines and infrastructure to power a variety of analytics and dashboards. • Work across teams, design new features, and take ownership of sourcing, verifying, and maintaining high-quality data.
• Lead and provide expert support for data collection, data validation, data visualization, and analytics initiatives • Apply disciplined methodologies for the planning, analysis, design, and development of information systems on an enterprise-wide basis or across a business sector • Develop analytical techniques and methodologies to solve complex business and technical problems • Perform strategic systems planning, business information planning, and business analysis • Organize and analyze large volumes of structured and unstructured data sets using data analytical tools • Locate, access, merge, clean, and standardize data from multiple sources, and develop derived metrics • Create and implement data collection and analysis tools using programming languages such as Python, Databricks, SQL, Scala, R, and Java • Design, script, debug, and analyze data engineering solutions • Implement and create machine learning-based tools and processes • Apply distributed and parallel processing technologies (e.g., Spark) to handle big data analytics tasks involving large data volumes • Perform SQL Server data imports from CSV and TXT files • Leverage Excel and Google Suite for data analysis, reporting, and collaboration • Utilize supporting tools and platforms such as Pentaho (data import/transformation), Azure Data Studio, GitHub, and Smartsheet as needed • Document task requirements, work completed, processes, and technical details thoroughly • Communicate effectively with stakeholders across technical and business teams • Operate independently as a subject matter expert in a fast-paced, entrepreneurial environment




