Tech is in our DNA
Senior Data Engineer
Location
Virginia + 1 moreAll locations: Virginia | Malta
Posted
78 days ago
Salary
0
Seniority
Senior
Job Description
Senior Data Engineer
Ventures Lab
• Design and implement robust, scalable data architecture to support the collection, storage, and processing of large volumes of igaming and customer data • Build and maintain efficient ETL and ELT processes to ensure seamless data ingestion from multiple sources into our data lake, DWH, and final consumers • Create and maintain well-structured data models that ensure accuracy, integrity, and optimal performance for analytical use cases following Kimball best practices • Collaborate with cross-functional teams to integrate data across systems, with several types of integration as API, Kafka, DB extractions, Webscrapping, Email, etc. • Develop and manage real-time data pipelines using streaming technologies (e.g., Kafka) to support time-sensitive use cases • Enforce data governance policies and best practices, ensuring high standards for data quality, security, and regulatory compliance • Continuously monitor, troubleshoot, and optimize data pipelines, queries, and infrastructure for speed, efficiency, and reliability • Support and expand production-grade orchestration workflows using Apache Airflow • Set up and maintain monitoring, alerting, and observability for data pipelines and systems using tools like Elementary, Datadog, Prometheus, or Grafana • Develop and maintain CI/CD pipelines for data projects using Bitbucket Pipelines or GitHub Actions, ensuring streamlined deployment and testing processes • Follow best practices in Git-based version control and work collaboratively in a modern DevOps environment.
Job Requirements
- 5+ years of experience with Python, preferably including 2+ years working with Airflow
- 5+ years of experience with SQL in big data environments, preferably including 2+ years with dbt
- 4+ years in cloud environments, preferably with 2+ years each in Snowflake and AWS
- 2+ years of experience with data streaming platforms, preferably on Kafka
- 2+ years of hands-on experience with containerization (Docker, Kubernetes)
- 3+ years managing CI/CD pipelines, ideally using Bitbucket Pipelines or GitHub Actions
- Deep understanding of data governance, data quality, and data lineage principles
- Proficient in using monitoring and alerting tools such as Datadog, Prometheus, or Grafana
- Solid experience with version control tools (Git) and IaaS (e.g., Terraform)
- Strong communication and problem-solving skills, with a proactive and self-driven attitude.
Benefits
- Competitive salary synonymous with skills and experience
- Performance and bonus structure dependent on achievement of set targets and personal performance
- The opportunity to making a real impact at a time of rapid growth
- Consultancy B2B contract on a full time basis with 25 days PTO
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Are you interested in making a world of difference in cancer care? Cancer strikes more than 10 million people worldwide each year. As the leading medical society representing doctors who care for people with cancer, the American Society of Clinical Oncology (ASCO) is committed to conquering cancer through research, education, and promotion of the highest quality care. Who we are: ASCO is a flexible, high-performance membership organization where employees collaborate to support our mission through evidence, care, and impact. Together with Conquer Cancer, the ASCO Foundation, we foster a culture that prioritizes customer-centricity, emphasizes teamwork, and commits to quality. Our culture, ASCO Works – Our Way of Working, has long enabled workplace flexibility and embraced technology to help us achieve balance. ASCO offers competitive salaries, an excellent benefits package, and opportunities to participate in professional development programs. To learn what it’s like to work at ASCO, click here. Who we are looking for: ASCO’s Information Technology Department is hiring a Data Product Engineer, Level 1. The successful hire will have an integral role in the seamless transfer of high-quality data from source systems to applications and decision-makers, empowering process improvement, informing decision-making, and supporting alignment with our organizational mission. If you’re looking for an exciting opportunity to have a key role in designing and overseeing the technical elements that provide data-driven value to end-users, apply today! Remote candidates welcome or at primary location in Alexandria, VA. Responsibilities - Collaborate with cross-functional teams to build, maintain, publish, and monitor Data Products across AWS, GCP, and Azure environments to serve the needs of both Operational and Analytical customers, ensuring high-quality standards and seamless multi-cloud integration. - Utilize a diverse set of technical skills to translate raw data from its original context into formats that are suitable and accessible to product customers. This process should involve data cleansing, transformation, and integration to ensure data accuracy and relevance. - Act as the subject matter expert for the data you manage, gaining in-depth knowledge of both the technical intricacies and the business implications. Facilitate informed decision-making within the organization by providing valuable insights and expertise. - Maintain comprehensive documentation of data processes, methodologies, and best practices. Proactively share your knowledge and insights with colleagues and stakeholders to promote data understanding and utilization across the organization. - Ensure the responsible management of data in alignment with ASCO's security requirements and standards. Implement and enforce data security protocols and practices to safeguard sensitive information, maintaining the highest levels of data integrity. - Apply Software Engineering principles, including but not limited to Version Control, Continuous Integration/Continuous Deployment (CI/CD), Test-Driven Development (TDD), and Data-Oriented Development (DOD), to seamlessly integrate Data Products into ASCO's broader Digital Ecosystem. This integration should optimize data utilization and decision-making processes. Required Education and Experience - Bachelor’s degree in related field or equivalent years of training/experience. - 2+ years of practical experience in building, delivering, and supporting work within technology systems with examples of manipulating and serving data to different types of customers. Preferred Education and Experience - Appropriate certifications or other evidence which demonstrates professional capability in the core competencies. Competencies - Proficiency in data manipulation, transformation, and integration techniques. - Strong technical expertise in data engineering, data modeling, and database management. - Knowledge of data security practices and the ability to implement them effectively. - Familiarity with Software Engineering principles and tools for managing data pipelines and products. - Familiarity with writing modular Python scripts using various libraries (i.e. Pandas) to extract data from APIs or flat files, apply basic cleaning and schema casting, and load the results into relational or no-SQL databases. - Experience working with version control tools - Familiarity working with AWS and GCP - Excellent communication and collaboration skills to work effectively with diverse teams and stakeholders. - Ability to translate complex technical concepts into understandable terms for non-technical audiences. - Strong problem-solving skills and meticulous attention to detail. - Ability to manage multiple tasks and priorities within a dynamic work environment. - Dedication to ASCO's mission and values, with a commitment to enhancing the availability and utility of high-quality data for organizational improvement. ADA/Physical Requirements Must lift up to 30 lbs. Extended periods seated or standing at a desk. High use of computer and other office technology equipment. Travel 4-6 days per year Generous Benefits Package: - Hybrid Work Environment - Open Leave Policy - Paid Family Leave - 13 Paid Holidays per Calendar Year - Staff Appreciation Days - 401(k): 7.5% Employer Contribution - Medical/Dental/Vision - Employee Assistance Program - Fertility and Family Forming - Healthcare Concierge - Flexible Spending Account(s) - Healthcare Savings Account - Disability and Life Insurance Applications are accepted and reviewed on a rolling basis. The job posting will remain active throughout the candidate application evaluation process. The American Society of Clinical Oncology (ASCO) is an Equal Employment Opportunity (EEO) employer. It is the policy of the Company to provide equal employment opportunities to all qualified applicants without regard to race, color, national origin, sex, and religion.
Bright Vision Technologies is a forward-thinking software development company dedicated to building innovative solutions that help businesses automate and optimize their operations. We leverage cutting-edge technologies to create scalable, secure, and user-friendly mobile applications. As we continue to grow, we’re looking for a skilled Kafka Engineer to join our dynamic team and contribute to our mission of transforming business processes through technology. This is a fantastic opportunity to join an established and well-respected organization offering tremendous career growth potential. We are looking for OPT/CPT/H4 EAD/TN/E3 or any other Non-immigrant visa people who are looking for an H1B sponsorship for the year 2027 quota. Company: Bright Vision Technologies ( www.bvteck.com ) Job Title: Kafka Engineer Job Location: Bridgewater, New Jersey – 08807 Onsite/Hybrid: Remote FULL-TIME ROLE WITH BRIGHT VISION Job Description: Environment: Apache Kafka, Kafka Streams, Kafka Connect, Schema Registry, Event-Driven Architecture, Real-Time Data Processing, Apache Avro, REST APIs, Microservices Integration, Cloud Platforms (AWS / Azure / GCP), Security & Authentication, Monitoring & Alerting, Linux, BUT STRICTLY NO C2C/1099/3RD PARTY COMPANIES. ALL OUR ROLES ARE W2 AND NO 3RD PARTY BROKERING PLEASE. For every role, a coding test is required, so apply only if you are confident and technically strong. We prefer at least 3 to 5 years real time experience. If you are a Kafka Engineer with the above skills and are looking for an H-1B sponsorship this year, please send your resume immediately to boon@bvteck.com We are committed to providing equal employment opportunities and fostering an inclusive work environment. We encourage applications from all qualified individuals regardless of race, ethnicity, religion, gender identity, sexual orientation, age, disability, or any other protected status. If you require accommodations during the recruitment process, please let us know. Position offered by “No Fee agency.” Equal Employment Opportunity (EEO) Statement Bright Vision Technologies (BV Teck) is committed to equal employment opportunity (EEO) for all employees and applicants without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, veteran status, or any other protected status as defined by applicable federal, state, or local laws. This commitment extends to all aspects of employment, including recruitment, hiring, training, compensation, promotion, transfer, leaves of absence, termination, layoffs, and recall. BV Teck expressly prohibits any form of workplace harassment or discrimination. Any improper interference with employees' ability to perform their job duties may result in disciplinary action up to and including termination of employment.
Bright Vision Technologies is a forward-thinking software development company dedicated to building innovative solutions that help businesses automate and optimize their operations. We leverage cutting-edge SAP data migration and transformation technologies to ensure smooth, accurate, and scalable enterprise data transitions. As we continue to grow, we’re looking for a skilled SAP Data Migration Engineer (LTMC / SLT / LVM) to join our dynamic team and contribute to our mission of transforming business processes through technology. This is a fantastic opportunity to join an established and well-respected organization offering tremendous career growth potential. We are looking for OPT/CPT/H4 EAD/TN/E3 or any other Non-immigrant visa people who are looking for an H1B sponsorship for the year 2027 quota. Company: Bright Vision Technologies ( www.bvteck.com ) Job Title: SAP Data Migration Engineer (LTMC / SLT / LVM) Job Location: Bridgewater, New Jersey - 08807 Onsite/Hybrid: Remote FULL-TIME ROLE WITH BRIGHT VISION Environment: SAP S/4HANA Data Migration, LTMC, LTMOM, SAP SLT, SAP LVM, Data Cleansing & Validation, Legacy System Migration, SAP Migration Cockpit, ETL Processes, IDocs, BAPIs, SAP HANA, SQL, Data Reconciliation, Cutover Planning, SAP Transport Management, Agile methodologies BUT STRICTLY NO C2C/1099/3RD PARTY COMPANIES. ALL OUR ROLES ARE W2 AND NO 3RD PARTY BROKERING PLEASE. For every role, a coding test is required, so apply only if you are confident and technically strong. We prefer at least 3 to 5 years real time experience. If you are a SAP Data Migration Engineer (LTMC / SLT / LVM) with the above skills and are looking for an H-1 B sponsorship this year, please send your resume immediately to Venkat.r@bvteck.com. We are committed to providing equal employment opportunities and fostering an inclusive work environment. We encourage applications from all qualified individuals regardless of race, ethnicity, religion, gender identity, sexual orientation, age, disability, or any other protected status. If you require accommodations during the recruitment process, please let us know. Position offered by “No Fee agency.” Equal Employment Opportunity (EEO) Statement Bright Vision Technologies (BV Teck) is committed to equal employment opportunity (EEO) for all employees and applicants without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, veteran status, or any other protected status as defined by applicable federal, state, or local laws. This commitment extends to all aspects of employment, including recruitment, hiring, training, compensation, promotion, transfer, leaves of absence, termination, layoffs, and recall. BV Teck expressly prohibits any form of workplace harassment or discrimination. Any improper interference with employees' ability to perform their job duties may result in disciplinary action up to and including termination of employment.
Role - Spark Engineer Bright Vision Technologies is a forward-thinking software development company dedicated to building innovative solutions that help businesses automate and optimize their operations. We leverage cutting-edge distributed data processing technologies to build scalable, high-performance analytics platforms. As we continue to grow, we’re looking for a skilled Spark Engineer to join our dynamic team and contribute to our mission of transforming business processes through technology. This is a fantastic opportunity to join an established and well-respected organization offering tremendous career growth potential. We are looking for OPT/CPT/H4 EAD/TN/E3 or any other Non-immigrant visa people who are looking for an H1B sponsorship for the year 2027 quota. Company: Bright Vision Technologies ( www.bvteck.com ) Job Title: Spark Engineer Job Location: Bridgewater, New Jersey - 08807 Onsite/Hybrid: Remote FULL-TIME ROLE WITH BRIGHT VISION Job Description: Environment: Apache Spark, PySpark, Spark SQL, Scala / Python, Hadoop ecosystem (HDFS, YARN), Kafka, DataFrames & Datasets, ETL pipelines, SQL, NoSQL databases, Cloud Platforms (AWS EMR / Azure Databricks), Performance tuning, Linux, CI/CD pipelines, Git, Agile methodologies BUT STRICTLY NO C2C/1099/3RD PARTY COMPANIES. ALL OUR ROLES ARE W2 AND NO 3RD PARTY BROKERING PLEASE. For every role, a coding test is required, so apply only if you are confident and technically strong. We prefer at least 3 to 5 years real time experience. If you are a Spark Engineer with the above skills and are looking for an H-1 B sponsorship this year, please send your resume immediately to Sairaj@bvteck.com. Best Regards Sairaj Sales Manager Bright Vision Technologies Email: Sairaj@bvteck.com (O): (908) 505-3454 | ext 1011 Address: #991 Route 22 West, Suite 200, Bridgewater, New Jersey - 08807 we are committed to providing equal employment opportunities and fostering an inclusive work environment. We encourage applications from all qualified individuals regardless of race, ethnicity, religion, gender identity, sexual orientation, age, disability, or any other protected status. If you require accommodations during the recruitment process, please let us know. Position offered by “No Fee agency.” Equal Employment Opportunity (EEO) Statement Bright Vision Technologies (BV Teck) is committed to equal employment opportunity (EEO) for all employees and applicants without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, veteran status, or any other protected status as defined by applicable federal, state, or local laws. This commitment extends to all aspects of employment, including recruitment, hiring, training, compensation, promotion, transfer, leaves of absence, termination, layoffs, and recall. BV Teck expressly prohibits any form of workplace harassment or discrimination. Any improper interference with employees' ability to perform their job duties may result in disciplinary action up to and including termination of employment. Equal Employment Opportunity (EEO) Statement Bright Vision Technologies (BV Teck) is committed to equal employment opportunity (EEO) for all employees and applicants without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, veteran status, or any other protected status as defined by applicable federal, state, or local laws. This commitment extends to all aspects of employment, including recruitment, hiring, training, compensation, promotion, transfer, leaves of absence, termination, layoffs, and recall. BV Teck expressly prohibits any form of workplace harassment or discrimination. Any improper interference with employees' ability to perform their job duties may result in disciplinary action up to and including termination of employment.
