Hanover Insurance Group has consistently been named one of America’s Best Midsized Employers by Forbes magazine and one of Business Insurance magazine's Best
Pipelines Operations - Data Engineer
Location
Massachusetts
Posted
62 days ago
Salary
0
Seniority
Senior
Job Description
Pipelines Operations - Data Engineer
Hanover Insurance Group
Title: PL Ops - Data Engineer ) Location: Worcester, MA, USA United States Remote Req #19719 Job Description: Our Personal Lines Operations team is currently seeking a Data Engineer II in our Worcester office. This is a full-time, exempt role. Fully remote arrangement will be considered for candidates with strong qualifications. POSITION OVERVIEW: The Hanover Insurance Group is hiring a Personal Lines Operations Data Engineer to own the day-to-day reliability and enhancement of our Azure Data Factory (ADF) pipelines and on-prem SQL Server data environment. This role builds and improves ETL workflows that populate SQL tables used by analysts to create Power BI semantic models and reports. The position is split approximately 50/50 between production operations/support and new development/enhancement, while also advancing platform maturity through improved monitoring/alerting, data quality validation, and formalized release practices. What You'll Own: Production support and daily health of scheduled ADF pipelines. Build new pipelines and enhance existing pipelines to improve resiliency, maintainability, and scalability. Implement data validation controls, improve monitoring/alerts, and help define SLAs for data freshness and availability. Establish foundational SDLC practices for data engineering (Git usage, Dev→Prod promotion practices, and a more formal release process). Coordinate cross-team dependencies where upstream internal ETL timelines affect downstream pipeline completion; design dependency-aware orchestration and readiness checks. Contribute to future-state Azure data strategy recommendations (e.g., Data Lake/Blob Storage, notebooks) and support long-term migration planning from on-prem SQL Server to cloud databases (timeline not yet defined). KEY RESPONSIBILITIES: Azure/Fabric Data Factory Engineering Design and maintain curated SQL tables/views used for analytics and reporting; optimize for refresh performance and downstream usability. Develop and maintain ADF pipelines (Copy activities and Data Flows) with consistent patterns for logging, error handling, retries, and notifications. Implement parameterization and reusable components to reduce duplication and speed enhancements. Implement incremental load and backfill strategies appropriate to volume. Production Support & Reliability Monitor daily pipeline execution and triage incidents quickly to restore successful processing. Perform root-cause analysis for failures and recurring issues; implement preventative controls and standardized patterns. Create and maintain operational documentation/runbooks for critical pipelines and common support scenarios. Data Quality, Validation & Trust Build automated validation routines and reconciliation checks (row counts, totals, null/duplicate thresholds, schema drift detection, anomaly flags). Partner with analysts and stakeholders to define key business rules and quality thresholds for trusted reporting datasets. Document data definitions, transformations, and lineage to improve transparency and troubleshooting. Stakeholder Collaboration & Dependency Management Work directly with internal Operations Data Analysts, business stakeholders and external partner analysts to gather requirements and deliver datasets that enable robust Power BI semantic models, reporting and other analytical solutions. Coordinate with upstream internal teams to align dependency readiness signals and timelines; implement orchestration controls to prevent downstream failures. Provide technical coaching and mentoring to less experienced team members; including the usage of best practices. KEY MEASURES OF SUCCESS: Pipelines meeting SLA / on-time delivery Pipeline success rate and reduced manual intervention Time-to-detect/time-to-resolve ETL failures (MTTD/MTTR) Improvements in query/runtime performance and cost efficiency Reduction in recurring data quality defects; completeness/accuracy checks passing Documentation coverage (runbooks for critical pipelines, data dictionary completeness) Required Qualifications Bachelor's Degree preferred in a related field. 6+ years of experience in data engineering, ETL/ELT, or related roles. Hands-on experience building and supporting production pipelines in Azure Data Factory. Strong SQL/T-SQL skills and experience with SQL Server environments supporting analytics/reporting workloads. Advanced data modeling Experience integrating data from multiple sources including databases, flat files/SFTP, and APIs/vendor feeds. Familiarity with Azure data platform components (Data Lake/Blob Storage, notebooks) and/or cloud migration planning, or similar platforms. Strong troubleshooting, root-cause analysis, and operational ownership mindset. Ability to work directly with stakeholders to gather requirements and translate them into technical solutions. Strong communication and documentation skills: Data engineers are often called to present their findings or translate the data into an understandable document. You will need to write and speak clearly, easily communicating complex ideas. Preferred Qualifications Experience establishing SDLC practices for data teams (Git, CI/CD, release management). Experience implementing monitoring/alerting and operational dashboards for data pipelines. Python and/or PowerShell for automation, data validation, and operational tooling. Insurance or operations/customer service experience Familiarity with prompt-based programming and tools (GitHub CoPilot) This job posting provides cursory examples of some of the job duties associated with this position. The examples provided are not complete, and the position may entail other essential and job-related functions and responsibilities that employees will be required to perform.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Engineer
SleekOne-stop-shop for entrepreneurs to start & grow their business in Australia, Hong Kong, Singapore and the UK.
• You will design, build, and maintain robust data pipelines on databricks and AWS infrastructure that power analytics and reporting capabilities across the organization. • Design and implement scalable ETL/ELT pipelines using both batch and streaming patterns • Build and maintain ingestion workflows from diverse sources (databases, APIs, event streams) • Implement Change Data Capture (CDC), full-load, and incremental ingestion strategies. • Develop and manage data workflows using Apache Airflow for orchestration. • Configure and manage data ingestion connectors using Airbyte. • Work with Databricks to build and optimize data engineering workloads on the Lakehouse platform. • Write and optimize complex SQL queries. • Solid hands-on experience on dbt with databricks, build modular, testable dbt models for data transformation. • Develop and maintain data models in staging, intermediate, and mart layers following data warehousing best practices. • Containerize data services and applications using Docker & EKS. • Ensure data quality, observability, and reliability across the data platform. • Document pipelines, models, and data dictionaries to maintain platform knowledge.
Senior Data Engineer - Databricks
EverforthEverforth Apex, a division of Everforth and formerly Apex Systems, an IT staffing and workforce solutions firm, provides recruiting and staffing services to lar
Senior Data Engineer - Databricks Location: Charlotte, NC, US Employee Type: Contract Job Type: Pay Range: $65 - $75 per hour Job Description: Job#: 3035030 Job Description: Sr. Data Engineer - Databricks Location: Charlotte, North Carolina- 3X A WEEK HYBRID Employment Type: Contract Role Overview We are seeking a Sr. Data Engineer with expertise in Databricks. This role involves the design, development, and maintenance of scalable data pipelines and solutions. The position requires a focus on workflow orchestration, performance optimization, and data governance within the Databricks ecosystem. Key Responsibilities - Build and schedule jobs using Databricks Workflows, Asset Bundles, or orchestrators like Airflow. - Optimize Spark jobs for performance, including cluster sizing, caching, AQE, and broadcast joins, reduce costs, and troubleshoot via the Spark UI. - Design and implement dimensional models, slowly changing dimensions (SCDs), and curated datasets for BI and Databricks SQL consumption. - Implement data governance and security using Unity Catalog, including schemas, lineage, access controls, and row/column-level security, and manage secrets. - Manage CI/CD pipelines and infrastructure using version control (Git) and deployment tools like Asset Bundles or Terraform. - Establish data quality frameworks using tools such as DLT or Great Expectations, along with monitoring and alerting systems. Required Qualifications Technical Skills: Databricks Compensation & Benefits The anticipated pay range for this position is $65.00 to $75.00 per hour. Please note that the final pay rate will be determined by a variety of factors, including the candidate's experience and qualifications. Benefits may be available to eligible employees. This employer is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or protected veteran status and will not be discriminated against on the basis of disability. Everforth Apex is a world-class IT services company that serves thousands of clients across the globe. When you join Everforth Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development opportunities, and a comprehensive benefits package. Everforth Apex Benefits Overview: Everforth Apex offers a range of supplemental benefits, including medical, dental, vision, life, disability, and other insurance plans that offer an optional layer of financial protection. We offer an ESPP (employee stock purchase program) and a 401K program which allows you to contribute typically within 30 days of starting, with a company match after 12 months of tenure. Everforth Apex also offers a HSA (Health Savings Account on the HDHP plan), a SupportLinc Employee Assistance Program (EAP) with up to 8 free counseling sessions, a corporate discount savings program and other discounts. In terms of professional development, Everforth Apex hosts an on-demand training program, provides access to certification prep and a library of technical and leadership courses/books/seminars once you have 6+ months of tenure, and certification discounts and other perks to associations that include CompTIA and IIBA.
Senior Data Analytics Engineer - Data Insights
AppfireAppfire is a computer software company that has been recognized as a “global authority” on Atlassian products and software. As an employer, the company aims
Title: Senior Data Analytics Engineer - Data Insights Location: Spain United States Job Description: At Appfire, we believe that great work happens when people get to choose how they work. After 20 years of creating software that empowers teams to break silos and collaborate seamlessly, we've learned that one size does not fit all. That's why at Appfire, you choose. Choose to work where you thrive: Whether from home, in one of our offices, or while exploring the world, you decide where you're most productive. From hardware and software to access to two decades of collective wisdom on working effectively in a remote-first company, we've got you covered. Choose to balance your life without compromise: Plan your workday around your needs and what matters most to you. Enjoy flexible time off without the hassle of complicated approvals. From day one, we trust you to deliver quality work without sacrificing your personal life, hobbies, loved ones, and well-being. Choose to grow on your terms: Take charge of your professional growth with access to online learning platforms, facilitated training, leadership programs, and internal hackathons. Collaborate with a global team to gain fresh perspectives. When you're ready for a new challenge, our internal mobility program is here to support your journey. No one knows you better than you do. So join us and customize your experience. Choose how you want to work. Choose Appfire. About the role: Appfire is seeking an experienced full-time Senior Analytics Engineer! In this role you will build, define and activate Appfire's company-wide data asset (Snowflake) from ground up data sources. You will work with stakeholders across Appfire to integrate, test, connect, and build brand new data and metrics within our Snowflake metrics layers, achieving a trusted, accessible and well-defined view of information across the company. Our Analytics Engineers build metric foundations for reporting, analysis and experimentation, pushing the boundaries of decisions across the company under an aligned view of the world. They leverage their expertise in metric development, DBT, data modeling, SQL and Python. Using these skills you will transform data from raw data warehouse into data artifacts powering impactful analytic use cases (e.g. metrics, dashboards, adhoc analysis and self-service solutions) that enable and push the boundaries of decisions across the company. As an Analytics Engineer you will sit at the intersection of business and data, working collaboratively to enable and achieve meaningful outcomes by evolving our ecosystems to scale impact. We're growing quickly and have a lot to accomplish. What You Will Do: - Understand decision needs by interfacing with fellow Analytics Engineers, Data Engineers, Data Analysts and Business Partners - Architect, build, and launch efficient & reliable data models and pipelines in partnership with Data Engineering, leveraging DBT and Snowflake - Design and implement metrics and dimensions enabling self-service analysis and structured views of Appfire's KPI's that scale trust, accountability and decision impact - Define and share best practices on metric, dimension, and data model development and definitions for analytics use cases What We Would Like to See: - 5+ years of relevant industry experience - Passion for high data quality and impactful analytics engineering work - Expert SQL skills for data modeling and query optimization. - Extensive experience with self service, advanced analytics, semantic and metrics layers - 3+ Years of Experience with Python and DBT - Strong understanding of data warehousing concepts and best practices. - Efficient problem-solving and analytical skills - Proven ability to succeed in both collaborative and independent work environments at a remote-work company - Proven experience managing concurrent complex data projects from start to finish - Continuous learner with an eye on industry trends and new technologies. - Excellent communication and documentation skills with the ability to explain technical concepts to a non-technical audience. - Ability to effectively leverage AI tools and techniques to enhance dbt development, increase efficiency, and strengthen documentation. Preferred Qualifications: - Experience with dbt (Data Build Tool) and its implementation for data transformation and modeling. - Proven experience with data wrangling and designing/developing experience owning end to end SDLC and implementation, with full ownership of stakeholder management and expectation setting, prioritization, development and delivery. - Proficiency in source code management tools such as GIT, BitBucket, etc. - Experience working with Cloud Technologies (AWS/GCP/Azure) - Experience working with various semi structured data formats (ex. JSON) - Experience building data solutions for future scale and analytics use-cases - 3+ years of development experience on industry standard BI tools - ML analytics engineering experience a plus - Marketing Analytics experience a plus Benefits & Perks Equity Every Appfire team member is eligible for company equity, fostering a true sense of ownership and connection to our growth. Time Off & Flexibility - 25 days of annual leave per calendar year (January-December) - Up to 10 unused vacation days may be carried over to the following year and must be used by December 31 - Reduced summer hours to support better work-life balance - Flexible bank holidays, allowing employees to exchange one public holiday for another - This role is fully remote within Spain, with the option to work from our Bilbao office whenever preferred Learning & Development Grow with Appfire University - our custom, on-demand learning platform designed to support continuous learning and professional development. Health & Wellbeing - Private health insurance - Family members may also be added at a discounted rate through payroll deduction - €400 gross annual sport allowance for gym memberships, outdoor activities, sports equipment, running gear, and other wellness-related expenses Work-from-Home Support Appfire provides €50 per month to help cover home office and remote work expenses. Community & Volunteering Employees receive 3 fully paid volunteering days each year through Appfire Town, our Corporate Social Responsibility (CSR) program supporting local communities. About Appfire A people-first approach to business Since its inception, Appfire has been a remote-first company. With 850+ employees (who we call fireflies) across 28 countries, we foster an environment where everyone is respected. We invest in team members by ensuring they grow professionally and personally. Making an impact At Appfire, CSR means embedding purpose, responsibility, and impact into everything we do. We use our people, products, and partnerships to make a meaningful difference in the world, we act responsibly as a business, and we empower communities while strengthening our skills and culture and fostering belonging across Appfire. In 2015, Appfire joined the Pledge 1% network of organizations committed to philanthropy. Appfire has since grown our Pledge 1% program to include all four pledge types - product, profit, equity, and employee time. We were among the first to do this, and we're proud that Pledge 1% is part of our evolution. Our business Appfire mission to equip and connect every team so they can plan and deliver their best work. We are committed to building a durable, multi-generational business, and to evolving, innovating, and scaling in a way that ensures stability and opportunity for years to come What's our secret sauce? - We follow teams. We do our research and build software that solves real-life collaboration challenges while being easy to implement and a joy to use. We're proud to support over 20,000 customers and growing, including 55% of the Fortune 500. From the entertainment delivered by Netflix to the devices crafted by Samsung, and Dell Technologies and the financial transactions handled by Visa, Edward Jones, and US Bank, Appfire's technology is indispensable. Our products also play a pivotal role in streamlining operations and fostering innovation at companies like Tesla and significant institutions such as NASA, Boeing, and many more. - We enhance. Our software is designed to give developers, knowledge workers, and teams the ability to extend and get greater value from the platforms they've invested in and enjoy. So far, our solutions extend and enhance the capabilities offered by Atlassian, Microsoft, monday.com, and Salesforce. - We build bridges and invest in our partners. Appfire's success is underscored by its channel program as its primary path to market. Today, Appfire has a dedicated Channel team supporting 800+ channel partners. - We make security and privacy a priority, but we also keep it simple for our customers. We've achieved International Organization for Standardization (ISO) 27001 and ISO 27017 and System and Organization Controls (SOC) SOC 2, Type I and SOC 2, Type II certifications. Our award-winning Appfire Trust Center, offers our customers, partners, and prospects the latest security, privacy, and compliance information, including pre-completed questionnaires (CAIQ, SIG, and VSA) with an accelerated NDA process and just one EULA to cover it all. Market recognition Appfire has been consistently recognized for company growth, culture, corporate social responsibility, and product excellence and has been included among the Deloitte Technology Fast 500, Inc. Best Workplaces, BuiltIn Best Places to Work, and Inc. 5000. Learn more about our accomplishments, which would not be possible without our team members, partners, and customers: https://appfire.com/awards. Equal Employer Opportunity (EEO) Appfire is an equal opportunity employer and does not discriminate based on race, color, religion, sex (including pregnancy), sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, veteran status, or any other protected characteristic as defined by applicable law. Our commitment extends to all employment practices, including recruitment, hiring, training, promotion, compensation, benefits, and termination.
Data Engineer
SleekOne-stop-shop for entrepreneurs to start & grow their business in Australia, Hong Kong, Singapore and the UK.
Role Description We are looking for a skilled and passionate Data Engineer to join our growing Data Platform team. Mission: - Design, build, and maintain robust data pipelines on Databricks and AWS infrastructure that power analytics and reporting capabilities across the organization. Key responsibilities: - Design and implement scalable ETL/ELT pipelines using both batch and streaming patterns. - Build and maintain ingestion workflows from diverse sources (databases, APIs, event streams). - Implement Change Data Capture (CDC), full-load, and incremental ingestion strategies. - Develop and manage data workflows using Apache Airflow for orchestration. - Configure and manage data ingestion connectors using Airbyte. - Work with Databricks to build and optimize data engineering workloads on the Lakehouse platform. - Write and optimize complex SQL queries. - Solid hands-on experience on dbt with Databricks, build modular, testable dbt models for data transformation. - Develop and maintain data models in staging, intermediate, and mart layers following data warehousing best practices. - Working knowledge of AWS services like S3, Lambda, EC2, IAM, etc. - Containerize data services and applications using Docker & EKS. - Ensure data quality, observability, and reliability across the data platform. - Document pipelines, models, and data dictionaries to maintain platform knowledge. Qualifications - 3+ years of professional experience in data engineering. - Strong understanding of data platform architecture: Lakehouse, Data Warehouse, Data Lake patterns. - Hands-on experience with ETL/ELT design patterns including batch processing and stream processing. - Familiarity with ingestion patterns: full load, incremental, CDC, event-driven. - 5+ years of hands-on experience as a Database Administrator in production environments. - Strong understanding of database internals — storage engines, transactions, isolation levels, locking, MVCC, query planners. - Proven experience supporting mission-critical OLTP workloads with high availability requirements. - Solid scripting skills in Bash and/or Python for automation. Requirements - Experience building data pipelines on Databricks (Delta Live Tables, Jobs, Notebooks). - Proficiency with PySpark or Spark SQL for large-scale data processing. - Familiarity with Delta Lake concepts: ACID transactions, time travel, schema evolution. - Proficiency with Apache Airflow — authoring, scheduling, and monitoring DAGs. - Experience with Airbyte for managing source-to-destination data connectors. - Hands-on experience administering MongoDB (self-managed and/or Atlas). - Replica set configuration, sharding, indexing strategies, and aggregation pipeline tuning. - Backup, restore, and disaster recovery using mongodump/mongorestore or Ops Manager. - Strong SQL skills — query optimization, window functions, CTEs, and complex joins. - Experience with dbt (data build tool) for transformation, testing, and documentation. - Practical experience with AWS services (S3, Lambda, IAM, CloudWatch, etc). Benefits - Humility and kindness: A culture of not taking ourselves too seriously and being able to laugh. - Flexibility: Work from home and the option to work fully remote from anywhere in the world for 1 month each year. - Financial benefits: Competitive market salaries, generous paid time off, and access to a flexi benefits scheme. - Personal growth: Responsibility and autonomy with a range of internal and external training programs. Company Description Sleek launched in 2017 and now has around 15,000 customers across our offices in Singapore, Hong Kong, Australia, and the UK. We have around 500 staff with an intact startup mindset. We have recently raised Series B financing off the back of >70% compound annual growth in Revenue over the last 5 years. Sleek has been recognised by The Financial Times, The Straits Times, Forbes, and LinkedIn as one of the fastest-growing companies in Asia. Backed by world-class investors, we are on track to be one of the few cash flow positive, tech-enabled unicorns based out of Asia Pacific.


