Miratech logo
Miratech

Helping Visionaries Change the World

Data Engineer, Azure Databricks

Data EngineerData EngineerFull TimeRemoteSeniorTeam 501-1,000Since 1989H1B No SponsorCompany SiteLinkedIn

Location

Spain

Posted

63 days ago

Salary

0

Seniority

Senior

Bachelor Degree3 yrs expEnglishAzureCloudETLPySparkSparkSQLSSIS

Job Description

Data Engineer, Azure Databricks

Miratech

• Design, develop, and maintain data pipelines using Azure Databricks • Build and optimize data transformations using PySpark and SQL in Databricks • Implement and maintain Lakehouse architectures using Delta Lake • Develop ETL/ELT pipelines orchestrated through Azure Data Factory • Integrate data from multiple sources into the data platform and analytical layers • Design and maintain data models and data warehouse structures for analytics • Ensure data quality, scalability, and performance of large-scale data processing pipelines • Collaborate with BI teams to support Power BI and reporting platforms • Support and evolve existing SQL Server data platforms and ETL solutions (SSIS) when required • Contribute to the design of modern cloud-based data architectures

Job Requirements

  • 3+ years of experience in Data Engineering or Data Warehouse development
  • Strong experience with Azure Databricks
  • Experience developing data pipelines using PySpark and Spark SQL
  • Solid understanding of distributed data processing and big data concepts
  • Experience working with Delta Lake and Lakehouse architectures
  • Strong SQL skills and experience with SQL Server relational databases
  • Experience building data pipelines using Azure Data Factory
  • Experience handling large datasets and performance optimization
  • Nice to have Experience with Spark optimization techniques (partitioning, caching, cluster tuning)
  • Experience with structured streaming in Databricks
  • Knowledge of CI/CD pipelines for data platforms (Azure Devops)
  • Familiarity with Power BI
  • Experience in migrating from traditional ETL process to cloud architectures

Benefits

  • Culture of Relentless Performance: join an unstoppable technology development team with a 99% project success rate and more than 30% year-over-year revenue growth.
  • Competitive Pay and Benefits: enjoy a comprehensive compensation and benefits package, including health insurance, and a relocation program.
  • Work From Anywhere Culture: make the most of the flexibility that comes with remote work.
  • Growth Mindset : reap the benefits of a range of professional development opportunities, including certification programs, mentorship and talent investment programs, internal mobility and internship opportunities.
  • Global Impact: collaborate on impactful projects for top global clients and shape the future of industries.
  • Welcoming Multicultural Environment: be a part of a dynamic, global team and thrive in an inclusive and supportive work environment with open communication and regular team-building company social events.
  • Social Sustainability Values: join our sustainable business practices focused on five pillars, including IT education, community empowerment, fair operating practices, environmental sustainability, and gender equality.

Related Categories

Related Job Pages

More Data Engineer Jobs

AppQuantum logo

Data Engineer

AppQuantum

Mobile F2P Game Publisher

Data Engineer63 days ago
Full TimeRemoteTeam 51-200Since 2017H1B No Sponsor

• Maintain and enhance our high-load DWH built on ClickHouse • Design, develop, and optimize low-level data marts for business-critical reporting • Support and improve our ETL/ELT pipelines for operational analytics • Ensure fault tolerance of our data infrastructure and cover key components with alerting • Support and optimize data storage, ensure cost-efficiency and scalability • Monitor and improve real-time data streaming via Push API

Cyprus
Job Closed
CAQH logo

Lead Data Strategist

CAQH

CAQH delivers technology-enabled solutions, operating rules and research to the healthcare industry.

Data Engineer63 days ago
Full TimeRemoteTeam 51-200Since 1998H1B No Sponsor

Position Summary The Lead Data Strategist is a senior individual contributor responsible for defining and advancing CAQH’s strategy for provider and member data, including reporting and analytics capabilities. This role operates with a high degree of autonomy, shaping how provider and member data are structured, governed, and leveraged to deliver actionable insights, improve data quality, and support enterprise and market-facing initiatives. The Lead Data Strategist leads work in complex and ambiguous problem spaces, translating business needs and market opportunities into scalable data strategies and reporting solutions. The role partners closely with Data Engineering, Data Governance, Data Science, Product, Technology, and Growth teams to ensure alignment between strategic data objectives and execution. This includes engaging external stakeholders to validate strategic direction and ensure continued market relevance. The Lead Data Strategist is a full-time, remote, exempt position and reports to the SVP, Data & Analytics. Specific Responsibilities - Define and lead CAQH’s enterprise strategy for provider and member data, with a focus on data integration, standardization, and quality improvement across internal and external sources. - Drive initiatives to unify fragmented data assets across the organization, establishing a cohesive and scalable approach to provider and member data management. - Identify, evaluate, and recommend third-party data vendors and partnerships to enhance CAQH’s data assets, coverage, and accuracy. - Develop frameworks for assessing external data sources, including data quality, interoperability, cost-benefit, and strategic fit. - Partner with Data Engineering, Data Governance, and Data Science teams to ensure alignment between data strategy, architecture, and analytical capabilities. - Collaborate with Product, Technology, and Growth teams to ensure data assets effectively support business use cases, client needs, and market expansion. - Establish and promote enterprise standards for data quality, data stewardship, and data lifecycle management across provider and member domains. - Define and implement approaches for data harmonization, normalization, and linkage across disparate systems and datasets. - Lead efforts to improve completeness, accuracy, and timeliness of provider and member data through both internal process improvements and external data enrichment. - Develop and maintain a strategic roadmap for enterprise data assets, with a focus on long-term scalability and reuse. - Conduct impact and value assessments for data initiatives, including operational efficiency, data quality improvements, and downstream business outcomes. - Serve as a subject matter expert on provider and member data strategy, advising senior leadership on data investments, risks, and opportunities. - Engage with industry stakeholders, including providers, health plans, and data vendors, to stay aligned with evolving standards and data ecosystem trends. Skills - Deep expertise in provider and member data, including data structures, data exchange, and healthcare administrative workflows (e.g., credentialing, enrollment, and roster management). - Strong understanding of data integration, data modeling, and data architecture concepts, particularly in complex, multi-source environments. - Experience evaluating and integrating third-party data sources, including vendor assessment and data due diligence. - Expertise in data quality management, including profiling, monitoring, and remediation strategies. - Strong knowledge of data governance principles, including stewardship models, standards, and controls. - Ability to connect data strategy to business outcomes, including growth, operational efficiency, and client value. - Demonstrated ability to influence cross-functional stakeholders without direct authority. - Strong analytical and structured problem-solving skills, particularly in ambiguous and complex environments. - Excellent communication skills, with the ability to translate data concepts into clear recommendations for executive and non-technical audiences. - Experience working across data, technology, and business teams to drive alignment and execution. Experience - 8 to 12 years of experience in data strategy, data management, data architecture, or related roles within healthcare or healthcare technology. - Direct experience working with provider and/or member data, including data integration, data quality, or data governance initiatives strongly preferred. - Experience evaluating or managing third-party data vendors and external data partnerships preferred. - Proven experience working across siloed data environments and leading efforts to unify and standardize data assets. - Advanced degree in data analytics, information systems, healthcare administration, public health, or related fields preferred. Who We Are CAQH is the trusted data connector at the core of healthcare. For more than 25 years, we have powered the industry with the largest and most complete healthcare data foundation in the U.S., including more than 4.8 million provider data records sourced directly from providers and member data representing 75% of covered lives supplied by health plans. By improving how essential information flows across the system, CAQH helps healthcare operate more efficiently, accurately, and with greater confidence. What You Get At CAQH, you will do meaningful work at the intersection of healthcare, data, and technology, helping solve complex problems that make the healthcare system work better. You will collaborate with experienced professionals who care deeply about accuracy, trust, and meaningful impact in a fully remote environment. CAQH offers competitive compensation and a comprehensive benefits package for full-time employees, including medical, dental, and vision coverage, a 401(k) with company contributions and matching, paid parental leave, tuition assistance, and generous paid time off. We are committed to investing in our people and supporting professional growth over time. Equal Opportunity Employer CAQH is proud to be an equal opportunity employer and is committed to fostering a workplace where all individuals are valued, respected, and empowered. Employment decisions at CAQH are made without regard to race, color, religion, sex, national origin or ancestry, age, marital status, disability, protected veteran status, personal appearance, sexual orientation, gender identity or expression, familial status, family responsibilities, matriculation, political affiliation, genetic information, source of income, place of residence, or any other characteristic protected by law. CAQH does not tolerate unlawful discrimination or harassment of any kind. Applicants have rights under the Family and Medical Leave Act (FMLA), Equal Employment Opportunity (EEO), and the Employee Polygraph Protection Act (EPPA). If you need a reasonable accommodation to apply for a posted position, please contact the CAQH People & Culture team at Careers@caqh.org or 202-517-0436.

United States
$165K - $180K / year
Job Closed
Visa logo

Staff Software Engineer - Data Platform

Visa

Based in Foster City, California, Visa is a global payments technology organization. Visa was founded in 1958, coinciding with Bank of America’s launch of the

Data Engineer63 days ago

Company Description Founded by experienced entrepreneurs and engineers in 2016, Pismo is a technology company that provides a comprehensive processing platform for banking, card issuing and financial market infrastructure and helps customers innovate and build the next generation of banking and payment solutions. Pismo joined Visa in 2024. Leveraging Visa’s solutions, our core platform, and an expanding suite of capabilities, Pismo addresses the technological challenges that large banks, marketplaces, and fintech companies face in migrating from legacy systems to more advanced technology in the market. Pismo’s cloud-based platform empowers firms to build and launch financial products rapidly, scaling as they grow to have a broader audience while keeping high security and availability standards. Pismo’s 500+ employees are located in more than 10 countries around the world. Job Description We're looking for a Staff Software Engineer to be a core technical contributor on our real-time event processing platform. You'll design and build event-driven services, drive engineering quality, and mentor junior engineers — all while working with limited supervision on complex problems. This is the first truly senior individual contributor level, and we expect you to bring strong technical judgment, ownership, and a bias for quality. What you will do: - Design and build event processing services and pipelines on AWS serverless (Lambda, Kinesis, EventBridge, Step Functions). - Write production-quality Golang and/or Python code with comprehensive testing and observability. - Own services end-to-end: development, deployment, monitoring, incident response, and SLA adherence. - Make sound architectural decisions for somewhat complex problems, exercising discretion and judgment. - Drive code quality through thorough code reviews, testing standards, and documentation. - Investigate and resolve production incidents independently; contribute to postmortem analysis. - Design and implement CI/CD improvements for better developer experience and deployment safety. - Mentor Analyst-level engineers through pairing, design guidance, and constructive review feedback. - Propose and implement improvements to platform resilience, performance, or cost. - Participate in on-call rotations independently with full accountability. This is a remote position. A remote position does not require job duties be performed within proximity of a Visa office location. Remote positions may be required to be present at a Visa office with scheduled notice. #LI-Remote Qualifications Basic Qualifications: - 5+ years of relevant work experience with a Bachelor’s Degree or at least 2 years of work experience with an Advanced degree (e.g. Masters, MBA, JD, MD) or 0 years of work experience with a PhD, OR 8+ years of relevant work experience. Preferred Qualifications: - Strong software engineering background building distributed systems at scale - Proficiency in Golang and/or Python - Experience with modern streaming processing (Kinesis, Flink, Spark) - Cloud computing skills (preferably AWS) and hybrid designs - Data platform tooling: Terraform (IaC), Docker and Kubernetes (production) and Airflow (or similar orchestration) - Observability and reliability mindset: Grafana, CloudWatch (logs, metrics, traces), on-call participation, SLA awareness - Strong SQL skills - Version control and CI/CD: Git/GitHub, pipelines (e.g. Codefresh) or similar - Soft skills: clear communication, adaptability, self-management, critical thinking, collaboration - English: B2 or above Additional Information Visa is an EEO Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law. - Job Family Group: Engineering and Technology

Brazil
Visa logo

Staff Software Engineer - Data Platform

Visa

Based in Foster City, California, Visa is a global payments technology organization. Visa was founded in 1958, coinciding with Bank of America’s launch of the

Data Engineer63 days ago

Company Description Founded by experienced entrepreneurs and engineers in 2016, Pismo is a technology company that provides a comprehensive processing platform for banking, card issuing and financial market infrastructure and helps customers innovate and build the next generation of banking and payment solutions. Pismo joined Visa in 2024. Leveraging Visa’s solutions, our core platform, and an expanding suite of capabilities, Pismo addresses the technological challenges that large banks, marketplaces, and fintech companies face in migrating from legacy systems to more advanced technology in the market. Pismo’s cloud-based platform empowers firms to build and launch financial products rapidly, scaling as they grow to have a broader audience while keeping high security and availability standards. Pismo’s 500+ employees are located in more than 10 countries around the world. Job Description We're looking for a Staff Software Engineer to be a core technical contributor on our real-time event processing platform. You'll design and build event-driven services, drive engineering quality, and mentor junior engineers — all while working with limited supervision on complex problems. This is the first truly senior individual contributor level, and we expect you to bring strong technical judgment, ownership, and a bias for quality. What you will do: - Design and build event processing services and pipelines on AWS serverless (Lambda, Kinesis, EventBridge, Step Functions). - Write production-quality Golang and/or Python code with comprehensive testing and observability. - Own services end-to-end: development, deployment, monitoring, incident response, and SLA adherence. - Make sound architectural decisions for somewhat complex problems, exercising discretion and judgment. - Drive code quality through thorough code reviews, testing standards, and documentation. - Investigate and resolve production incidents independently; contribute to postmortem analysis. - Design and implement CI/CD improvements for better developer experience and deployment safety. - Mentor Analyst-level engineers through pairing, design guidance, and constructive review feedback. - Propose and implement improvements to platform resilience, performance, or cost. - Participate in on-call rotations independently with full accountability. This is a remote position. A remote position does not require job duties be performed within proximity of a Visa office location. Remote positions may be required to be present at a Visa office with scheduled notice. #LI-Remote Qualifications Basic Qualifications: - 5+ years of relevant work experience with a Bachelor’s Degree or at least 2 years of work experience with an Advanced degree (e.g. Masters, MBA, JD, MD) or 0 years of work experience with a PhD, OR 8+ years of relevant work experience. Preferred Qualifications: - Strong software engineering background building distributed systems at scale - Proficiency in Golang and/or Python - Experience with modern streaming processing (Kinesis, Flink, Spark) - Cloud computing skills (preferably AWS) and hybrid designs - Data platform tooling: Terraform (IaC), Docker and Kubernetes (production) and Airflow (or similar orchestration) - Observability and reliability mindset: Grafana, CloudWatch (logs, metrics, traces), on-call participation, SLA awareness - Strong SQL skills - Version control and CI/CD: Git/GitHub, pipelines (e.g. Codefresh) or similar - Soft skills: clear communication, adaptability, self-management, critical thinking, collaboration - English: B2 or above Additional Information Visa is an EEO Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law. - Job Family Group: Engineering and Technology

Brazil