We’re a tech consultancy, expert in software engineering and cloud transformation.
Senior Data Engineer, Microsoft Fabric
Location
Portugal
Posted
3 days ago
Salary
0
Seniority
Senior
Job Description
Senior Data Engineer, Microsoft Fabric
Vigil
• Design, build and maintain enterprise data solutions using Microsoft Fabric • Develop scalable data ingestion pipelines from APIs, databases and file-based sources • Build and maintain Lakehouse architectures using the Medallion (Bronze, Silver and Gold) approach • Develop robust data models and semantic layers to support business reporting • Create and maintain Power BI semantic models and dashboards • Implement data quality, validation and reconciliation frameworks • Design secure, governed data solutions including Row-Level Security and access controls • Build and maintain PySpark notebooks for data transformation • Develop SQL and Python-based ETL/ELT processes • Configure and maintain Fabric workspaces, OneLake environments and deployment pipelines • Work closely with solution architects to implement scalable platform designs • Produce technical documentation and engineering standards • Collaborate with UK-based stakeholders and delivery teams • Support future client projects across the TXP Data & AI Practice
Job Requirements
- Strong commercial experience with Microsoft Fabric
- Experience with Fabric Data Factory, Dataflows Gen2, Lakehouse and OneLake
- Strong SQL skills
- Strong Python and/or PySpark experience
- Experience designing enterprise data models
- Strong understanding of Medallion Architecture
- Experience building scalable ETL/ELT pipelines
- REST API integration experience
- Experience implementing data quality and reconciliation processes
- Solid Power BI experience, including semantic modelling and DAX
- Understanding of Azure fundamentals, including Entra ID and Key Vault
- Experience working independently on complex technical projects
- Excellent communication skills in English
Benefits
- Be part of our collegial environment where responsibility and authority are shared equally amongst colleagues and help create our company culture
- A culture in which we don’t criticise failure but ensure we learn from our mistakes
- An Agile environment where your ideas are welcome
- The possibility to grow and experience different projects
- Ongoing Training & Mentoring
- The possibility to travel
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Engineer – Data Conversions, Junior-Mid
EnrouteWe deliver IT services and solutions provided by a team of passionate problem solving individuals highly skilled.
• Play a critical role in migrating insurance policy data from administration systems to a modern policy administration platform. • Perform analysis-first role with a data engineering component. • Deep-dive into data, trace and reconcile information across multiple data layers (databases and files). • Build mappings and transformations that move legacy data into the target format. • Resolve data quality issues to deliver validated loads. • Utilize strong, hands-on command of SQL for analysis, tracing, and transforming data. • Use AI tooling to accelerate work processes. • Receive mentorship from an experienced conversion manager. • Work through entire data migration lifecycle, from initial discovery through production cutover. • Support iterative delivery cycles with effective communication.
Arquitecto de Datos
Sofka TechnologiesTo transform people’s lives being the most trusted technology partner
• Definir la estrategia analítica y el modelo funcional de la solución. • Diseñar y construir el pipeline de ingesta y transformación sobre AWS. • Configurar la seguridad de la infraestructura (IAM). • Diseñar el modelo dimensional (star schema). • Actuar como autoridad técnica frente al cliente.
• Maintain a clear view of each team member's delivery commitments, growth plans, and internal contributions. • Develop and grow team members through regular coaching, clear expectations, and constructive feedback. Build a culture of trust, inclusion, and collaboration. • Monitor team health, morale, and workload balance, acting early to address engagement or performance concerns in partnership with practice leadership. • Maintain regular one-to-one connections with each team member, provide clear feedback, and help them remove blockers related to client work, pursuits, or internal initiatives. • Oversee staffing and resource alignment across teams, balancing utilization, development goals, client needs, and sustainable workloads. • Communicate practice priorities and organizational objectives clearly, ensuring teams understand how their work contributes to broader organizational goals. • Define interview standards and mentorship strategies, training others to apply them consistently and improving hiring outcomes and talent growth at scale. • Create and maintain development plans for team members, including stretch assignments, shadowing, certifications, and opportunities for external visibility. • Lead or support performance reviews, promotion recommendations, and compensation input for your teams, using consistent standards that reflect both impact and behavior. • Represent your teams in leadership forums, communicate expectations and decisions clearly back to the group, and bring forward patterns, risks, and successes that should shape practice strategy. • Lead architecture and technical strategy for large Azure programs and portfolios so solutions meet client goals for value, security, scalability, reliability, performance, and cost. • Define modernization blueprints and reference architectures across applications, platforms, and data, including system and subsystem designs, patterns, and standards that simplify delivery and operations over time. • Translate business and product strategy into cloud architecture roadmaps and execution plans that drive measurable outcomes and adapt based on production feedback and telemetry. • Guide major architecture decisions across programs, promote shared design standards and reusable models, and resolve high-impact technical trade-offs. • Work across teams to connect solutions into cohesive end-to-end architectures with clear ownership boundaries and integration contracts. • Provide architectural oversight during delivery through design reviews, forums, and governance, keeping cross-cutting concerns (security, performance, resilience, availability, compliance, observability, and cost) aligned across teams. • Promote modular design, clear boundaries, testability, CI/CD, and observability so non-functional requirements are treated as first-class and systems remain reliable and maintainable. • Use AI tools to accelerate discovery, design, code, tests, documentation, and analysis, and scale successful practices so teams adopt them consistently. • Define and own enterprise data platform strategy centered on Databricks or equivalent lakehouse architectures. • Establish reference architectures spanning ingestion, lakehouse, analytics, and AI/ML. • Set standards and best practices for: Databricks lakehouse design and workload isolation. Delta Lake optimization and data lifecycle management. Streaming and batch processing patterns. Unity Catalog–aligned governance and security models. Guide multi-workspace and multi-region Databricks deployments. • Ensure tight integration between Databricks, Azure-native services, and downstream analytics tools. • Oversee architectural alignment with complementary platforms (e.g., Snowflake, Fabric, Power BI).
• Design, develop, and maintain scalable data pipelines to support ingestion, transformation, and delivery into centralized feature stores, model-training workflows, and real-time inference services. • Build and optimize workflows for extracting, storing, and retrieving semantic representations of unstructured data to enable advanced search and retrieval patterns. • Architect and implement lightweight analytics and dashboarding solutions that deliver natural language query experience and AI-backed insights. • Define and execute processes for managing prompt engineering techniques, orchestration flows, and model fine-tuning routines to power conversational interfaces. • Oversee vector data stores and develop efficient indexing methodologies to support retrieval-augmented generation (RAG) workflows. • Partner with data stakeholders to gather requirements for language-model initiatives and translate into scalable solutions. • Create and maintain comprehensive documentation for all data processes, workflows and model deployment routines. • Should be willing to stay informed and learn emerging methodologies in data engineering, MLOps and LLM operations.




