We are a Y-Combinator-backed startup building your AI-powered Recruiter Agent
Senior Data Engineer / Microsoft Fabric Engineer
Location
India
Posted
1 day ago
Salary
₹1,100K - ₹5,000K / year
Seniority
Senior
Job Description
Senior Data Engineer / Microsoft Fabric Engineer
Weekday (YC W21)
Role Description We are looking for an experienced Data Engineer to design, develop, and optimize scalable data platforms and modern cloud-based data solutions. This role is ideal for professionals with strong expertise in Microsoft Azure Fabric, Databricks, Azure data services, and enterprise data engineering, along with a passion for building reliable data pipelines and leveraging AI-assisted development to improve engineering productivity. As a Data Engineer, you will take ownership of the complete data engineering lifecycle—from solution design and implementation to production support and continuous optimization. You will collaborate closely with business stakeholders, architects, analysts, and development teams to transform complex data requirements into scalable, high-performance data solutions that enable analytics and informed business decision-making. Key Responsibilities - Design, develop, and maintain scalable data pipelines using Microsoft Azure Fabric, Databricks, and Azure data services. - Build and optimize robust ETL/ELT processes for structured, semi-structured, and unstructured data across enterprise environments. - Collaborate with business stakeholders to understand data requirements and translate them into scalable technical solutions. - Develop and maintain reliable data integration workflows that ensure data accuracy, consistency, and availability. - Optimize data processing pipelines for performance, scalability, cost efficiency, and operational reliability. - Leverage AI-assisted development tools and modern engineering practices to accelerate solution delivery and improve code quality. - Monitor, troubleshoot, and resolve production data pipeline issues while proactively identifying opportunities for automation and optimization. - Work closely with architects, analysts, developers, and cross-functional teams throughout the project lifecycle to deliver high-quality data solutions. - Implement best practices for data engineering, governance, documentation, testing, and operational support. - Take ownership of project deliverables by ensuring quality, meeting timelines, communicating risks proactively, and continuously improving engineering processes. Qualifications - 5+ years of hands-on experience as a Data Engineer with expertise in enterprise-scale data engineering solutions. - Strong experience with Microsoft Azure Fabric, Databricks, and Azure data services in production environments. - Solid understanding of SQL, data modeling, data warehousing concepts, and cloud-based data platforms. - Proven experience designing and implementing scalable ETL/ELT pipelines and enterprise data integration solutions. - Proficiency in Python, Apache Spark, or similar technologies for large-scale data processing. - Experience optimizing cloud-based data workflows for performance, scalability, reliability, and cost efficiency. - Strong analytical and problem-solving skills with the ability to troubleshoot complex production issues effectively. - Comfortable using AI-assisted development tools and automation techniques to improve engineering productivity and solution quality. - Excellent communication and stakeholder management skills, with the ability to collaborate effectively across global, cross-functional teams. - A proactive, ownership-driven mindset with the flexibility to adapt to evolving priorities while consistently delivering high-quality, business-critical data solutions.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
• Design, build, and maintain data pipelines that serve models, data, and AI workflows to internal and client-facing applications. • Work across database types — relational, vector, and graph — to model and store data appropriately for each access pattern, in partnership with data scientists. • Build and maintain dbt models — writing transformation logic, tests, and documentation that ensure data quality and traceability. • Operate what you build : instrument pipelines with logging, metrics, and tracing, and help diagnose and resolve production data issues. • Write clean, tested, production-quality code and contribute to CI/CD pipelines and infrastructure-as-code. • Participate in code reviews, design discussions, and retrospectives.
• architect and build production data pipelines and data platforms that serve models, data, and AI workflows to internal and client-facing applications, accountable for them under live traffic • own non-functional quality – latency and throughput budgets, scalability, reliability, observability, and cost – for the systems in your domain • lead the design and operation of multi-model data stores – relational databases (PostgreSQL, MySQL), vector databases (Pinecone, Weaviate, pgvector), and graph databases (Neo4j, Neptune) – ensuring the right tool for each access pattern • set technical direction: write design docs, make build-vs-buy decisions, and defend your approach with evidence • work across the stack when needed – services, data access, infrastructure-as-code, CI/CD – and debug it when things drift in production • mentor and set the quality standards for mid-level and junior data engineers
Data Engineering Lead
RadixA Radix está sempre no topo das Melhores Empresas para se trabalhar porque: Temos profissionais comprometidos, dedicados, curiosos e inovadores. O espírito de equipe é a nossa maior força. Trabalhamos de forma cooperativa e sabemos que estamos juntos, remando na mesma direção. Temos um ambiente diverso, que valoriza equidade e inclusão. Nossa jornada de trabalho é flexível e em quase todos os projetos é possível trabalhar de qualquer lugar do Brasil. Valorizamos o bem-estar e o cuidado com as nossas pessoas, com programas de apoio à saúde mental, psiquiatra e médico consultor disponíveis.
Role Description A primeira coisa que você precisa saber é que aqui você não vai cair na rotina. A Radix desenvolve soluções para empresas de diferentes setores e indústrias. Cada projeto tem suas tecnologias, soluções e prazos e você terá oportunidade de atuar e experimentar diferentes desafios. Como Profissional Líder em Engenharia de Dados você vai: - Apoiar a coordenação das atividades de captura da realidade (Reality Capture) nas plantas, incluindo laser scanning, fotogrametria e demais atividades relacionadas. - Participar de reuniões recorrentes dos projetos com stakeholders das plantas para acompanhar o progresso, alinhar prioridades e compartilhar lições aprendidas. - Apoiar atividades de implementação, incluindo identificação de ativos (asset tagging), mapeamento de dados e fluxos de integração. - Realizar revisões periódicas da qualidade dos ambientes de Digital Twin para garantir a organização dos dados, a integridade das integrações e o desempenho geral da plataforma. - Ministrar treinamentos para usuários finais por meio de sessões programadas, plantões de atendimento (office hours) e suporte sob demanda. - Colaborar com gerentes de projeto e patrocinadores locais (site champions) para identificar, desenvolver e expandir casos de uso de Digital Twin. - Atuar como principal ponto de contato entre as equipes das plantas e as equipes técnicas especializadas responsáveis por captura da realidade, modelos 3D, integração de dados e administração da plataforma. - Facilitar a comunicação entre as equipes do projeto e os stakeholders das plantas, garantindo a resolução ágil de problemas, a coleta de feedbacks e a melhoria contínua. - Auxiliar no acompanhamento do progresso dos projetos, das ações pendentes e das atividades de implantação em múltiplas plantas. Qualifications - Inglês avançado / fluente para conversação. - Experiência em liderança de projetos, com capacidade para gerenciar múltiplos stakeholders e prioridades simultaneamente. - Excelentes habilidades de comunicação e relacionamento interpessoal, com capacidade para interagir de forma eficaz tanto com engenheiros quanto com técnicos. - Inglês fluente (conversação, leitura e escrita). - Experiência no setor de Óleo & Gás, preferencialmente em plantas operacionais ou em ambientes industriais semelhantes. - Bons conhecimentos em informática, facilidade para aprender novas ferramentas de software e perfil proativo e autodirigido. - Experiência com gestão da mudança (Change Management) será considerada um diferencial, especialmente em projetos de transformação digital. - Vivência ou conhecimento em softwares 3D, laser scanning, modelagem 3D ou ferramentas de digitalização e habilitação digital será altamente desejável. Benefits - Assistência Médica Nacional (para o titular e dependentes, com quarto privativo) - Assistência odontológica nacional (para o titular e dependentes) - Vale refeição / alimentação flexível - Auxílio home office - Day off (no mês do aniversário) - Wellhub (antigo Gympass) - Licença Maternidade (6 meses) e Paternidade (20 dias) estendidas - Auxílio creche para filhos de até 3 anos (por filho) - Apoio em saúde mental com a Wellz - Clube de Vantagens com descontos em diversos parceiros - Convênio com instituições de ensino e cursos de idioma - Desenvolvimento Profissional (Universidade Corporativa) - Parceria com empresa de coworkings no Brasil - Programa de Qualidade de Vida e Bem-Estar - Médico consultor para acompanhamento de radixers - Planos de incentivos Company Description A Radix está sempre no topo das Melhores Empresas para se trabalhar porque: - Temos profissionais comprometidos, dedicados, curiosos e inovadores. - O espírito de equipe é a nossa maior força. Trabalhamos de forma cooperativa e sabemos que estamos juntos, remando na mesma direção. - Temos um ambiente diverso, que valoriza equidade e inclusão. - Nossa jornada de trabalho é flexível e em quase todos os projetos é possível trabalhar de qualquer lugar do Brasil. - Valorizamos o bem-estar e o cuidado com as nossas pessoas, com programas de apoio à saúde mental, psiquiatra e médico consultor disponíveis.
• Design, develop, and operate scalable and maintainable data pipelines in the Azure Databricks environment • Develop all technical artefacts as code, implemented in professional IDEs, with full version control and CI/CD automation • Enable data-driven decision-making in Supply Chain Management (SCM) by ensuring high data availability, quality, and reliability • Implement data products and analytical assets using software engineering principles in close alignment with business domains and functional IT • Apply rigorous software engineering practices such as modular design, test-driven development, and artifact reuse in all implementations • Global delivery footprint; cross-functional data engineering support across SCM domains • Collaboration with business stakeholders, functional IT partners, product owners, architects, ML/AI engineers, and Power BI developers • Agile, product-team structure embedded in an enterprise-scale Azure environment • Design scalable batch and streaming pipelines in Azure Databricks using PySpark and/or Scala • Implement ingestion from structured and semi-structured sources (e.g., SAP, APIs, flat files) • Build bronze/silver/gold data layers following the defined lakehouse layering architecture & governance • Implement use-case driven dimensional models (star/snowflake schema) tailored to SCM needs • Ensure compatibility with reporting tools (e.g., Power BI) via curated data marts and semantic models • Implement enterprise-level data warehouse models (domain-driven 3NF models) for SCM data, closely aligned with data engineers for other business domains • Develop and apply master data management strategies (e.g., Slowly Changing Dimensions) • Develop automated data validation tests using frameworks • Monitor pipeline health, identify anomalies, and implement quality thresholds • Establish data quality transparency by defining and implementing meaningful data quality rules with source system and business stakeholders and implementing related reports • Develop and structure pipelines using modular, reusable code in a professional IDE • Apply test-driven development (TDD) principles with automated unit, integration, and validation tests • Integrate tests into CI/CD pipelines to enable fail-fast deployment strategies • Commit all artifacts to version control with peer review and CI/CD integration • Work closely with Product Owners to refine user stories and define acceptance criteria • Translate business requirements into data contracts and technical specifications • Participate in agile events such as sprint planning, reviews, and retrospectives • Document pipeline logic, data contracts, and technical decisions in markdown or auto-generated docs from code • Align designs with governance and metadata standards (e.g., Unity Catalog) • Track lineage and audit trails through integrated tooling • Profile and tune data transformation performance • Reduce job execution times and optimize cluster resource usage • Refactor legacy pipelines or inefficient transformations to improve scalability


