3Pillar Global logo
3Pillar Global

Building digital businesses, together.

Lead Data Engineer, AI

Data EngineerData EngineerFull TimeRemoteSeniorTeam 1,001-5,000H1B SponsorCompany SiteLinkedIn

Location

India

Posted

1 day ago

Salary

0

Seniority

Senior

Job Description

Lead Data Engineer, AI

3Pillar Global

• Build, test, and maintain production pipelines (batch & real-time) on Snowflake, PySpark, Delta Lake, and Kafka. • Implement data quality checks, schema validation, and alerting at every pipeline stage. • Migrate legacy ETL/DWH to cloud-native AWS/Azure architectures with measurable latency and cost improvements. • Maintain CI/CD pipelines: automated testing, deployment, rollback, and IaC (Terraform, GitHub Actions). • Build end-to-end retrieval infrastructure: document ingestion, embedding pipelines, vector store management (Pinecone, FAISS, ChromaDB, OpenSearch), and hybrid retrieval layers. • Implement chunking, metadata filtering, and re ranking — tuning for precision, recall, and latency. • Maintain data freshness and index consistency; instrument with context relevance and faithfulness metrics. • Implement and maintain business entity mappings, ontologies, and knowledge graphs (Neo4j) per Architect design. • Build and version the feature store and semantic data contracts serving both ML models and LLM applications. • Manage metadata, data lineage, and audit trail instrumentation across the platform. • Build ML data infrastructure: training curation, feature engineering, MLflow experiment tracking, dataset versioning. • Support LLM fine-tuning workflows — corpus curation, quality filtering, dataset formatting. • Implement automated evaluation pipelines: factual accuracy, hallucination detection, regression tracking. • Maintain production monitoring dashboards for pipeline health, model metrics, and alerting. • Build and maintain data APIs, tool schemas, and memory/state stores that autonomous agents depend on. • Implement agent observability: capture inputs, retrieved context, tool calls, reasoning traces, and outputs. • Maintain text-to-SQL layers, semantic query interfaces, and context APIs for conversational AI consumers. • Implement RBAC, attribute-based access, PII detection/masking, data classification, and audit logging. • Enforce data contracts and schema governance with automated breaking-change detection and versioned migrations. • Build data quality monitoring (completeness, freshness, consistency) with automated alerting and root-cause tooling. • Support compliance readiness: audit trails, data provenance, and regulatory documentation.

Job Requirements

  • 7+ years data engineering using Cloud services
  • 2+ years production AI/ML or LLM-era data infrastructure. Proven experience building production pipelines at scale — batch and streaming, Snowflake,AWS/Azure.
  • Deep expertise: Python, PySpark, Snowflake, Delta Lake, Kafka, Spark Structured Streaming.
  • Hands-on with vector stores, embedding pipelines, and retrieval infrastructure in production RAG environments.
  • Working knowledge of MLOps: MLflow, CI/CD for AI, automated evaluation, and production monitoring.
  • Strong grounding in data governance, quality frameworks, and compliance-**aligned engineering.

Benefits

  • Health insurance
  • 401(k) matching
  • Flexible work hours
  • Paid time off
  • Remote work options

Related Categories

Related Job Pages

More Data Engineer Jobs

Keyrus logo

Senior Data Engineer, Azure

Keyrus

#MakeDataMatter #HumanizingTheFuture

Data Engineer1 day ago
ContractRemoteTeam 1,001-5,000Since 1996H1B Sponsor

• Design, develop, and maintain scalable data pipelines and integration solutions within the Azure ecosystem • Build and support enterprise data platforms that consolidate information from multiple source systems • Develop ETL/ELT processes to ingest, transform, validate, and distribute business-critical data • Contribute to data modelling, data quality, and governance initiatives • Support Master Data Management (MDM) processes and ensure consistency across systems • Work with structured and semi-structured data from ERP, business applications, APIs, databases, and external sources • Collaborate with business and technical stakeholders to translate requirements into scalable data solutions • Participate in deployment, testing, monitoring, and continuous improvement activities • Contribute to CI/CD pipelines, automation, and DevOps best practices • Actively participate in Agile ceremonies and team collaboration

United States
€250 - €350 / day
Jimdo logo

Data Engineer

Jimdo

Everything for your business: all the tools you need for your business to succeed

Data Engineer1 day ago
Full TimeRemoteTeam 201-500H1B No Sponsor

• Build and maintain scalable data pipelines, transformations, and reporting models using dbt, SQL, Python, and Snowflake. • Develop and optimize data ingestion workflows, API integrations, and CDC pipelines using established engineering frameworks and best practices. • Implement data quality checks, testing, monitoring, and observability mechanisms to ensure reliable and trustworthy data products. • Design and maintain analytics-ready data models, including reporting marts and dimensional models that support business insights. • Leverage AI-powered development tools such as GitHub Copilot and Claude to improve productivity, code quality, and documentation. • Collaborate with analysts, BI teams, and stakeholders to define metrics, clarify business logic, and deliver impactful data solutions. • Contribute to data governance, compliance, and platform reliability by following data contracts, security standards, and engineering best practices.

Germany
Full TimeRemoteTeam 1,001-5,000Since 1973H1B No Sponsor

• Build, lead, and develop an established, high-performing, globally distributed platform engineering team, fostering a culture of technical excellence, accountability, and continuous improvement. • Define and execute the enterprise cloud data platform strategy, aligning platform investments with business priorities, data/AI objectives, and long-term technology roadmaps. • Lead the engineering and operations of the full data platform ecosystem — including cloud data warehouses/Lakehouses, data integration, transformation, orchestration, observability, and data quality services. • Establish and enforce technology standards, reference architectures, and engineering guardrails that ensure platform consistency, scalability, security, and interoperability across the data ecosystem. • Drive platform adoption and enablement across analytics, data science, and business teams by delivering self-service capabilities, reusable data products, and well-documented platform services. • Partner with business, product, and technology leaders to translate data needs into prioritized platform capabilities and investment cases with clear ROI. • Embed robust data governance, security, and regulatory compliance controls into platform design and operations, ensuring least-privileged access, data classification, lineage, and auditability. • Own platform reliability, performance, and cost optimization (FinOps) — establishing SLOs/SLAs, monitoring frameworks, incident response protocols, and unit-cost accountability for cloud data services. • Manage strategic vendor relationships, technology evaluations, and licensing negotiations to optimize the platform ecosystem and maintain a forward-looking technology roadmap. • Define and track measurable outcomes (KPIs/OKRs) for platform value realization, adoption, delivery predictability, and operational excellence — reporting progress to executive leadership. • Drives reliability, cost efficiency, and continuous improvement through metric-driven operations. • Attracts, develops, and retains top engineering talent with clear career paths and a strong team culture.

Missouri
$150.8K - $224.6K / year
BlackRock logo

Senior Software Engineer, Data Integrations

BlackRock

Based in New York, New York, BlackRock is a publicly traded, international investment company serving millions of individuals worldwide. The company's clients a

Data Engineer1 day ago

Title: Senior Software Engineer, Data Integrations (Java/Python) Location: San Francisco, Sausalito, CA, United States Job Description: time type Full time job requisition id R260814 About this role About BlackRock SMA Solutions: At BlackRock SMA Solutions, our strategies are designed to put our clients’ and their clients’ interests at the center of our investment advice; to minimize costs and taxes; and to incorporate each client’s unique values-aligned preferences into their investment portfolio. Offering a full suite of tools for our clients, from direct-indexing to active equity and fixed income strategies—with evolution a constant part of our game. As a business, we are fast-growing and our systems need to grow along with it. We are looking for engineers who like to innovate and seek complex problems. We recognize that strength comes from teams with a variety of perspectives, and will embrace your unique skills, curiosity, drive, and passion while giving you the opportunity to grow technically and as an individual. Engineers looking to work in the areas of orchestration, data modeling, data pipelines, APIs, storage, distribution, distributed computation, consumption and infrastructure are ideal candidates. About this Role: We are expanding the team of engineers that owns and operates the data and integration platform underpinning the SMA business. As a member of this team, you will design, build, and support data‑centric and integration‑focused services that power portfolio construction, reporting, and downstream client and business workflows. In this role, you will work across APIs, data pipelines, microservices, and relational data stores, partnering closely with product, analytics, and business stakeholders to ensure data and services are reliable, secure, and delivered at scale. The role combines strong data engineering fundamentals with modern integration and platform engineering practices, with opportunities to contribute across the full lifecycle from design through production support. Key Responsibilities - Design, develop, and own end‑to‑end features across data pipelines, APIs, and microservices, from ingestion and transformation through consumption. - Build and enhance integration services that exchange data with internal platforms and external systems. - Develop and evolve components of a microservices ecosystem supporting equity portfolio construction and direct indexing workflows. - Design relational database schemas, write efficient SQL, and ensure data quality, completeness, and timeliness. - Support building, releasing, and operating services in a cloud‑native environment, including deployment to Kubernetes‑based platforms. - Troubleshoot issues in live production environments and provide real‑time support for critical incidents. - Continuously improve the platform through modernization of existing components, adoption of contemporary technologies, and use of enterprise‑grade third‑party services. - Write code that is clear, maintainable, well‑tested, and easy for other engineers to understand and extend. - Collaborate effectively with engineers, product partners, and business stakeholders in a fast‑paced, interdisciplinary environment. Requirements - BA/BS in Computer Science or equivalent practical experience. - 6+ years of post-university professional experience in a data engineering, integration engineering, or backend/full‑stack engineering role, preferably in a financial services or trading environment. - Strong understanding of programming fundamentals, including algorithms, data structures, design patterns, and paradigms. - Proficiency in one or more modern object‑oriented languages (Java/Python). - Solid experience with SQL and relational databases - Experience designing and consuming APIs using modern approaches (RESTful services; familiarity with GraphQL or gRPC is a plus). - Ability to troubleshoot and resolve issues in production systems and provide real‑time support for critical issues. - Strong written and verbal communication skills, with the ability to work effectively across engineering and business teams. - Ability to work effectively in a fast‑paced, interdisciplinary environment. Desired Additional Qualifications - Experience working in a cloud environment - Familiarity with messaging and event‑driven technologies (AWS SQS, etc.). - Familiarity with containerization and orchestration technologies - Understanding of secure distributed systems concepts, including OAuth2 / OpenID Connect, secrets management, caching, and observability (logging, metrics, tracing). - Exposure to front‑end technologies (e.g., React or Angular) or statistical / financial analysis is a plus. For San Francisco, CA and Sausalito, CA Only the salary range for this position is USD$162,000.00 - USD$215,000.00 . Additionally, employees are eligible for an annual discretionary bonus, and benefits including healthcare, leave benefits, and retirement benefits. BlackRock operates a pay-for-performance compensation philosophy and your total compensation may vary based on role, location, and firm, department and individual performance. Our benefits To help you stay energized, engaged and inspired, we offer a wide range of benefits including a strong retirement plan, tuition reimbursement, comprehensive healthcare, support for working parents and Flexible Time Off (FTO) so you can relax, recharge and be there for the people you care about. Our hybrid work model BlackRock’s hybrid work model is designed to enable a culture of collaboration and apprenticeship that enriches the experience of our employees, while supporting flexibility for all. Employees are currently required to work at least 4 days in the office per week, with the flexibility to work from home 1 day a week. Some business groups may require more time in the office due to their roles and responsibilities. We remain focused on increasing the impactful moments that arise when we work together in person – aligned with our commitment to performance and innovation. As a new joiner, you can count on this hybrid model to accelerate your learning and onboarding experience here at BlackRock. Guidance on AI use for candidates At BlackRock, AI has long been part of how we work – enhancing decision-making, improving operations, and helping us deliver better outcomes for clients. We encourage candidates to use AI thoughtfully to learn, prepare, and work more effectively; but during our interview process, we want to focus on getting to know you through your own experiences, thinking, and judgment. To support you, we’ve provided guidance on when and how to use AI during our hiring process so you can approach each step with confidence and showcase your best self. About BlackRock At BlackRock, we are all connected by one mission: to help more and more people experience financial well-being. Our clients, and the people they serve, are saving for retirement, paying for their children’s educations, buying homes and starting businesses. Their investments also help to strengthen the global economy: support businesses small and large; finance infrastructure projects that connect and power cities; and facilitate innovations that drive progress. This mission would not be possible without our smartest investment – the one we make in our employees. It’s why we’re dedicated to creating an environment where our colleagues feel welcomed, valued and supported with networks, benefits and development opportunities to help them thrive. BlackRock is proud to be an equal opportunity workplace. We are committed to equal employment opportunity to all applicants and existing employees, and we evaluate qualified applicants without regard to race, creed, color, national origin, sex (including pregnancy and gender identity/expression), sexual orientation, age, ancestry, physical or mental disability, marital status, political affiliation, religion, citizenship status, genetic information, veteran status, or any other basis protected under applicable federal, state, or local law. We recruit, hire, train, promote, pay, and administer all personnel actions without regard to race, color, religion, sex (including pregnancy, childbirth, and medical conditions related to pregnancy, childbirth, or breastfeeding), sex stereotyping (including assumptions about a person’s appearance or behavior, gender roles, gender expression, or gender identity), gender, gender identity, gender expression, national origin, age, mental or physical disability, ancestry, medical condition, marital status, military or veteran status, citizenship status, sexual orientation, genetic information, or any other status protected by applicable law. We interpret these protected statuses broadly to include both the actual status and also any perceptions and assumptions made regarding these statuses.BlackRock will consider for employment qualified applicants with arrest or conviction records in a manner consistent with the requirements of the law, including any applicable fair chance law.

California
$162K - $215K / year