3Pillar Global logo
3Pillar Global

Building digital businesses, together.

Cloud Data Architect, AI Experience

Data EngineerData EngineerFull TimeRemoteLeadTeam 1,001-5,000H1B SponsorCompany SiteLinkedIn

Location

India

Posted

1 day ago

Salary

0

Seniority

Lead

Job Description

Cloud Data Architect, AI Experience

3Pillar Global

• Architect and own the enterprise AI data platform — the unified, governed layer that ingests, transforms, stores, and serves all data consumed by AI systems across the organisation. • Design multi-domain data models (lakehouse, data mesh, event-driven) that are structured from day one to serve AI workloads: clean lineage, versioned schemas, well-documented contracts, and low-latency serving APIs. • Own the full data stack: real-time streaming (Kafka, Spark Structured Streaming), batch processing (Databricks, PySpark, Delta Lake), cloud storage and compute (AWS, Azure), and data quality /metadata management. • Ensure this platform is the single, authoritative data source for all downstream consumers — conversational AI, dashboard assistants, autonomous agents, ML models, and reporting — eliminating data silos and conflicting truths. • Drive modernisation of legacy pipelines (on-prem ETL, batch DWH) to cloud-native, AI-ready architectures with measurable improvements in cost, latency, and delivery velocity. • Design the semantic layer that sits above raw data — business-aligned ontologies, entity relationships, domain taxonomies, and knowledge graphs — so AI systems understand context, not just tokens. • Build and maintain knowledge graphs (Neo4j or equivalent) that capture relationships between business entities, policies, KPIs, hierarchies, and domain rules — enabling structured reasoning alongside unstructured retrieval. • Define and govern a feature store and semantic data contracts that serve both classical ML models and LLM-based applications from a single, well-versioned, trusted source. • Own metadata management, data lineage, and audit trails across the semantic layer — ensuring every AI system can trace its outputs back to source data with full accountability. • Design and enforce a comprehensive data governance model that governs access for both human users and AI agents — with role-based access control (RBAC), attribute-based policies, and agent-specific permission scopes that prevent privilege escalation.

Job Requirements

  • 15+ years of hands-on data engineering and architecture experience, with 3–5+ years building production AI/ML and LLM-era data infrastructure.
  • Proven experience designing enterprise-scale AI data platforms that serve multiple AI consumers — not just one application or pipeline.
  • Deep expertise in lakehouse and data mesh architectures: Databricks, Delta Lake, PySpark, Kafka, Spark Structured Streaming, cloud-native data services (AWS, Azure).
  • Hands-on experience with vector stores, semantic models, knowledge graphs, and retrieval infrastructure in production environments.
  • Working knowledge of LLMOps: model serving pipelines, MLflow, CI/CD for AI, automated evaluation, and production monitoring.
  • Strong background in data governance, security, and compliance in regulated industries (financial services, payments, cybersecurity, healthcare).
  • Experience defining data access controls for AI agents and automated systems — not just human users.

Benefits

  • Health insurance
  • Flexible work hours
  • Professional development opportunities

Related Categories

Related Job Pages

More Data Engineer Jobs

Keyrus logo

Senior Data Engineer, Azure

Keyrus

#MakeDataMatter #HumanizingTheFuture

Data Engineer1 day ago
ContractRemoteTeam 1,001-5,000Since 1996H1B Sponsor

• Design, develop, and maintain scalable data pipelines and integration solutions within the Azure ecosystem • Build and support enterprise data platforms that consolidate information from multiple source systems • Develop ETL/ELT processes to ingest, transform, validate, and distribute business-critical data • Contribute to data modelling, data quality, and governance initiatives • Support Master Data Management (MDM) processes and ensure consistency across systems • Work with structured and semi-structured data from ERP, business applications, APIs, databases, and external sources • Collaborate with business and technical stakeholders to translate requirements into scalable data solutions • Participate in deployment, testing, monitoring, and continuous improvement activities • Contribute to CI/CD pipelines, automation, and DevOps best practices • Actively participate in Agile ceremonies and team collaboration

United States
€250 - €350 / day
Jimdo logo

Data Engineer

Jimdo

Everything for your business: all the tools you need for your business to succeed

Data Engineer1 day ago
Full TimeRemoteTeam 201-500H1B No Sponsor

• Build and maintain scalable data pipelines, transformations, and reporting models using dbt, SQL, Python, and Snowflake. • Develop and optimize data ingestion workflows, API integrations, and CDC pipelines using established engineering frameworks and best practices. • Implement data quality checks, testing, monitoring, and observability mechanisms to ensure reliable and trustworthy data products. • Design and maintain analytics-ready data models, including reporting marts and dimensional models that support business insights. • Leverage AI-powered development tools such as GitHub Copilot and Claude to improve productivity, code quality, and documentation. • Collaborate with analysts, BI teams, and stakeholders to define metrics, clarify business logic, and deliver impactful data solutions. • Contribute to data governance, compliance, and platform reliability by following data contracts, security standards, and engineering best practices.

Germany
Full TimeRemoteTeam 1,001-5,000Since 1973H1B No Sponsor

• Build, lead, and develop an established, high-performing, globally distributed platform engineering team, fostering a culture of technical excellence, accountability, and continuous improvement. • Define and execute the enterprise cloud data platform strategy, aligning platform investments with business priorities, data/AI objectives, and long-term technology roadmaps. • Lead the engineering and operations of the full data platform ecosystem — including cloud data warehouses/Lakehouses, data integration, transformation, orchestration, observability, and data quality services. • Establish and enforce technology standards, reference architectures, and engineering guardrails that ensure platform consistency, scalability, security, and interoperability across the data ecosystem. • Drive platform adoption and enablement across analytics, data science, and business teams by delivering self-service capabilities, reusable data products, and well-documented platform services. • Partner with business, product, and technology leaders to translate data needs into prioritized platform capabilities and investment cases with clear ROI. • Embed robust data governance, security, and regulatory compliance controls into platform design and operations, ensuring least-privileged access, data classification, lineage, and auditability. • Own platform reliability, performance, and cost optimization (FinOps) — establishing SLOs/SLAs, monitoring frameworks, incident response protocols, and unit-cost accountability for cloud data services. • Manage strategic vendor relationships, technology evaluations, and licensing negotiations to optimize the platform ecosystem and maintain a forward-looking technology roadmap. • Define and track measurable outcomes (KPIs/OKRs) for platform value realization, adoption, delivery predictability, and operational excellence — reporting progress to executive leadership. • Drives reliability, cost efficiency, and continuous improvement through metric-driven operations. • Attracts, develops, and retains top engineering talent with clear career paths and a strong team culture.

Missouri
$150.8K - $224.6K / year
BlackRock logo

Senior Software Engineer, Data Integrations

BlackRock

Based in New York, New York, BlackRock is a publicly traded, international investment company serving millions of individuals worldwide. The company's clients a

Data Engineer1 day ago

Title: Senior Software Engineer, Data Integrations (Java/Python) Location: San Francisco, Sausalito, CA, United States Job Description: time type Full time job requisition id R260814 About this role About BlackRock SMA Solutions: At BlackRock SMA Solutions, our strategies are designed to put our clients’ and their clients’ interests at the center of our investment advice; to minimize costs and taxes; and to incorporate each client’s unique values-aligned preferences into their investment portfolio. Offering a full suite of tools for our clients, from direct-indexing to active equity and fixed income strategies—with evolution a constant part of our game. As a business, we are fast-growing and our systems need to grow along with it. We are looking for engineers who like to innovate and seek complex problems. We recognize that strength comes from teams with a variety of perspectives, and will embrace your unique skills, curiosity, drive, and passion while giving you the opportunity to grow technically and as an individual. Engineers looking to work in the areas of orchestration, data modeling, data pipelines, APIs, storage, distribution, distributed computation, consumption and infrastructure are ideal candidates. About this Role: We are expanding the team of engineers that owns and operates the data and integration platform underpinning the SMA business. As a member of this team, you will design, build, and support data‑centric and integration‑focused services that power portfolio construction, reporting, and downstream client and business workflows. In this role, you will work across APIs, data pipelines, microservices, and relational data stores, partnering closely with product, analytics, and business stakeholders to ensure data and services are reliable, secure, and delivered at scale. The role combines strong data engineering fundamentals with modern integration and platform engineering practices, with opportunities to contribute across the full lifecycle from design through production support. Key Responsibilities - Design, develop, and own end‑to‑end features across data pipelines, APIs, and microservices, from ingestion and transformation through consumption. - Build and enhance integration services that exchange data with internal platforms and external systems. - Develop and evolve components of a microservices ecosystem supporting equity portfolio construction and direct indexing workflows. - Design relational database schemas, write efficient SQL, and ensure data quality, completeness, and timeliness. - Support building, releasing, and operating services in a cloud‑native environment, including deployment to Kubernetes‑based platforms. - Troubleshoot issues in live production environments and provide real‑time support for critical incidents. - Continuously improve the platform through modernization of existing components, adoption of contemporary technologies, and use of enterprise‑grade third‑party services. - Write code that is clear, maintainable, well‑tested, and easy for other engineers to understand and extend. - Collaborate effectively with engineers, product partners, and business stakeholders in a fast‑paced, interdisciplinary environment. Requirements - BA/BS in Computer Science or equivalent practical experience. - 6+ years of post-university professional experience in a data engineering, integration engineering, or backend/full‑stack engineering role, preferably in a financial services or trading environment. - Strong understanding of programming fundamentals, including algorithms, data structures, design patterns, and paradigms. - Proficiency in one or more modern object‑oriented languages (Java/Python). - Solid experience with SQL and relational databases - Experience designing and consuming APIs using modern approaches (RESTful services; familiarity with GraphQL or gRPC is a plus). - Ability to troubleshoot and resolve issues in production systems and provide real‑time support for critical issues. - Strong written and verbal communication skills, with the ability to work effectively across engineering and business teams. - Ability to work effectively in a fast‑paced, interdisciplinary environment. Desired Additional Qualifications - Experience working in a cloud environment - Familiarity with messaging and event‑driven technologies (AWS SQS, etc.). - Familiarity with containerization and orchestration technologies - Understanding of secure distributed systems concepts, including OAuth2 / OpenID Connect, secrets management, caching, and observability (logging, metrics, tracing). - Exposure to front‑end technologies (e.g., React or Angular) or statistical / financial analysis is a plus. For San Francisco, CA and Sausalito, CA Only the salary range for this position is USD$162,000.00 - USD$215,000.00 . Additionally, employees are eligible for an annual discretionary bonus, and benefits including healthcare, leave benefits, and retirement benefits. BlackRock operates a pay-for-performance compensation philosophy and your total compensation may vary based on role, location, and firm, department and individual performance. Our benefits To help you stay energized, engaged and inspired, we offer a wide range of benefits including a strong retirement plan, tuition reimbursement, comprehensive healthcare, support for working parents and Flexible Time Off (FTO) so you can relax, recharge and be there for the people you care about. Our hybrid work model BlackRock’s hybrid work model is designed to enable a culture of collaboration and apprenticeship that enriches the experience of our employees, while supporting flexibility for all. Employees are currently required to work at least 4 days in the office per week, with the flexibility to work from home 1 day a week. Some business groups may require more time in the office due to their roles and responsibilities. We remain focused on increasing the impactful moments that arise when we work together in person – aligned with our commitment to performance and innovation. As a new joiner, you can count on this hybrid model to accelerate your learning and onboarding experience here at BlackRock. Guidance on AI use for candidates At BlackRock, AI has long been part of how we work – enhancing decision-making, improving operations, and helping us deliver better outcomes for clients. We encourage candidates to use AI thoughtfully to learn, prepare, and work more effectively; but during our interview process, we want to focus on getting to know you through your own experiences, thinking, and judgment. To support you, we’ve provided guidance on when and how to use AI during our hiring process so you can approach each step with confidence and showcase your best self. About BlackRock At BlackRock, we are all connected by one mission: to help more and more people experience financial well-being. Our clients, and the people they serve, are saving for retirement, paying for their children’s educations, buying homes and starting businesses. Their investments also help to strengthen the global economy: support businesses small and large; finance infrastructure projects that connect and power cities; and facilitate innovations that drive progress. This mission would not be possible without our smartest investment – the one we make in our employees. It’s why we’re dedicated to creating an environment where our colleagues feel welcomed, valued and supported with networks, benefits and development opportunities to help them thrive. BlackRock is proud to be an equal opportunity workplace. We are committed to equal employment opportunity to all applicants and existing employees, and we evaluate qualified applicants without regard to race, creed, color, national origin, sex (including pregnancy and gender identity/expression), sexual orientation, age, ancestry, physical or mental disability, marital status, political affiliation, religion, citizenship status, genetic information, veteran status, or any other basis protected under applicable federal, state, or local law. We recruit, hire, train, promote, pay, and administer all personnel actions without regard to race, color, religion, sex (including pregnancy, childbirth, and medical conditions related to pregnancy, childbirth, or breastfeeding), sex stereotyping (including assumptions about a person’s appearance or behavior, gender roles, gender expression, or gender identity), gender, gender identity, gender expression, national origin, age, mental or physical disability, ancestry, medical condition, marital status, military or veteran status, citizenship status, sexual orientation, genetic information, or any other status protected by applicable law. We interpret these protected statuses broadly to include both the actual status and also any perceptions and assumptions made regarding these statuses.BlackRock will consider for employment qualified applicants with arrest or conviction records in a manner consistent with the requirements of the law, including any applicable fair chance law.

California
$162K - $215K / year