AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
Data Engineer (GCP) ID56375
Location
India
Posted
55 days ago
Salary
0
Seniority
Mid Level
Job Description
Data Engineer (GCP) ID56375
AgileEngine
AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards. WHY JOIN US If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you! ABOUT THE ROLE WHAT YOU WILL DO - Build and maintain scalable, distributed, fault-tolerant data pipelines on GCP; - Develop and manage lakehouse layers and Delta Lake workflows using BigQuery and Dataproc; - Collaborate with stakeholders across data engineering, compliance, and business teams; - Design and implement pipelines to acquire, normalise, transform, and release large volumes of financial data; - Design and implement bitemporal data models on BigQuery for regulatory-grade time-series datasets; - Build and maintain testing frameworks for data pipelines and transformation logic; - Own end-to-end solutions including ingestion pipelines, QA workflows, correction management, and audit trails; - Contribute to shared platform services in a collaborative environment; - Support implementation of AI solutions including data ingestion, anomaly detection, and semantic search using Vertex AI. MUST HAVES - 6–8 years of experience in data engineering; - Proficiency in Python for data pipelines, transformation logic, and automation; - Proficiency in SQL with hands-on experience in BigQuery including partitioning, clustering, and time-series queries; - Experience with Cloud Composer (Apache Airflow) for pipeline orchestration; - Working knowledge of Dataproc (Apache Spark) for batch ingestion and incremental processing; - Experience with AI-assisted development tools such as GitHub Copilot or similar; - Experience with Git version control and collaboration workflows; - Familiarity with REST APIs for integrations; - Familiarity with GCP technologies (Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM, VPC Service Controls); - Understanding of financial data concepts related to equities and& other asset classes; - Upper-intermediate English level. NICE TO HAVES - Knowledge of data libraries such as pandas or PySpark; - Experience with columnar storage and time-series analytics tools such as ClickHouse; - Familiarity with Dataplex for data governance and lineage; - Understanding of Change Data Capture (CDC) using Datastream; - Understanding of bitemporal data modeling concepts; - Knowledge of financial reference data such as equities, fixed income, or corporate actions; - Experience with BigQuery cost management techniques; - Experience with CI/CD pipelines and Terraform for infrastructure as code; - Exposure to LLMs and& Agentic AI using Vertex AI for data-related use cases. PERKS AND BENEFITS - Remote work & Local connection: Work where you feel most productive and connect with your team in periodic meet-ups to strengthen your network and connect with other top experts. - Legal presence in India: We ensure full local compliance with a structured, secure work environment tailored to Indian regulations. - Competitive Compensation in INR: Fair compensation in INR with dedicated budgets for your personal growth, education, and wellness. - Innovative Projects: Leverage the latest tech and create cutting-edge solutions for world-recognized clients and the hottest startups.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Engineer
IRIUM PortugalIRIUM is a company with dynamic and proactive professionals. Our values are responsibility and commitment to work quality. This is the spirit we are looking for at IRIUM, whatever your age is. If you recognize yourself in this, this is your company! We can build the future together. Let’s talk! At IRIUM, we defend a world without stereotypes or limitations and we believe in equality for all, principles that we subscribe to in our Equality Plan and Code of Ethics, guaranteeing equal treatment and opportunities regardless of any personal, physical or social condition.
At IRIUM we want you to always chase your dreams. Here, prepare yourself to conquer your goals, while enjoying the journey. We are currently looking for. Data Engineer Requirements: WHAT WE'RE LOOKING FOR Expert in Apache Kafka: configuring, managing brokers, partitioning, replication factors, and performance tuning. Advanced proficiency in Kafka Streams and Kafka Connect to build real-time pipelines, integrating systems from various sources and destinations. Strong understanding of the Confluent ecosystem, including Schema Registry, KSQL, and monitoring tools to ensure robust operations. Expertise in event-driven architectures and patterns such as publish-subscribe for distributed systems. Skilled in developing and managing APIs and microservices to integrate applications with data platforms. Proficient in programming languages like Java, Scala, and Python for real-time data stream processing. Hands-on experience with complementary technologies like Spark Streaming, Flink, or other real-time processing tools. Knowledgeable in relational and NoSQL databases (e.g., PostgreSQL, Cassandra, MongoDB). Practical experience with cloud environments (AWS, Azure, GCP) focusing on big data solutions. Familiarity with CI/CD tools (e.g., Git, Jenkins, Docker, Kubernetes) for automation and continuous delivery. Technical leadership skills to coordinate and mentor data engineering teams. Strong analytical abilities to solve complex problems and identify bottlenecks in data processing systems. Excellent communication skills to translate business requirements into technical solutions. Proactive in exploring new technologies and crafting strategic approaches to optimize processes and architectures. Designed and implemented high-performance streaming architectures using Kafka and Confluent for enterprise-scale projects. Built real-time data pipelines to process large volumes, integrating multiple enterprise systems. Applied best practices for data governance, quality control, and compliance (e.g., GDPR). Led initiatives to migrate monolithic systems to Kafka Streams-based architectures. Certifications (plus): Confluent Certified Developer for Apache Kafka. Cloud certifications (e.g., AWS Certified Solutions Architect, Google Cloud Professional Data Engineer). Additional certifications in Big Data tools or related technologies. Location: remote, Portugal What do we offer? ➡ An innovative and growing company, with a lot of opportunities for professional development. ➡ Retribution according to your experience and performance. Access to flexible pay and medical insurance as a social benefit. ➡ Unlimited access to technological training in free mode. IRIUM is a company with dynamic and proactive professionals. Our values are responsibility and commitment to work quality. This is the spirit we are looking for at IRIUM, whatever your age is. If you recognize yourself in this, this is your company! We can build the future together. Let’s talk! Send your CV to: recrutamento@irium.pt At IRIUM we defend a world without stereotypes or limitations and we believe in equality for all, principles that we subscribe to in our Equality Plan and Code of Ethics, guaranteeing equal treatment and opportunities regardless of any personal, physical or social condition.
Senior Data Engineer ***EST Preferred
Reed TechnologyLexisNexis Legal & Professional® provides legal, regulatory, and business information and analytics that help customers increase their productivity, improve decision-making, achieve better outcomes, and advance the rule of law around the world.
About Us LexisNexis Legal & Professional, which serves customers in more than 150 countries with 11,300 employees worldwide, is part of RELX, a global provider of information-based analytics and decision tools for professional and business customers. About the Role We are seeking a highly skilled Senior Data Engineer with expertise in Retrieval-Augmented Generation (RAG) and strong Python programming skills to join our innovative team. The ideal candidate will have a deep understanding of AI and machine learning principles, experience with RAG models, and a passion for developing cutting-edge AI solutions. Responsibilities: - Test, evaluate and implement AI models with a focus on Retrieval-Augmented Generation (RAG). - Strong prompt engineering and prompt testing skills to derive the desired outcome - Collaborate with cross-functional teams to integrate AI solutions into existing systems. - Optimize and fine-tune RAG models for performance and scalability cost saving. - Conduct research and stay up-to-date with the latest advancements in AI and machine learning. - Write clean, efficient, and maintainable code in Python. - Perform data preprocessing, feature engineering, and model evaluation. - Troubleshoot and debug AI models and applications in a mono-repo settings. - Document AI models, processes, and workflows. - Train entry-level data engineers as directed by department management, ensuring they are knowledgeable in critical aspects of their roles. - Keep abreast of new technology developments. - Design and works with complex data models. - Identify opportunities to apply automation or other tools to improve effectiveness or efficiency. - Work closely with other development team members to understand complex product requirements and translate them into data engineering and/or data management designs. - Mentor less senior data engineers on methodologies and optimization techniques. - All other duties as assigned Requirements - Proven experience as a Data Engineer/ AI Engineer or similar role. - Strong proficiency in Python programming. - In-depth knowledge of Retrieval-Augmented Generation (RAG) models. - Proven experience in prompt engineering - Experience with machine learning frameworks and libraries (e.g., TensorFlow, PyTorch). - Proficiency in data preprocessing and feature engineering techniques. - Excellent problem-solving skills and attention to detail. - Strong communication and teamwork abilities. Preferred Skills: - Experience with cloud platforms (e.g., AWS, Azure, Google Cloud). - Knowledge of natural language processing (NLP) techniques. - Familiarity with version control systems (e.g., Git). - Experience with deploying AI models in production environments. Work in a way that works for you We promote a healthy work/life balance across the organization. We offer an appealing working prospect for our people. With numerous well-being initiatives, shared parental leave, study assistance and sabbaticals, we will help you meet your immediate responsibilities and your long-term goals. Working for you We know that your wellbeing and happiness are key to a long and successful career. These are some of the benefits we are delighted to offer: - Health Benefits: Comprehensive, multi-carrier program for medical, dental and vision benefits - Retirement Benefits: 401(k) with match and an Employee Share Purchase Plan - Wellbeing: Wellness platform with incentives, Headspace app subscription, Employee Assistance and Time-off Programs - Short-and-Long Term Disability, Life and Accidental Death Insurance, Critical Illness, and Hospital Indemnity - Family Benefits, including bonding and family care leaves, adoption and surrogacy benefits - Health Savings, Health Care, Dependent Care and Commuter Spending Accounts - Up to two days of paid leave each to participate in Employee Resource Groups and to volunteer with your charity of choice About the Business LexisNexis Legal & Professional® provides legal, regulatory, and business information and analytics that help customers increase their productivity, improve decision-making, achieve better outcomes, and advance the rule of law around the world. As a digital pioneer, the company was the first to bring legal and business information online with its Lexis® and Nexis® services. U.S. National Base Pay Range: $78,800 - $131,300. Geographic differentials may apply in some locations to better reflect local market rates. This job is eligible for an annual incentive bonus. We know your well-being and happiness are key to a long and successful career. We are delighted to offer country specific benefits. Click here to access benefits specific to your location. We are committed to providing a fair and accessible hiring process. If you have a disability or other need that requires accommodation or adjustment, please let us know by completing our Applicant Request Support Form or please contact 1-855-833-5120. Criminals may pose as recruiters asking for money or personal information. We never request money or banking details from job applicants. Learn more about spotting and avoiding scams here. Please read our Candidate Privacy Policy. We are an equal opportunity employer: qualified applicants are considered for and treated during employment without regard to race, color, creed, religion, sex, national origin, citizenship status, disability status, protected veteran status, age, marital status, sexual orientation, gender identity, genetic information, or any other characteristic protected by law. USA Job Seekers: EEO Know Your Rights.
Oncology Data Specialist
Brown MedicineOne of the largest nonprofit, academic, multi-specialty medical groups in RI.
SUMMARY: Reports to the Oncology Data Management Manager, and under general direction, abstracts information from medical records of patients with positive or possible cancer diagnoses. The information is entered into a computerized database (IMPAC) system to establish a Cancer Registry data bank from which to draw information for annual reports, site specific studies, data requests and to assist physicians with short and long-term cancer patient care evaluations, etc. Brown University Health employees are expected to successfully role model the organization's values of Compassion, Accountability, Respect, and Excellence as these values guide our everyday actions with patients, customers and one another. In addition to our values, all employees are expected to demonstrate the core Success Factors which tell us how we work together and how we get things done. The core Success Factors include: Instill Trust and Value Differences Patient and Community Focus and Collaborate RESPONSIBILITIES: Operates computers and peripheral equipment and is skilled in the use of various software applications necessary to develop, prepare and maintain computerized records and develop reports. Responds to user questions regarding system functionality, capabilities, usage, limitations and the like. Assists in resolving complex issues related to review, analysis and abstraction of medical record information pertinent to documenting information on cancer patients. Utilizes specialized software for cancer registry database management to establish import and export links to other software packages and to download and upload data for review and reporting. Develops and prepares various reports derived from electronic database utilizing a variety of software packages such as word processing, spreadsheets, graphics, Microsoft query applications and the like. Regularly exports new cancer care data to the Rhode Island State Department of Health utilizing communications software. Retrieves Patient Disease Index and reviews against the active and suspense database contained in the Registry to ascertain if a patient is a case for follow-up, newly diagnosed and requires accessioning into the database or has developed a subsequent new primary site, requires a staging form to be appended to the medical record and/or completed/signed by the attending physician, etc. Reviews the entire medical record to ascertain patient identification data, history of admissions, method of diagnosis, history and types of treatments and intent of treatment (cure or palliation) against the criteria for cancer case selection set forth by the American College of Surgeons (AcoS), the Rhode Island Department of Health, Rhode Island Hospital's Cancer Committee and other regulatory agencies. Abstracts cases thoroughly and concisely, substantiating codes, histology and staging. Contacts outside institutions for additional treatment information if necessary. Adds follow-up to all cases. Interprets data and uses the ACoS Guidelines, Collaborative Staging Manual, and Multiple Primary and Histology Manual for recording extent of primary tumor and absence or presence of distant metastases, ICD-O (International Classification of Disease for Oncology) for tumor nomenclature and coding, FORDS Manual for coding, and the Rhode Island Census Tract Coding Guide for assignment of Census Tract Codes. Familiarity with CoC Cancer Program Standards and participates in continuous review of the status of compliance with Standards. Accesses the health database on the World Wide Web to review cases lost to follow-up against various national death indexes. Notes date of death, cause of death and presence of cancer at death in the electronic record. Receives pathology reports and reviews both clinical history and microscopic diagnosis to identify new, potential and established patients requiring follow-up and updates information in appropriate electronic database. Receives changes from other source lists from the Hospital Association of Rhode Island which are reviewed against the electronic databases for completeness and accuracy of the data. Discrepancies are investigated before any changes are made to the electronic record and death information is entered into the appropriate databases. Monitors and records lifetime follow-up on each patient listed in the registry via phone contacts, letters to physicians, patients and other agencies as appropriate. Compares daily obituaries in the newspaper and on the Internet against master files or registered and suspected cases. Notes date of death in appropriate electronic records. Ensures staging form is in the record if appropriate, and, if not, refers to appropriate source to obtain the form. Participates in the evaluation of current and potential system capabilities against current and future needs. Ensures proper operating condition of computer equipment, making minor adjustments as appropriate and contacting others for more extensive maintenance. Resolves hardware and software problems. Ensures new hardware and/or software is functional and meets data requirements. Prepares and maintains related system procedural/operating manuals. Occasional travel to local, regional or nationally approved continuing education programs is required. MINIMUM QUALIFICATIONS: BASIC KNOWLEDGE: College-level knowledge of computer operations and software applications. Anatomy and Physiology, Medical Terminology. Demonstrates solid knowledge of Cancer Registry data collection requirements of the American College of Surgeons, automated information and data management systems including hardware and software interfaces, expertise in internet research. Requires analytical, communication, organizational, and time-management skills to assess information needs, perform quality assurance, and reporting. Requires interpersonal skills to effectively interact with various cancer registry data users within and outside the Hospital regarding cancer registry data. Thorough knowledge of medical terminology, anatomy and physiology, disease processes, current governmental/regulatory requirements. EXPERIENCE: A minimum three years progressively more responsible related experience in performing full range of duties of cancer registry, including familiarity with data base development and maintenance. Experience should demonstrate ability to interpret more complex medical records, extent of disease and absence/presence of distant metastasis and to prepare a variety of records using computerized registry data. CTR required. WORK ENVIRONMENT AND PHYSICAL REQUIREMENTS: Works in normal office environment with generally good working conditions. SUPERVISORY RESPONSIBILITY: None Pay Range: $25.41-$40.10Brown University Health is committed to providing equal employment opportunities and maintaining a work environment free from all forms of unlawful discrimination and harassment. Location: Remote-Rhode Island - N/A Providence, Rhode Island 02901Work Type: Flexible Schedule: M-F 7:00 am - 3:30 pm vs T-F 7:00 am - 5:30 pmWork Shift: DayDriving Required: NoUnion: United Nurses And Allied Professional
Data Engineer - United States
cyberuCornerstone powers the potential of organizations and their people to thrive in a changing world. Cornerstone Galaxy, the complete AI-powered workforce agility platform, meets organizations where they are. With Galaxy, organizations can identify skills gaps and development opportunities, retain and engage top talent, and provide multimodal learning experiences to meet the diverse needs of the modern workforce. More than 7,000 organizations and 100 million+ users in 180+ countries and in nearly 50 languages use Cornerstone Galaxy. Build high-performing, future-ready organizations and people today.
We're looking for a Data Engineer - United States This role is Remote, United StatesResponsibilities • Design, build and maintain batch or real-time data pipelines in production. • Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources. • Develop ETL (extract, transform, load) processes to help extract and manipulate data from multiple sources. • Automate data workflows such as data ingestion, aggregation, and ETL processing. • Prepare raw data in Data Warehouses into a consumable dataset for both technical and non-technical stakeholders. • Partner with data scientists and functional leaders in sales, marketing, and product to deploy machine learning models in production. • Build, maintain, and deploy data products for analytics and data science teams on cloud platforms (e.g. AWS, Azure, GCP). • Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures. • Monitor data systems performance and implement optimization strategies. • Leverage data controls to maintain data privacy, security, compliance, and quality for allocated areas of ownership. Minimum Qualifications • Advanced SQL skills and experience with relational databases and database design. • Experience working with cloud Data Warehouse solutions – Databricks, Apache Spark. • Experience working with data ingestion tools such as Fivetran, stitch, or Matillion. • Working knowledge of Cloud-based solutions (e.g. AWS, Azure, GCP). • Experience building and deploying machine learning models in production. • Strong proficiency in object-oriented languages: Python, Java, C++, Scala. • Strong proficiency in scripting languages like Bash. • Strong proficiency in data pipeline and workflow management tools (e.g., Airflow). • Strong project management and organizational skills. • Excellent problem-solving, communication, and organizational skills. • Proven ability to work independently and with a team. What will make you stand out • Good understanding of NoSQL databases like CrateDB, Redis, Cassandra, MongoDB, or Neo4j. • Experience with working on large data sets and distributed computing (e.g. Hive/Hadoop/Spark/Presto/MapReduce) Our Culture: Spark Greatness. Shatter Boundaries. Share Success. Are you ready? Because here, right now – is where the future of work is happening. Where curious disruptors and change innovators like you are helping communities and customers enable everyone – anywhere – to learn, grow and advance. To be better tomorrow than they are today. Who We Are: Cornerstone powers the potential of organizations and their people to thrive in a changing world. Cornerstone Galaxy, the complete AI-powered workforce agility platform, meets organizations where they are. With Galaxy, organizations can identify skills gaps and development opportunities, retain and engage top talent, and provide multimodal learning experiences to meet the diverse needs of the modern workforce. More than 7,000 organizations and 100 million+ users in 180+ countries and in nearly 50 languages use Cornerstone Galaxy to build high-performing, future-ready organizations and people today. Total Rewards: At Cornerstone, we are dedicated to inspiring excellence and pushing boundaries in everything we do. Our compensation strategy is based on three fundamental principles: equitable pay, market-driven research, and skill-based appraisals. As part of our mission to share success and empower individuals to thrive in an ever-changing world, the listed salary range is just one element of Cornerstone’s comprehensive compensation package. This compensation package may also include annual bonuses, short- and program-specific awards depending on the role, and a comprehensive benefit offering. The disclosed salary range reflects the geographic differential based on the location of the position if applicable. The starting salary for the successful applicant will depend on several job-related factors, including education, training, experience, certifications, location, business needs, and market demands. This range is based on a full-time position and may be adjusted in the future. Join us in shaping the future of work — tomorrow, together. Experience flexibility and empowerment in your career at Cornerstone. The BASE salary range for this position is: 88100 - 141000 USD. Check us out on LinkedIn, Comparably, Glassdoor, and Facebook! Equal Employment Opportunity has been, and will continue to be, a fundamental commitment at Cornerstone OnDemand. All qualified applicants are given consideration regardless of race, religion, color, gender, sex, age, sexual orientation, gender identity, national origin, marital status, citizenship status, disability, veteran status, or any other protected class as provided in applicable Federal, State, or Local fair employment laws. If you have a disability or special need that requires accommodation, please contact us at careers@csod.com or +1 855 454 8433. Read the EEO is the Law poster here, and the supplementary poster here Read the Read the Pay Transparency Nondiscrimination Provision poster here

