Job Closed

This listing is no longer active.

Welo Data logo
Welo Data

Join our team at Welo Data and embark on a journey of growth and innovation.

Senior Prompt Engineer

LLM EngineerMachine Learning EngineerTemporaryRemoteSeniorTeam 1,001-5,000

Location

United States

Posted

127 days ago

Salary

0

Seniority

Senior

Job Description

Senior Prompt Engineer

Welo Data

Role Description Are you an expert at navigating the complex architecture of Large Language Models? Welo Data is seeking a Senior Prompt Engineer to lead the technical migration of template workflows into high-performance LLM autoraters. This role is designed for a technical specialist who understands that "perfecting a prompt" is a rigorous engineering discipline. You will leverage advanced APG/APO tools and manual refinement to ensure our automated systems meet—and exceed—human crowd baselines in accuracy and nuance. - The Mission: Automated Quality at Scale - Architectural Migration: Take full ownership of the end-to-end technical migration of templates to LLM autoraters. - Optimization Leadership: Utilize Automatic Prompt Generation (APG) and supervise Automated Prompt Optimization (APO) tools to push model performance past plateaus and deadlocks. - Metrics-Driven Excellence: Continuously measure quality against "gold data" baselines, tracking precision, recall, and $F_1$ scores to justify launch readiness. - Edge-Case Engineering: Manually draft and refine complex prompts to overcome anti-patterns and architecture gaps that automated tools can't solve. Qualifications - Bachelor’s, Master’s, or PhD in Computer Science, Data Science, Computational Linguistics, or a related analytical field. - 4+ years of experience tuning LLMs for strict, structured outputs, complex classification, and few-shot learning. - High proficiency in identifying error patterns and using SQL or data analytics tools to monitor performance. - Fast learner capable of mastering proprietary internal tools and "Goose API" style interfaces with minimal oversight. Requirements - Familiarity with shadowbot monitoring and disagreement tracking. - Experience in AI model evaluation and software engineering. - Deep understanding of semantics, logic, and Chain-of-Thought (CoT) prompting. - Proven ability to draft high-level Launch Certification Documentation. Recruitment & Onboarding - Technical Review: Submit your CV and portfolio of LLM optimization work. - Prompt Assessment: Demonstrate your ability to navigate complex template clusters and APO deadlocks. - Tooling Deep-Dive: Get access to our internal technical suite and migration workflows. - Launch: Begin your part-time engagement and lead the shift to LLM-driven autorating.

Job Requirements

  • Bachelor’s, Master’s, or PhD in Computer Science, Data Science, Computational Linguistics, or a related analytical field.
  • 4+ years of experience tuning LLMs for strict, structured outputs, complex classification, and few-shot learning.
  • High proficiency in identifying error patterns and using SQL or data analytics tools to monitor performance.
  • Fast learner capable of mastering proprietary internal tools and "Goose API" style interfaces with minimal oversight.
  • Familiarity with shadowbot monitoring and disagreement tracking.
  • Experience in AI model evaluation and software engineering.
  • Deep understanding of semantics, logic, and Chain-of-Thought (CoT) prompting.
  • Proven ability to draft high-level Launch Certification Documentation.
  • Recruitment & Onboarding
  • Technical Review: Submit your CV and portfolio of LLM optimization work.
  • Prompt Assessment: Demonstrate your ability to navigate complex template clusters and APO deadlocks.
  • Tooling Deep-Dive: Get access to our internal technical suite and migration workflows.
  • Launch: Begin your part-time engagement and lead the shift to LLM-driven autorating.

Related Job Pages

More LLM Engineer Jobs

GenPeach AI logo

Member of Technical Staff – MLOps, AI Infrastructure

GenPeach AI

We are a research lab building multi-modal foundation models for unmatched creativity & human-centered AI experiences.

LLM Engineer127 days ago
Full TimeRemoteTeam 1-10H1B No Sponsor

• Own the AI execution and infrastructure layer used by research and product teams • Design and build high-performance Python systems for: - scalable model inference - training orchestration - large-scale data processing • Partner closely with research and backend engineers to productionize models and expose them via APIs • Design and operate distributed pipelines and task queues for batch and streaming workloads • Optimize GPU inference for latency, throughput, and cost efficiency • Own the MLOps lifecycle, model deployment and versioning; monitoring and alerting • Build and maintain CI/CD pipelines for services and ML workflows • Debug and resolve performance bottlenecks across Python, GPUs, networking, and storage • Contribute to infrastructure design decisions and long-term architecture

Switzerland
Job Closed
Grupo RBS logo

Generative AI Engineer III

Grupo RBS

Fazemos jornalismo e entretenimento que conectam os gaúchos e contribuem para uma vida melhor.

LLM Engineer128 days ago
Full TimeRemoteTeam 1,001-5,000Since 1957H1B No Sponsor

• Design system architectures involving LLMs, intelligent agents, and multi-agent systems; • Develop end-to-end Generative AI pipelines, from experimentation to production deployment; • Build autonomous agents, custom tools, functions, and integrations with external APIs; • Implement orchestration between agents, memory systems, expanded context windows, and decision-making mechanisms; • Conduct evaluation, monitoring, and observability of models (RAG, agents, prompts, and behaviors); • Optimize cost, latency, and quality of production solutions; • Work with prompt versioning, automated testing, and continuous validation; • Collaborate with product, marketing, journalism, analytics, and data engineering teams to evolve the company’s Generative AI platform.

Brazil
Job Closed
Grupo Boticário logo

Specialist I, Generative AI and Agents Engineer

Grupo Boticário

Criamos oportunidades para a beleza transformar a vida das pessoas, e assim transformar o mundo ao nosso redor.

LLM Engineer128 days ago
Full TimeRemoteTeam 10,001+Since 2010H1B No Sponsor

• The focus is on implementing autonomous reasoning flows, integration via MCP, and control of non-deterministic state. • **Orchestration of Agentive Flows:** Build loop logics (Think, Act, Observe) for complex Supply Chain tasks (stockout analysis, recalculating Sell-In) using frameworks such as **LangGraph, CrewAI, Agno or Bedrock AgentCore**. • **Implementation of MCP Servers:** Develop and expose tools via *Model Context Protocol*, creating secure bridges (APIs) between LLMs and **PostgreSQL, Iceberg** databases and Rule Engines. • **Reasoning and State Engineering:** Manage short- and long-term memory and implement **Thought Signatures** to ensure cohesion in multi-step executions (Gemini 3 and Claude 3.5). • **Tool/Function Calling Optimization:** Design strict contracts using **JSON Schemas**, implementing fallback and retry logic to mitigate hallucinations in external calls. • **AgentOps Discipline:** Ensure full AI observability (tracing, latency, token consumption, and decision paths) via **LangSmith, Phoenix or AWS CloudWatch**. • **Guardrails and Security:** Configure automated barriers (e.g., **Amazon Bedrock Guardrails**) to intercept hallucinations and block unauthorized actions on master data. **

Brazil
Social Discovery Group logo

Senior NLP/LLM Engineer

Social Discovery Group

Top world’s largest social discovery company uniting 70+ brands with 500M+ users

LLM Engineer129 days ago
Full TimeRemoteTeam 1,001-5,000Since 20 yearsH1B No Sponsor

• Conduct experiments with LLMs and explore different architectures and techniques such as SFT, RLHF, Adapters, LoRA, and related approaches to improve model capabilities • Develop and maintain evaluation methodologies and frameworks to measure model quality, accuracy, and user satisfaction using both offline and online metrics • Optimize models for inference efficiency, speed, and scalability to meet production requirements • Collaborate closely with data scientists, software engineers, and product managers to integrate AI solutions into the product pipeline

Serbia
Job Closed