Welo Data logo
Welo Data

Join our team at Welo Data and embark on a journey of growth and innovation.

Senior Prompt Engineer

LLM EngineerMachine Learning EngineerTemporaryRemoteSeniorTeam 1,001-5,000

Location

United States

Posted

72 days ago

Salary

0

Seniority

Senior

Job Description

Senior Prompt Engineer

Welo Data

Role Description Are you an expert at navigating the complex architecture of Large Language Models? Welo Data is seeking a Senior Prompt Engineer to lead the technical migration of template workflows into high-performance LLM autoraters. This role is designed for a technical specialist who understands that "perfecting a prompt" is a rigorous engineering discipline. You will leverage advanced APG/APO tools and manual refinement to ensure our automated systems meet—and exceed—human crowd baselines in accuracy and nuance. - The Mission: Automated Quality at Scale - Architectural Migration: Take full ownership of the end-to-end technical migration of templates to LLM autoraters. - Optimization Leadership: Utilize Automatic Prompt Generation (APG) and supervise Automated Prompt Optimization (APO) tools to push model performance past plateaus and deadlocks. - Metrics-Driven Excellence: Continuously measure quality against "gold data" baselines, tracking precision, recall, and $F_1$ scores to justify launch readiness. - Edge-Case Engineering: Manually draft and refine complex prompts to overcome anti-patterns and architecture gaps that automated tools can't solve. Qualifications - Bachelor’s, Master’s, or PhD in Computer Science, Data Science, Computational Linguistics, or a related analytical field. - 4+ years of experience tuning LLMs for strict, structured outputs, complex classification, and few-shot learning. - High proficiency in identifying error patterns and using SQL or data analytics tools to monitor performance. - Fast learner capable of mastering proprietary internal tools and "Goose API" style interfaces with minimal oversight. Requirements - Familiarity with shadowbot monitoring and disagreement tracking. - Experience in AI model evaluation and software engineering. - Deep understanding of semantics, logic, and Chain-of-Thought (CoT) prompting. - Proven ability to draft high-level Launch Certification Documentation. Recruitment & Onboarding - Technical Review: Submit your CV and portfolio of LLM optimization work. - Prompt Assessment: Demonstrate your ability to navigate complex template clusters and APO deadlocks. - Tooling Deep-Dive: Get access to our internal technical suite and migration workflows. - Launch: Begin your part-time engagement and lead the shift to LLM-driven autorating.

Job Requirements

  • Bachelor’s, Master’s, or PhD in Computer Science, Data Science, Computational Linguistics, or a related analytical field.
  • 4+ years of experience tuning LLMs for strict, structured outputs, complex classification, and few-shot learning.
  • High proficiency in identifying error patterns and using SQL or data analytics tools to monitor performance.
  • Fast learner capable of mastering proprietary internal tools and "Goose API" style interfaces with minimal oversight.
  • Familiarity with shadowbot monitoring and disagreement tracking.
  • Experience in AI model evaluation and software engineering.
  • Deep understanding of semantics, logic, and Chain-of-Thought (CoT) prompting.
  • Proven ability to draft high-level Launch Certification Documentation.
  • Recruitment & Onboarding
  • Technical Review: Submit your CV and portfolio of LLM optimization work.
  • Prompt Assessment: Demonstrate your ability to navigate complex template clusters and APO deadlocks.
  • Tooling Deep-Dive: Get access to our internal technical suite and migration workflows.
  • Launch: Begin your part-time engagement and lead the shift to LLM-driven autorating.

Related Job Pages

More LLM Engineer Jobs

GenPeach AI logo

Member of Technical Staff – MLOps, AI Infrastructure

GenPeach AI

We are a research lab building multi-modal foundation models for unmatched creativity & human-centered AI experiences.

LLM Engineer72 days ago
Full TimeRemoteTeam 1-10H1B No Sponsor

• Own the AI execution and infrastructure layer used by research and product teams • Design and build high-performance Python systems for: - scalable model inference - training orchestration - large-scale data processing • Partner closely with research and backend engineers to productionize models and expose them via APIs • Design and operate distributed pipelines and task queues for batch and streaming workloads • Optimize GPU inference for latency, throughput, and cost efficiency • Own the MLOps lifecycle, model deployment and versioning; monitoring and alerting • Build and maintain CI/CD pipelines for services and ML workflows • Debug and resolve performance bottlenecks across Python, GPUs, networking, and storage • Contribute to infrastructure design decisions and long-term architecture

Switzerland
Job Closed
Grupo RBS logo

Generative AI Engineer III

Grupo RBS

Fazemos jornalismo e entretenimento que conectam os gaúchos e contribuem para uma vida melhor.

LLM Engineer72 days ago
Full TimeRemoteTeam 1,001-5,000Since 1957H1B No Sponsor

• Design system architectures involving LLMs, intelligent agents, and multi-agent systems; • Develop end-to-end Generative AI pipelines, from experimentation to production deployment; • Build autonomous agents, custom tools, functions, and integrations with external APIs; • Implement orchestration between agents, memory systems, expanded context windows, and decision-making mechanisms; • Conduct evaluation, monitoring, and observability of models (RAG, agents, prompts, and behaviors); • Optimize cost, latency, and quality of production solutions; • Work with prompt versioning, automated testing, and continuous validation; • Collaborate with product, marketing, journalism, analytics, and data engineering teams to evolve the company’s Generative AI platform.

Brazil
Job Closed
Grupo Boticário logo

Especialista I – Engenheiro de IA Generativa, Agentes

Grupo Boticário

Criamos oportunidades para a beleza transformar a vida das pessoas, e assim transformar o mundo ao nosso redor.

LLM Engineer72 days ago
Full TimeRemoteTeam 10,001+Since 2010H1B No Sponsor

• O foco é a implementação de fluxos de raciocínio autônomo, integração via MCP e controle de estado não-determinístico. • **Orquestração de Fluxos Agentivos:** Construir lógicas de loops (Pensar, Agir, Observar) para tarefas complexas de Supply Chain (análise de rupturas, recalcular Sell-In) utilizando frameworks como **LangGraph, CrewAI, Agno ou Bedrock AgentCore**. • **Implementação de Servidores MCP:** Desenvolver e expor ferramentas via *Model Context Protocol*, criando pontes seguras (APIs) entre LLMs e bases **PostgreSQL, Iceberg** e Motores de Regras. • **Engenharia de Raciocínio e Estado:** Gerenciar memória de curto/longo prazo e implementar **Thought Signatures** para garantir coesão em execuções multi-passos (Gemini 3 e Claude 3.5). • **Otimização de Tool/Function Calling:** Desenhar contratos estritos de **JSON Schemas**, implementando lógicas de fallback e retentativas para mitigar alucinações em chamadas externas. • **Disciplina de AgentOps:** Garantir a observabilidade total da IA (tracing, latência, consumo de tokens e caminhos de decisão) via **LangSmith, Phoenix ou AWS CloudWatch**. • **Guardrails e Segurança:** Configurar barreiras automatizadas (ex: **Amazon Bedrock Guardrails**) para interceptar alucinações e bloquear ações não autorizadas em dados mestres.

Brazil
Social Discovery Group logo

Senior NLP/LLM Engineer

Social Discovery Group

Top world’s largest social discovery company uniting 70+ brands with 500M+ users

LLM Engineer73 days ago
Full TimeRemoteTeam 1,001-5,000Since 20 yearsH1B No Sponsor

• Conduct experiments with LLMs and explore different architectures and techniques such as SFT, RLHF, Adapters, LoRA, and related approaches to improve model capabilities • Develop and maintain evaluation methodologies and frameworks to measure model quality, accuracy, and user satisfaction using both offline and online metrics • Optimize models for inference efficiency, speed, and scalability to meet production requirements • Collaborate closely with data scientists, software engineers, and product managers to integrate AI solutions into the product pipeline

Serbia
Job Closed