Job Closed
This listing is no longer active.
Join our team at Welo Data and embark on a journey of growth and innovation.
Senior Prompt Engineer
Location
United States
Posted
127 days ago
Salary
0
Seniority
Senior
Job Description
Senior Prompt Engineer
Welo Data
Role Description Are you an expert at navigating the complex architecture of Large Language Models? Welo Data is seeking a Senior Prompt Engineer to lead the technical migration of template workflows into high-performance LLM autoraters. This role is designed for a technical specialist who understands that "perfecting a prompt" is a rigorous engineering discipline. You will leverage advanced APG/APO tools and manual refinement to ensure our automated systems meet—and exceed—human crowd baselines in accuracy and nuance. - The Mission: Automated Quality at Scale - Architectural Migration: Take full ownership of the end-to-end technical migration of templates to LLM autoraters. - Optimization Leadership: Utilize Automatic Prompt Generation (APG) and supervise Automated Prompt Optimization (APO) tools to push model performance past plateaus and deadlocks. - Metrics-Driven Excellence: Continuously measure quality against "gold data" baselines, tracking precision, recall, and $F_1$ scores to justify launch readiness. - Edge-Case Engineering: Manually draft and refine complex prompts to overcome anti-patterns and architecture gaps that automated tools can't solve. Qualifications - Bachelor’s, Master’s, or PhD in Computer Science, Data Science, Computational Linguistics, or a related analytical field. - 4+ years of experience tuning LLMs for strict, structured outputs, complex classification, and few-shot learning. - High proficiency in identifying error patterns and using SQL or data analytics tools to monitor performance. - Fast learner capable of mastering proprietary internal tools and "Goose API" style interfaces with minimal oversight. Requirements - Familiarity with shadowbot monitoring and disagreement tracking. - Experience in AI model evaluation and software engineering. - Deep understanding of semantics, logic, and Chain-of-Thought (CoT) prompting. - Proven ability to draft high-level Launch Certification Documentation. Recruitment & Onboarding - Technical Review: Submit your CV and portfolio of LLM optimization work. - Prompt Assessment: Demonstrate your ability to navigate complex template clusters and APO deadlocks. - Tooling Deep-Dive: Get access to our internal technical suite and migration workflows. - Launch: Begin your part-time engagement and lead the shift to LLM-driven autorating.
Job Requirements
- Bachelor’s, Master’s, or PhD in Computer Science, Data Science, Computational Linguistics, or a related analytical field.
- 4+ years of experience tuning LLMs for strict, structured outputs, complex classification, and few-shot learning.
- High proficiency in identifying error patterns and using SQL or data analytics tools to monitor performance.
- Fast learner capable of mastering proprietary internal tools and "Goose API" style interfaces with minimal oversight.
- Familiarity with shadowbot monitoring and disagreement tracking.
- Experience in AI model evaluation and software engineering.
- Deep understanding of semantics, logic, and Chain-of-Thought (CoT) prompting.
- Proven ability to draft high-level Launch Certification Documentation.
- Recruitment & Onboarding
- Technical Review: Submit your CV and portfolio of LLM optimization work.
- Prompt Assessment: Demonstrate your ability to navigate complex template clusters and APO deadlocks.
- Tooling Deep-Dive: Get access to our internal technical suite and migration workflows.
- Launch: Begin your part-time engagement and lead the shift to LLM-driven autorating.
Related Guides
Related Job Pages
More LLM Engineer Jobs
Member of Technical Staff – MLOps, AI Infrastructure
GenPeach AIWe are a research lab building multi-modal foundation models for unmatched creativity & human-centered AI experiences.
• Own the AI execution and infrastructure layer used by research and product teams • Design and build high-performance Python systems for: - scalable model inference - training orchestration - large-scale data processing • Partner closely with research and backend engineers to productionize models and expose them via APIs • Design and operate distributed pipelines and task queues for batch and streaming workloads • Optimize GPU inference for latency, throughput, and cost efficiency • Own the MLOps lifecycle, model deployment and versioning; monitoring and alerting • Build and maintain CI/CD pipelines for services and ML workflows • Debug and resolve performance bottlenecks across Python, GPUs, networking, and storage • Contribute to infrastructure design decisions and long-term architecture
Generative AI Engineer III
Grupo RBSFazemos jornalismo e entretenimento que conectam os gaúchos e contribuem para uma vida melhor.
• Design system architectures involving LLMs, intelligent agents, and multi-agent systems; • Develop end-to-end Generative AI pipelines, from experimentation to production deployment; • Build autonomous agents, custom tools, functions, and integrations with external APIs; • Implement orchestration between agents, memory systems, expanded context windows, and decision-making mechanisms; • Conduct evaluation, monitoring, and observability of models (RAG, agents, prompts, and behaviors); • Optimize cost, latency, and quality of production solutions; • Work with prompt versioning, automated testing, and continuous validation; • Collaborate with product, marketing, journalism, analytics, and data engineering teams to evolve the company’s Generative AI platform.
Specialist I, Generative AI and Agents Engineer
Grupo BoticárioCriamos oportunidades para a beleza transformar a vida das pessoas, e assim transformar o mundo ao nosso redor.
• The focus is on implementing autonomous reasoning flows, integration via MCP, and control of non-deterministic state. • **Orchestration of Agentive Flows:** Build loop logics (Think, Act, Observe) for complex Supply Chain tasks (stockout analysis, recalculating Sell-In) using frameworks such as **LangGraph, CrewAI, Agno or Bedrock AgentCore**. • **Implementation of MCP Servers:** Develop and expose tools via *Model Context Protocol*, creating secure bridges (APIs) between LLMs and **PostgreSQL, Iceberg** databases and Rule Engines. • **Reasoning and State Engineering:** Manage short- and long-term memory and implement **Thought Signatures** to ensure cohesion in multi-step executions (Gemini 3 and Claude 3.5). • **Tool/Function Calling Optimization:** Design strict contracts using **JSON Schemas**, implementing fallback and retry logic to mitigate hallucinations in external calls. • **AgentOps Discipline:** Ensure full AI observability (tracing, latency, token consumption, and decision paths) via **LangSmith, Phoenix or AWS CloudWatch**. • **Guardrails and Security:** Configure automated barriers (e.g., **Amazon Bedrock Guardrails**) to intercept hallucinations and block unauthorized actions on master data. **
Senior NLP/LLM Engineer
Social Discovery GroupTop world’s largest social discovery company uniting 70+ brands with 500M+ users
• Conduct experiments with LLMs and explore different architectures and techniques such as SFT, RLHF, Adapters, LoRA, and related approaches to improve model capabilities • Develop and maintain evaluation methodologies and frameworks to measure model quality, accuracy, and user satisfaction using both offline and online metrics • Optimize models for inference efficiency, speed, and scalability to meet production requirements • Collaborate closely with data scientists, software engineers, and product managers to integrate AI solutions into the product pipeline



