"Any sufficiently advanced technology is indistinguishable from magic." - Arthur C. Clarke, Third Law of Science Fiction
LLM Evaluation Engineer
Location
United States
Posted
117 days ago
Salary
0
Seniority
Lead
Job Description
LLM Evaluation Engineer
ThirdLaw Molecular
• Build the evaluation layer in the ThirdLaw platform for LLM prompts and responses • Design and tune guardrails, classifiers, and semantic judgment systems in real-time • Implement evaluation strategies with semantic similarity, foundation model scoring, and rule-based systems • Integrate model outputs with downstream enforcement actions (e.g. redaction, escalation, blocking) • Prototype, tune, and productize small language models for classification, labeling, or scoring • Collaborate with data infrastructure engineers to connect evaluation logic with ingestion and storage • Build tools to observe, debug, and improve evaluator performance across data distributions • Define abstractions for reusable evaluation components that can scale across use cases
Job Requirements
- 7+ years of experience in ML systems or AI engineering roles
- At least 1–2 years working directly with LLMs, NLP pipelines, or semantic search
- Deep understanding of foundation models (e.g. OpenAI, Claude, Mistral, Llama) and APIs
- Hands-on experience with vector search (e.g. FAISS, Qdrant, Weaviate) and embeddings pipelines
- Proven ability to implement real-time or near-real-time evaluation logic using semantic similarity, classifier scoring, or structured rules
- Strong in Python, with familiarity using libraries like Hugging Face Transformers, LangChain, and PyTorch or TensorFlow
- Ability to reason about model behavior, test prompt configurations, and debug complex decision logic in production
Benefits
- Generous benefits
- Market cash compensation
- Above-market equity
- Well-designed benefits
Related Guides
Related Categories
Related Job Pages
More Engineer Jobs
Chief Engineer – Container Vessel
Columbia ShipmanagementA leading organisation with 45 years of experience providing world-class ship management and maritime services globally
• Responsible for leading Engine room team to ensure smooth operation of Engines and machinery equipment. • Experience in repairs and dry docking is required. • Good communication and leadership skills. • Effective English communication. • Safety culture and mindset. • Experience with multicultural working environment.
• Monitor, review, and execute all facets of commissioning-related projects providing exceptional technical and field activities • Liaise with team members, outside consultants, and contractors to assist in leading cross-functional teams on various projects • Develop strong Commissioning Plans and Commissioning Specifications while jointly managing commissioning projects and developing activity schedules • Conduct comprehensive field-based commissioning activities including site visits, development and approval of start-up and test forms, and managing execution of Functional Test scripts • Perform Commissioning Design Drawing & Specification Reviews as well as Submittal (shop drawing) reviews • Chair Commissioning Meetings while using calibrated testing equipment in the commissioning process • Test mechanical and HVAC&R systems, building management & automation systems, plumbing equipment, lighting controls systems, and normal/emergency power systems • Develop, track, and resolve issues using an Issues Log while developing and executing seasonal/performance tests
Critical Power Field Technician, UPS Engineer
Pearce ServicesProviding mission-critical infrastructure solutions to create a more connected and sustainable future.
• Install electrical equipment such as small rack-mounted UPS equipment, various-sized batteries, and UPS/Battery monitoring equipment. • Maintain Battery Cabinets and racks, Maintenance Bypass, Battery Monitoring Equipment, DC Systems, and other electrical equipment. • Periodic and lifecycle-driven performance of remedial services, including failure analysis and troubleshooting, component replacements and repairs, and battery string replacements and installations. • Customer relationship building – Provide excellent customer service and upsell products. • Field troubleshooting calls from customers to a satisfactory resolution. • Quickly respond to critical situations. • Teach and mentor crew members on proper techniques for site work, troubleshooting, repair, and analyzing site data. • Isolate basic hardware, software, and interface malfunctions utilizing various test equipment on multiple products and technologies. • Specific duties may vary or evolve over time based on business and client needs.
• Implement and maintain critical systems, provide recommendations for changes and enhancements, and perform duties support and maintain implementation of solutions that meet ESS/ITS and Juniper/Velera requirements. • Design, install, configure, monitor, and/or maintain network applications with could include the use of VMware Hypervisor, Microsoft, Red Hat, Cisco Networking and Telephony infrastructure. • Provide technical insight and serve as both guide and implementation specialist in support of current technical dependencies and new, long-range strategic system design and development. • Conduct system/environment needs assessments, including analysis, design, vendor research, and comparisons. • Identify opportunities through new technological advancements and provide recommendations to management. • Utilize electronic software distribution and system management tools to anticipate, identify, and quickly resolve technical issues that affect system availability. • Accurately diagnose and correct common-to-advanced system issues, and work to rapidly restore affected services. • Develop and maintain proper documentation for all system configurations; Follow established change control procedures for all system configuration changes. • Provide advanced technical support and training to operational staff in complex system projects and configurations. • Remain current on Information Technologies advances through professional groups, trade publications, and personal contacts; participate in testing and implementation of new technologies. • Assist as needed in preventative maintenance and necessary repairs on file servers, operating systems, sub-systems, and applications. • Partner with Information Security team to design, implement, and maintain enterprise-wide security functions and practices related to assigned area of responsibility. • Perform other duties as assigned.




