Rapsys Technologies Pte Ltd.
AI Engineer – Agent Evaluation Specialist
Los Angeles, USPresencialPermanenteTempo integral
Publicado 16 de nov. de 2026
Esta vaga foi publicada em EN
Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & EvaluationWe are looking for an AI Engineer with strong, practical experience inevaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.Key Requirements (Must-Have):
- LLM-as-a-Judge:Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
- Golden Dataset Development:Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
- Langfuse:Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
- Agent Evaluation:Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.
Resumo da função
Tipo de vaga
Tempo integral
mitali.s@rapsystechnologies.com
Competências necessárias
LLM-as-a-Judge EvaluationLLM Evaluation Framework Design and ImplementationGolden Dataset Development and ValidationAgent Behavior EvaluationAgent Tool-Usage EvaluationReasoning Workflow EvaluationLangfuse Agent InstrumentationAgent Execution TracingEvaluation Score and Performance MonitoringAgent Failure TroubleshootingStakeholder Collaboration on Evaluation Benchmarks
Vagas similares
Sr. Claims Manager
McCulloh Consulting
Los Angeles, USPresencialPermanenteTempo integral
anteontem
AI Engineer – Agent Evaluation Specialist
Tap Growth ai
Los Angeles, USPresencialPermanenteTempo integral
há 5 dias
Senior Human Resources Manager - Full-Time | Bilingual preferred
The Staff Pad
Los Angeles, USPresencialPermanenteTempo integral
há 6 dias
Human Resource Business Partner (HRBP) - Full-Time
The Staff Pad
Los Angeles, USPresencialPermanenteTempo integral
há 6 dias
International B2B Sales Manager - Music Software
The Audio Programmer
Los Angeles, USPresencialPermanenteTempo integral
há 20 dias
Sales Representative – Talk Sports & Make $200+ Hourly
Absolute Value
Los Angeles, USPresencialPermanenteTempo integralUS$ 95.000 – US$ 200.000 / ano
mês passado