Voltar às vagas
ScovaiScovaiJobs
Rapsys Technologies Pte Ltd.

Rapsys Technologies Pte Ltd.

AI Engineer – Agent Evaluation Specialist

Los Angeles, USPresencialPermanenteTempo integral

Publicado 16 de nov. de 2026

Esta vaga foi publicada em EN

Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & EvaluationWe are looking for an AI Engineer with strong, practical experience inevaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.Key Requirements (Must-Have):
  • LLM-as-a-Judge:Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
  • Golden Dataset Development:Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
  • Langfuse:Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
  • Agent Evaluation:Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.
Critical Screening Criteria:Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development.Agent evaluation is the core requirement and will be the primary selection criterion.

Resumo da função

Tipo de vaga

Tempo integral

Email

mitali.s@rapsystechnologies.com

Competências necessárias

LLM-as-a-Judge EvaluationLLM Evaluation Framework Design and ImplementationGolden Dataset Development and ValidationAgent Behavior EvaluationAgent Tool-Usage EvaluationReasoning Workflow EvaluationLangfuse Agent InstrumentationAgent Execution TracingEvaluation Score and Performance MonitoringAgent Failure TroubleshootingStakeholder Collaboration on Evaluation Benchmarks

Vagas similares

McCulloh Consulting

Sr. Claims Manager

McCulloh Consulting

Los Angeles, USPresencialPermanenteTempo integral
anteontem
Tap Growth ai

AI Engineer – Agent Evaluation Specialist

Tap Growth ai

Los Angeles, USPresencialPermanenteTempo integral
há 5 dias
The Staff Pad

Senior Human Resources Manager - Full-Time | Bilingual preferred

The Staff Pad

Los Angeles, USPresencialPermanenteTempo integral
há 6 dias
The Staff Pad

Human Resource Business Partner (HRBP) - Full-Time

The Staff Pad

Los Angeles, USPresencialPermanenteTempo integral
há 6 dias
The Audio Programmer

International B2B Sales Manager - Music Software

The Audio Programmer

Los Angeles, USPresencialPermanenteTempo integral
há 20 dias
Absolute Value

Sales Representative – Talk Sports & Make $200+ Hourly

Absolute Value

Los Angeles, USPresencialPermanenteTempo integralUS$ 95.000 – US$ 200.000 / ano
mês passado