Zurück zu Jobs
ScovaiScovaiJobs
Rapsys Technologies Pte Ltd.

Rapsys Technologies Pte Ltd.

AI Engineer – Agent Evaluation Specialist

Los Angeles, USVor OrtUnbefristetVollzeit

Veröffentlicht am 16. Nov. 2026

Diese Stelle wird in EN ausgeschrieben

Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & EvaluationWe are looking for an AI Engineer with strong, practical experience inevaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.Key Requirements (Must-Have):
  • LLM-as-a-Judge:Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
  • Golden Dataset Development:Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
  • Langfuse:Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
  • Agent Evaluation:Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.
Critical Screening Criteria:Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development.Agent evaluation is the core requirement and will be the primary selection criterion.

Rollenübersicht

Jobart

Vollzeit

E-Mail

mitali.s@rapsystechnologies.com

Erforderliche Kompetenzen

LLM-as-a-Judge EvaluationLLM Evaluation Framework Design and ImplementationGolden Dataset Development and ValidationAgent Behavior EvaluationAgent Tool-Usage EvaluationReasoning Workflow EvaluationLangfuse Agent InstrumentationAgent Execution TracingEvaluation Score and Performance MonitoringAgent Failure TroubleshootingStakeholder Collaboration on Evaluation Benchmarks

Ähnliche Stellen

McCulloh Consulting

Sr. Claims Manager

McCulloh Consulting

Los Angeles, USVor OrtUnbefristetVollzeit
vor 3 Tagen
Tap Growth ai

AI Engineer – Agent Evaluation Specialist

Tap Growth ai

Los Angeles, USVor OrtUnbefristetVollzeit
vor 6 Tagen
The Staff Pad

Senior Human Resources Manager - Full-Time | Bilingual preferred

The Staff Pad

Los Angeles, USVor OrtUnbefristetVollzeit
vor 7 Tagen
The Staff Pad

Human Resource Business Partner (HRBP) - Full-Time

The Staff Pad

Los Angeles, USVor OrtUnbefristetVollzeit
vor 7 Tagen
The Audio Programmer

International B2B Sales Manager - Music Software

The Audio Programmer

Los Angeles, USVor OrtUnbefristetVollzeit
vor 20 Tagen
Absolute Value

Sales Representative – Talk Sports & Make $200+ Hourly

Absolute Value

Los Angeles, USVor OrtUnbefristetVollzeit95.000 $ – 200.000 $ / Jahr
letzten Monat