Zurück zu Jobs
ScovaiScovaiJobs
Tap Growth ai

Tap Growth ai

AI Engineer – Agent Evaluation Specialist

Los Angeles, USVor OrtUnbefristetVollzeit

Veröffentlicht am 16. Nov. 2026

Diese Stelle wird in EN ausgeschrieben

Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & EvaluationWe are looking for an AI Engineer with strong, practical experience inevaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.Key Requirements (Must-Have):
  • LLM-as-a-Judge:Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
  • Golden Dataset Development:Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
  • Langfuse:Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
  • Agent Evaluation:Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.
Critical Screening Criteria:Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development.Agent evaluation is the core requirement and will be the primary selection criterion.

Rollenübersicht

Jobart

Vollzeit

E-Mail

recruiter@tapgrowth.ai

Erforderliche Kompetenzen

LLM-as-a-Judge Evaluation Framework Design and ImplementationEnd-to-End Agent EvaluationAgent Behavior TestingAgent Tool-Usage EvaluationAgent Reasoning-Workflow EvaluationLLM Response Quality Assessment (Accuracy, Relevance, and Reliability)Golden Dataset DevelopmentGolden Dataset Validation and MaintenanceBusiness Analyst and Stakeholder CollaborationLangfuse Agent Instrumentation and TracingAI Performance Monitoring and Evaluation-Score TrackingAgent Failure Troubleshooting

Ähnliche Stellen

McCulloh Consulting

Sr. Claims Manager

McCulloh Consulting

Los Angeles, USVor OrtUnbefristetVollzeit
vor 3 Tagen
Rapsys Technologies Pte Ltd.

AI Engineer – Agent Evaluation Specialist

Rapsys Technologies Pte Ltd.

Los Angeles, USVor OrtUnbefristetVollzeit
vor 6 Tagen
The Staff Pad

Senior Human Resources Manager - Full-Time | Bilingual preferred

The Staff Pad

Los Angeles, USVor OrtUnbefristetVollzeit
vor 7 Tagen
The Staff Pad

Human Resource Business Partner (HRBP) - Full-Time

The Staff Pad

Los Angeles, USVor OrtUnbefristetVollzeit
vor 7 Tagen
The Audio Programmer

International B2B Sales Manager - Music Software

The Audio Programmer

Los Angeles, USVor OrtUnbefristetVollzeit
vor 20 Tagen
Absolute Value

Sales Representative – Talk Sports & Make $200+ Hourly

Absolute Value

Los Angeles, USVor OrtUnbefristetVollzeit95.000 $ – 200.000 $ / Jahr
letzten Monat