Tap Growth ai
AI Engineer – Agent Evaluation Specialist
Los Angeles, USOn-sitePermanentFull-time
Posted Nov 16, 2026
Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & EvaluationWe are looking for an AI Engineer with strong, practical experience inevaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.Key Requirements (Must-Have):
- LLM-as-a-Judge:Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
- Golden Dataset Development:Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
- Langfuse:Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
- Agent Evaluation:Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.
Role snapshot
Job type
Full-time
recruiter@tapgrowth.ai
Required skills
LLM-as-a-Judge Evaluation Framework Design and ImplementationEnd-to-End Agent EvaluationAgent Behavior TestingAgent Tool-Usage EvaluationAgent Reasoning-Workflow EvaluationLLM Response Quality Assessment (Accuracy, Relevance, and Reliability)Golden Dataset DevelopmentGolden Dataset Validation and MaintenanceBusiness Analyst and Stakeholder CollaborationLangfuse Agent Instrumentation and TracingAI Performance Monitoring and Evaluation-Score TrackingAgent Failure Troubleshooting
Similar jobs
Sr. Claims Manager
McCulloh Consulting
Los Angeles, USOn-sitePermanentFull-time
2 days ago
AI Engineer – Agent Evaluation Specialist
Rapsys Technologies Pte Ltd.
Los Angeles, USOn-sitePermanentFull-time
5 days ago
Senior Human Resources Manager - Full-Time | Bilingual preferred
The Staff Pad
Los Angeles, USOn-sitePermanentFull-time
6 days ago
Human Resource Business Partner (HRBP) - Full-Time
The Staff Pad
Los Angeles, USOn-sitePermanentFull-time
6 days ago
International B2B Sales Manager - Music Software
The Audio Programmer
Los Angeles, USOn-sitePermanentFull-time
20 days ago
Sales Representative – Talk Sports & Make $200+ Hourly
Absolute Value
Los Angeles, USOn-sitePermanentFull-time$95,000 – $200,000 / year
last month