AI Evaluations Engineer
Role: AI Evaluations Engineer (Senior) Location: US (remote) Duration: 12 Months +
Key Responsibilities
Design and operate end-to-end evaluation frameworks for agentic AI systems (offline regression, online monitoring, A/B testing) Define quality rubrics, scoring architectures, and evaluation standards across multi-agent workflows Build LLM-as-judge and Agent-as-judge pipelines assessing trajectory, tool-call accuracy, groundedness, and safety Own and deliver evaluation infrastructure end-to-end; mentor teams and uplevel customers on evals best practices Must Have 5-7+ years building production-grade ML or GenAI systems; advanced Python Hands-on experience with agent evaluation: offline/on-demand regression testing and online/continuous live-traffic scoring LLM-as-judge proficiency: rubric design, scoring at session, response, and tool-call granularity Core eval metrics: task completion, tool-call accuracy, trajectory quality, groundedness, safety Agentic framework experience (Strands, LangGraph, CrewAI or similar) AWS Bedrock (Models, Guardrails) + CloudWatch; Lambda, S3, IAM fundamentals Should Have Bedrock AgentCore Evaluations (on-demand SDK runner, online monitoring, A/B testing) Agent-as-a-Judge framework; HITL evaluation pipelines Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction RAG quality evaluation (faithfulness, context precision, recall) CI/CD integration for eval regression; observability tooling (tracing, structured logging) Legacy technology familiarity (Java/Spring, Sybase, DB2, or COBOL) as source context for agentic modernization or integration pattern design Financial services or regulated-industry experience (compliance-aware delivery: trade surveillance, data controls, audit requirements) Large-scale portfolio delivery experience - reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement