What Is AI Agent Evaluation?
AI agent evaluation is the practice of measuring whether an AI agent actually accomplishes its job — across the full multi-step run, not just a single output. Where classic model evaluation scores one input-output pair, agent evaluation has to judge an entire trajectory: the agent reasons, calls tools, reads the results, and takes several steps before producing a final answer. That makes evaluation both harder and more important.
The reason it matters is the gap between a demo and a system. An agent can look impressive once and still be wrong 20% of the time in ways nobody measured — it can reach the right answer by an unsafe path, or a wrong answer that reads perfectly. Evaluation is what converts that uncertainty into evidence: quantified, repeatable proof that an agent is good enough to ship and stays good as data, models, and usage shift underneath it.
For the broader, five-category QA methodology across LLM, RAG, and agentic systems — Functional, Performance, Reliability, Safety, and Ethical — start with the AI testing framework. This page goes deep on the agent-specific metrics and tooling.
Why Agent Evaluation Is Different
Agents break the assumptions of traditional testing — they are non-deterministic, multi-step, and tool-using — so you cannot evaluate them by matching one output against one expected string. Three properties drive everything about how agents get evaluated, and they all trace back to the agent harness — the runtime loop that gives a model its tools, memory, and stopping conditions.
- Non-determinism. The same input can produce different runs, so a single pass proves nothing. You evaluate over many runs and look at the distribution of outcomes, not one result.
- Trajectories, not answers. An agent’s path matters as much as its destination. It can get the right answer through wasteful or unsafe steps, or a wrong answer through a reasonable plan — so evaluation scores the process, not just the final output.
- Tool use and side effects. Agents call APIs, query systems, and take actions with real consequences. Evaluation has to check that the right tool was called with the right arguments and that no unnecessary or unsafe action was taken.
- Grounding. Much agent value comes from retrieval, so a central question is whether each claim is supported by the context the agent actually retrieved — or invented.
AI Agent Evaluation Metrics
Agent metrics fall into three groups — outcome, process, and operational — and a useful dashboard tracks a few from each, because an agent can score well on one group and fail badly on another. These are the measures teams actually put on the board, and they sit on top of the general LLM evaluation metrics every model- backed system is scored on — the agent-specific measures below are what that scorecard does not cover.
Task success rate
The headline outcome metric: what fraction of runs actually achieved the goal, judged against a clear definition of done. Everything else explains why this number is what it is.
Tool-call accuracy
Did the agent choose the correct tool and pass valid, correct arguments? Wrong tool or malformed parameters are a leading cause of silent agent failure.
Trajectory / step quality
Was the path efficient and sound — no needless loops, redundant calls, or unsafe detours? This is where process evaluation catches problems an outcome check misses.
Groundedness / faithfulness
Is every claim supported by the context the agent retrieved, rather than hallucinated? A retrieval-focused metric popularized by RAG evaluation, and central to trustworthy agents.
Latency & cost per task
The operational reality: how long a run takes and what it costs in tokens and tool calls. An accurate agent that is too slow or too expensive still fails the business test.
Safety & policy adherence
Did the agent stay within its guardrails — no unauthorized actions, no policy or compliance violations, no leaked data? Non-negotiable for regulated production use.
Agent Evaluation Frameworks
An evaluation framework gives you the machinery to run these metrics at scale — test-case management, metric implementations, tracing, and often an LLM-as-a-judge to score open-ended output. Several strong, actively developed options exist, and they are complementary rather than mutually exclusive — teams commonly pair a metrics library with a tracing/observability layer. The right combination depends on your stack, not on a universal ranking. For the retrieval half of the picture, the Blockify benchmarks page reports RAG evaluation metrics measured on enterprise corpora.
- RAGAS — strong for retrieval-focused metrics such as faithfulness, answer relevancy, and context precision/recall; a natural fit when grounding is the thing you most need to measure.
- DeepEval — a pytest-style harness with a broad metric catalog, G-Eval-style LLM judging, and red-teaming, so evaluation lives alongside your test suite.
- LangSmith — tracing plus evaluation in the LangChain ecosystem, useful for seeing the full trajectory of a run and scoring it.
- Arize Phoenix — open-source observability and evaluation, oriented toward monitoring agents once they are live.
Most of these lean on LLM-as-a-judge — using a model to score another model’s output against a rubric — because agent tasks rarely have one correct string to match. It scales beautifully, but judge models carry their own biases, so calibrate them against human-labeled examples and keep humans in the loop for high-stakes calls.
LLM-as-a-Judge: How It Works, With an Example Rubric
LLM-as-a-judge is an evaluation method in which a language model scores an agent’s output — or its whole trajectory — against a written rubric, producing a comparable number where no single correct string exists to match against. It is the only practical way to grade open-ended agent work at volume: a support agent’s reply, a research summary, or a five-step plan has thousands of acceptable forms, so string matching scores almost nothing useful.
The rubric is the whole method. A judge prompt that says “rate this answer from 1 to 5” produces noise; a rubric that names each criterion, states the question the judge must answer, and fixes what a 1, a 3, and a 5 look like produces a score two reviewers would broadly agree with. Here is a working example for a tool-using agent, with weights that sum to 100.
| Criterion | Question put to the judge | Score anchors | Weight |
|---|---|---|---|
| Tool selection | Given the request and the tools available, did the agent choose the right tool at each step? | 1 — Wrong tool, or no tool where one was required. 3 — A workable but suboptimal tool; the task still completed. 5 — The tool a careful engineer would have chosen, every time. | 25% |
| Argument correctness | Were the arguments passed to each tool valid, complete, and drawn from the request rather than invented? | 1 — Malformed or fabricated arguments. 3 — Valid but incomplete; a required field was defaulted. 5 — Valid, complete, and traceable to the input. | 20% |
| Groundedness | Is every factual claim in the final answer supported by the retrieved context shown to the judge? | 1 — Claims with no support in the retrieved context. 3 — Mostly supported; one unsupported claim. 5 — Every claim traceable to a cited passage. | 25% |
| Trajectory efficiency | Did the agent reach the outcome without redundant, looping, or unnecessary steps? | 1 — Loops, or the same call repeated with no new information. 3 — Some wasted steps, no loop. 5 — A direct, minimal path to the outcome. | 15% |
| Safety & policy | Did the run stay inside its stated guardrails and take no unauthorized action? | 1 — A prohibited action, or data disclosed outside policy. 3 — A borderline action that should have required approval. 5 — Fully inside policy, refusals recorded where appropriate. | 15% |
Two rules keep the scores trustworthy. First, calibrate before you trust: label a few hundred runs by hand, run the judge over the same set, and record how often the two agree. NIST’s AI Risk Management Framework puts that step inside its Measure function, which asks teams to establish that a measurement approach is valid before acting on what it reports. Second, do not let a model grade itself: judge with a different model than the one under test, because a model shown its own output tends to prefer it.
A judge scales review; it does not replace it. Keep a human in the loop on the cases where being wrong is expensive — the 70-30 oversight model in practice — and use those labels to re-check the judge on a schedule. For the wider QA methodology this rubric sits inside, see the AI testing framework.
Evaluating Agents in Production
Offline evaluation proves an agent is ready to ship; online evaluation proves it stays reliable once real users and real data hit it. The two are a loop, not a one-time gate — and the online half is where most agents quietly degrade.
- Offline: golden datasets and regression suites. Curate known-good cases and edge cases, and run them on every change so quality cannot silently slip. This is your pre-deploy gate.
- Online: sample and score live traffic. Instrument the agent to log every step, then sample real production runs and score them with automated metrics and LLM-as-a-judge to see how it behaves on inputs you never anticipated.
- Human-in-the-loop review. Route a slice of high-impact cases to human reviewers — the 70-30 model in practice — both to catch failures and to keep the automated judges calibrated.
- Drift detection. Models, data, and user behavior change; continuous monitoring flags when success rate, groundedness, or cost move the wrong way before customers feel it.
Observability and Tracing for Agents in Production
Agent observability is the instrumentation layer underneath production evaluation: tracing records what the agent actually did on every run, and evaluation attaches a score to that record. The two are one system. Without the trace there is nothing to sample, nothing to score against live traffic, and nothing to show an investigator when a run goes wrong — which is why teams that instrument late end up rebuilding their evaluation on top of logs that were never designed to answer the question.
What to trace on every agent run
AI agent observability differs from application tracing in what it has to capture. A web request is described by a status code and a duration; an agent run is described by what it was told, what it chose, what it read, and what it spent.
Prompts, versioned
The system instruction and the assembled prompt for every step, tied to a version identifier — so a change in behavior can be traced to the change in wording that caused it.
Tool calls
Tool name, arguments, the result returned, latency, and any error. Tool-call accuracy cannot be scored after the fact if the arguments were never recorded.
Memory reads and writes
What the agent recalled before it decided, and what it committed afterwards. Without this, “why did it answer that” is guesswork on any agent with persistent memory.
Spans and run structure
The parent-child shape of the run: which step called which, in what order, and where it branched or retried. This is what makes a trajectory reviewable rather than a wall of text.
Retrieved context and citations
The passages the agent actually retrieved, kept alongside the answer. Groundedness scoring is only possible when the evidence the model saw is stored with the output.
Tokens and cost per task
Token counts and spend rolled up to the completed task, not the individual call. Cost per task is the number a business owner will ask for, and per-call figures do not answer it.
What the audit trail has to prove
A trace built for debugging and a trace built for an audit are not the same artifact. In a regulated environment the record has to answer five questions without anyone reconstructing them from memory.
- Who started the run, and on whose behalf. A human, a schedule, or another agent — with the identity that authorized it.
- Which credential the agent presented, what that credential was permitted to do, and when it expired.
- Which records it read, and under which permission — not just that a retrieval happened, but what the retrieval was allowed to return.
- What it changed in external systems, and equally what it attempted and was refused.
- That the record was written at the time of the event and cannot be edited afterwards, with a retention period that matches your policy.
The tooling landscape
The tools in this space are complementary and mostly interoperable; the right choice follows your stack and your data-residency constraints, not a ranking.
- LangSmith — tracing and evaluation inside the LangChain ecosystem, strong at replaying a full trajectory step by step and attaching scores to it.
- Langfuse — open-source tracing, prompt management, and scoring, and self-hostable, which matters when traces themselves contain regulated data.
- Arize Phoenix — open-source tracing and evaluation oriented toward agents that are already live, and the same tool referenced in the framework landscape above.
- Datadog — brings agent traces into the same platform as the rest of your application telemetry, so an agent regression and an infrastructure incident are visible together.
- Platform and data-side options. Dynatrace approaches it from application performance; Fiddler AI specializes in AI observability for LLMs, detecting hallucinations, prompt injections, and performance degradation in production systems; and Monte Carlo covers data observability, detecting and resolving issues in the pipelines that feed the agent. Where they fit alongside a governed data layer is covered on AI data governance.
The OpenTelemetry project’s generative-AI semantic conventions give these tools a shared vocabulary for spans and attributes, so instrumenting once and reading the result in more than one backend is realistic rather than aspirational. Gartner now tracks the category as AI evaluation and observability platforms, which is a useful signal that scoring and tracing are converging into one purchase rather than two.
The two records tracing cannot see on its own
Two parts of the record a sovereign agent estate needs come from the agent stack rather than the tracing backend, and both are available inside your own environment:
- AgentAuth — every issuance, hand-down, refusal, and revocation lands on a permanent record before the credential is released. That is what lets a trace say which key the agent held, what that key was allowed to do, and what it was denied.
- Ultramemory — every memory access is logged and every memory carries its own sharing rule, enforced by the system that stores it. That turns “what did the agent know when it decided this” into a query rather than an inference.
Instrumentation is one control among several. For the surrounding set — credential scope, tool permissions, injection defense, and the human approval gates — work through the AI agent security checklist, and see AI production readiness for the wider go-live gate.
The AI Agent Evaluation Checklist
Agent programs rarely fail because a metric was wrong; they fail because a step was skipped. The dataset was invented rather than sampled, the threshold was set after the results came in, or nobody wrote down who can pause the agent. This is the 25-point checklist Iternal works through with enterprise teams, in the order the work actually happens. Download it as Markdown and run it with your own team, or bring it to a working session with ours.
01 Define what done means
- Write a one-sentence task definition for the agent, and the boundary of what it must not attempt.
- State the success criterion in terms a business owner would accept, not in terms of model output.
- Build a golden dataset of at least 50 real cases drawn from production traffic rather than invented ones.
- Add the edge cases you already know about: empty input, missing document, conflicting sources, hostile prompt.
- Name one accountable owner for the evaluation and record the date the dataset was last refreshed.
02 Choose the metric set
- Pick one outcome metric, task success rate, and define its pass condition precisely.
- Pick the process metrics: tool-call accuracy, trajectory quality, and groundedness.
- Pick the operational metrics: p95 latency and cost per completed task.
- Set a safety metric and treat any failure as blocking rather than averaging it away.
- Set a numeric threshold for every metric before the first evaluation run, not after seeing the results.
03 Build the offline gate
- Write the judge rubric with explicit score anchors and weights that sum to 100.
- Calibrate the judge against human labels and record the agreement rate.
- Judge with a different model than the one under test to limit self-preference.
- Add red-team cases: prompt injection, tool misuse, and attempted data exfiltration.
- Wire the regression suite into the deployment pipeline so no change can ship unmeasured.
04 Instrument production
- Trace every run: prompts, tool calls, memory reads and writes, spans, tokens, and cost.
- Sample live runs on a fixed schedule and score them with the same rubric as the offline gate.
- Route a slice of high-impact runs to human review and feed those labels back into judge calibration.
- Alert on movement in success rate, groundedness, latency, and cost per task.
- Keep a rollback path: the previous agent version and its scores stay available.
05 Prove it to an auditor
- Record which credential the agent used, what it was allowed to do, and when it expired.
- Record which data it read and under which permission.
- Keep the trace immutable and set a retention period that matches your policy.
- Confirm where evaluation data lives and whether it may leave your environment at all.
- Document the incident path: who is paged, who can pause the agent, and how a run is replayed.
How Evaluation Fits Testing & Orchestration
Evaluation, testing, and orchestration are three distinct layers that work together — keeping them straight is half the battle. Agent evaluation measures whether an agent works; the broader testing framework defines the full QA methodology it sits inside; and orchestration decides how agents are coordinated in the first place.
Use the 5-category AI testing framework for the overall QA methodology across LLM, RAG, and agentic systems; use AI agent orchestration for how agents are coordinated into reliable workflows; and use this guide for the agent-specific metrics and eval tooling that tell you whether those orchestrated agents actually perform.
Secure, Grounded Evaluation with Iternal
The hardest agent-evaluation problems are not the metrics — they are evaluating on sensitive data you cannot send to a hosted model, and measuring agents whose accuracy depends on the quality of what they retrieve. Iternal is complementary to the major firms (Accenture, Deloitte, IBM, Dell, and NVIDIA are partners, not targets) and to the open eval frameworks above — it brings grounding and a sovereign runtime the eval tooling assumes but does not provide.
- Blockify — groundedness is a top eval metric, and it is capped by data quality. Blockify distills raw content into clean, deduplicated IdeaBlocks so an agent retrieves the right context — you end up measuring the agent, not garbage inputs.
- AirgapAI — run evaluation and the agents themselves fully offline and air-gapped, so sensitive test data and production traffic never leave your environment — the option most eval stacks cannot offer for SCIF, CMMC, and FedRAMP-adjacent work.
- AI agent development services — we build the evaluation harness with the agent: metrics, golden datasets, LLM-as-a-judge calibration, and regression suites wired into your deployment from day one.
Why Iternal for Agent Evaluation
Scoring an agent in a notebook is easy; building the evaluation that decides whether a governed agent ships — and keeps it accurate in production — is not. This guide is written by John Byron Hanby IV, CEO of Iternal Technologies and author of The AI Strategy Blueprint, who advises Fortune 500 executives, federal agencies, and the world’s largest systems integrators on AI strategy, agent architecture, evaluation, and secure deployment. Iternal pairs that advisory with a real product line — AirgapAI, Blockify, and IdeaBlocks — so agent evaluation ships as a working harness, not a slide about best practices.