AI agents are moving from answering questions to doing work, such as searching for information, calling tools, changing files, and completing multi-step tasks for users.

The hard part is knowing whether an agent is actually dependable. A final answer can look correct even when the agent used the wrong source, made an unsafe tool call, or succeeded only once by chance.

This guide explains the AI Agent Evaluation Metrics that reveal those problems, how to calculate them, how to grade full agent traces, and how to turn evals into release decisions.

What AI agent evaluation metrics actually measure

An ordinary text model can often be evaluated by looking at its input and final output. An agent has more places to fail because it can plan, select tools, pass arguments, observe results, retry, and change its next action.

That is why agent evaluation should look at the result and the process. LangSmith's official evaluation guidance separates agent evaluation into the final response, a single step, and the full trajectory of tool calls and messages.

A practical evaluation model has four layers:

1. Outcome: Did the requested task succeed?
2. Trajectory: Did the agent take a sensible path?
3. Actions: Did it choose the right tools and send the right inputs?
4. Operations: Was the run reliable, safe, fast enough, and affordable enough?

This layered view matters because the same final result can hide very different behavior. One agent may answer correctly after one approved database lookup, while another may answer correctly after six unnecessary calls and a risky fallback to an untrusted source.

A useful metric should therefore answer a decision question. If a number does not help you decide whether to ship, rollback, investigate, or change the agent, it probably does not deserve a prominent place on the dashboard.

The same idea appears in software quality work. Test quality metrics are useful when they connect behavior to a release decision, while a test intelligence platform becomes useful when it keeps the history needed to explain why a result changed.

The core AI agent evaluation metrics that matter

The exact metric set depends on the job. A research agent, coding agent, support agent, and purchase agent can share a common structure, but their definitions of success and unacceptable behavior will differ.

The table below gives a practical starting set.

Microsoft's current Agent Framework evaluation documentation exposes evaluators for tool-call accuracy, tool selection, tool-input accuracy, tool-output use, quality, and safety, which is a useful signal that tool behavior needs its own measurement layer.

Start with task success, then explain it

Task success rate is normally the top outcome metric because it asks whether the job was done correctly. The problem is that teams often define success too loosely.

For a support agent, "produced a reply" is not success. A better definition might require the correct customer record, correct policy source, an accurate proposed action, no restricted data exposure, and an answer that needs no major correction.

For a coding or testing agent, success can include code that builds, tests that exercise the requested behavior, and evidence that the changed suite still behaves reliably. That is why how to test AI-generated code needs more than a code-generation score.

Measure tools separately from the final answer

Tool selection accuracy checks whether the agent called an appropriate tool. Tool input accuracy checks whether the parameters were correct.

These are separate failure modes. An agent can select the correct CRM function but send the wrong customer ID, or choose the right search tool but apply the wrong date range.

IBM's agent-evaluation guidance lists function-calling failures such as wrong function names, missing parameters, wrong parameter types, invalid values, and hallucinated parameters.

Add consistency before trusting a demo

Agents are nondeterministic, so one successful run is weak evidence. Repeat representative tasks and compare success, tool choices, sources, step counts, and safety behavior.

This is similar to flaky test analysis: the important question is not whether something passed once, but whether the behavior stays reliable under repeated execution and changing conditions.

Keep latency and cost attached to quality

A cheap failed task is not efficient, and a fast unsafe action is not good performance. Operational metrics make sense only when viewed beside successful completion and policy compliance.

Track total latency, tool latency, model latency, token usage, retries, and external API calls. Then calculate cost per successful task rather than celebrating lower spend on runs that fail more often.

Tip: Put outcome quality beside cost and latency in every experiment review. This makes regressions visible when an optimization saves tokens by skipping a useful verification step.

How to calculate and interpret agent metrics

Metrics become useful when their numerator, denominator, sampling rules, and pass criteria are explicit. Keep the formulas simple enough that another engineer can reproduce the score from the raw runs.

Task success rate

Use:

task success rate = successful tasks / attempted tasks × 100

The important work is defining "successful." Prefer objective acceptance criteria where possible, such as a database state, file change, test result, calculated value, or required set of fields.

For open-ended work, use a rubric with named levels such as fail, partial, and pass. Calibrate the rubric against human examples before using an LLM judge at scale.

Completion rate

Use:

completion rate = completed runs / attempted runs × 100

A completed run is not automatically a correct run. Keep completion separate so you can distinguish quality failures from crashes, timeouts, max-step exits, or loops.

Tool selection accuracy

When there is a known acceptable tool set, use:

tool selection accuracy = correct tool selections / evaluated tool selections × 100

Do not require one exact tool sequence when several paths are valid. LangSmith trajectory matching supports strict, unordered, subset, and superset comparisons for this reason.

Tool input accuracy

Use a deterministic validator for anything a schema or business rule can prove. Check required parameters, allowed values, identifiers, date ranges, permissions, and whether inputs are grounded in prior observations.

You can report:

tool input accuracy = valid and correct tool calls / evaluated tool calls × 100

A schema-valid argument can still be semantically wrong, so combine type checks with task-specific validation when the value itself matters.

Repeated-run reliability

For each important eval case, run the same configuration more than once and record the proportion of successful attempts. Report the distribution across cases, not just one global average.

A useful view includes the median case success rate, worst critical-case success rate, and count of cases that changed from stable to intermittent after a release.

This pattern is familiar in Playwright test execution history, where run history matters because a single result cannot reveal a long-term stability pattern.

Safety violation rate

Use:

safety violation rate = runs with a defined violation / evaluated runs × 100

Split this by policy class. A prompt-injection failure, unauthorized write, sensitive-data exposure, and missing approval should not disappear inside one blended safety score.

Latency and cost per successful task

Use percentiles for latency, especially p50 and p95, because averages can hide slow tails.

For economics, use:

cost per successful task = total evaluation cost / successful tasks

This gives a more useful comparison when one version is cheap per attempt but requires more retries to produce a correct result.

How to evaluate tools, steps, and trajectories

An agent trace is the evidence trail of a run. It can contain user messages, model responses, tool names, tool inputs, tool outputs, retries, timing, and the final answer.

That trace lets you evaluate three different targets: final outcome, individual actions, and the trajectory as a whole.

Use deterministic graders when the answer is provable

Deterministic checks should handle exact facts your system already knows how to verify. Examples include JSON schema validity, function name, required approval, database state, test pass/fail, allowed tool list, or exact identifiers.

OpenAI's current grader API includes string-check, text-similarity, score-model, Python, and combined multi-graders. The useful principle is broader than any one platform: use code for objective rules, and reserve model judges for judgment that code cannot express cleanly.

Use LLM-as-a-judge for bounded qualitative decisions

A judge model is useful when several answers can be correct and the evaluator needs a rubric, such as whether a research summary is supported, whether a plan is reasonable, or whether a response satisfies a user's constraints.

The rubric should describe observable criteria. Avoid prompts that simply ask whether an answer is "good," because a vague grader produces a vague score.

Human review still matters. Sample judge passes and failures, compare them with human labels, and revise the rubric when the judge rewards behavior your team would reject.

Evaluate trajectories when the path carries risk

Trajectory evaluation asks whether the sequence of actions was acceptable. This matters when the agent can write data, spend money, contact people, change code, or access systems with different permission levels.

LangSmith supports deterministic trajectory matching and LLM-based trajectory judging. Its deterministic modes can require an exact order, accept calls in any order, allow only a subset, or require at least a reference set of actions.

A strict reference is useful for a workflow such as "check policy, verify identity, then issue refund." It is a poor fit for research tasks where several valid search paths can reach the same evidence.

Give partial credit only when it helps diagnosis

Binary pass/fail is ideal for release gates with a clear invariant. Partial credit is useful during development when you need to know whether an agent completed four of five required steps or chose the right tools but failed the final synthesis.

LangSmith's complex-agent guidance explicitly discusses trajectory evaluation that can award credit for expected steps even when the final outcome is not fully correct (complex agent evaluation).

For agents that generate or modify Playwright tests, the evaluation should not stop when the code looks plausible. AI-native test intelligence, Playwright observability, and a Playwright Trace Viewer provide different forms of evidence once those tests actually run.

The open-source TestDino Playwright Skill can help coding agents follow Playwright-specific patterns, while the resulting tests still need execution evidence and regression checks.

How to build an evaluation workflow for production

Good AI Agent Evaluation Metrics are part of a loop, not a one-time benchmark. The loop begins with clear success criteria and grows as production reveals cases the original dataset missed.

LangSmith separates offline evaluation on curated datasets from online evaluation on live interactions, and recommends feeding useful production traces back into datasets. Microsoft similarly describes agent evaluation as an iterative process from design through deployment and regression detection.

Step 1: define the decision before the metric

Write the release question first. Examples:

• Can the agent complete the top 20 customer workflows with no critical safety violation?
• Does the new model improve task success without increasing cost per successful task by more than the team accepts?
• Does a prompt change reduce tool errors without making p95 latency unacceptable?
• Can the agent modify generated tests without increasing unstable CI behavior?
• Then choose the smallest metric set that can answer those questions.

Step 2: build a representative evaluation set

Start with core happy paths, then add known failures, ambiguous requests, adversarial cases, permission boundaries, missing data, tool errors, and realistic long-context cases.

Do not let synthetic cases become the whole dataset. Production traces, support escalations, bug reports, and human corrections are valuable because they represent failures your users actually encounter.

For test-generation agents, spec-driven testing can provide explicit expected behavior, while Playwright Skill and Claude Code E2E testing illustrate how agent-generated Playwright work can be grounded in repeatable testing patterns.

Step 3: run repetitions and useful slices

Repeat important cases so you can measure reliability instead of one-shot luck. Keep the model, tools, prompt, environment, and grader versions attached to each experiment.

Slice results by task type, tool, customer workflow, risk level, language, input length, and any other dimension that can reveal a hidden failure cluster. A 92% global success rate can still contain a critical workflow that succeeds only half the time.

Step 4: compare versions, not isolated scores

Every meaningful agent change should run against the same core evaluation set. Compare the candidate with a baseline and inspect cases that changed status.

This mirrors the value of Playwright CI reports: a result becomes more actionable when it is tied to a specific run, change, environment, and history instead of appearing as an isolated number.

Step 5: turn production failures into eval cases

When a real user exposes a new failure mode, preserve the trace, remove sensitive data, define the expected behavior, and add a minimized version to the regression set.

This creates a ratchet: once a meaningful failure is understood, a future version should not be able to reintroduce it silently.

The same principle is useful for AI-written tests. Playwright test failure evidence, Playwright flaky test debugging, and a Playwright custom reporter can surface run-level signals that become new evaluation examples for the agent or the generated test workflow.

Common mistakes that make agent scores misleading

A metric can be numerically correct and still lead to a bad decision. Most evaluation problems come from weak definitions, unrepresentative data, unreliable graders, or scores that hide important slices.

Treating completion as success

An agent that reaches a final message has completed execution. It has not necessarily completed the user's goal correctly.

Keep completion rate separate from task success so crashes and wrong answers do not collapse into the same number.

Scoring only the final answer

A correct answer can come from an unacceptable path. If an agent has tool access, measure the tool names, arguments, permissions, sources, required approvals, and trajectory where they matter.

This is especially important for agents that can perform side effects. A final text grader cannot detect every wrong write or skipped approval unless that evidence is included in the evaluation.

Requiring one golden trajectory for open tasks

Some workflows have one required order. Many do not.

Use strict matching only where the order itself is part of correctness. For flexible work, score required actions, prohibited actions, evidence quality, unnecessary steps, and final outcome without punishing a different valid path.

Trusting an LLM judge without calibration

LLM-as-a-judge reduces manual work, but its score is still model output. Validate it against human-labeled examples, keep the rubric versioned, and periodically inspect disagreements.

Use deterministic checks wherever the system can prove the answer. OpenAI's grader API supports both rule-like graders and model-based score graders, which makes it possible to combine these approaches rather than forcing every criterion through a judge model.

Evaluating each case only once

A one-shot pass can hide intermittent behavior. Repeat critical cases and report variance or per-case pass rate, especially after model, prompt, tool, retrieval, or orchestration changes.

The testing analogy is direct: a suite can look healthy while intermittent failures create noisy delivery. Test orchestration platforms and historical test analysis exist partly because execution context and repeat behavior matter.

Averaging away critical failures

A composite score can be useful for ranking experiments, but it should never hide a zero-tolerance condition. A 95/100 average is meaningless if the missing five points represent unauthorized payments or private-data exposure.

Keep hard gates for critical safety and business rules. Use weighted aggregate scores only after those gates pass.

Optimizing tokens before defining value

Token count is an operating metric, not the purpose of the agent. Reduce cost after you can measure task success and required verification behavior.

The better question is usually, "What does a successful task cost, and what quality changed when that cost moved?"

Forgetting that the environment is part of the eval

Agents depend on tools, APIs, permissions, data freshness, network behavior, and external systems. Record environment versions and separate agent mistakes from tool or infrastructure failures.

When an AI agent is used in a testing workflow, this distinction becomes particularly important. A failing generated test might indicate product behavior, test logic, timing, environment trouble, or a bad agent change, and Playwright failure analysis helps frame that diagnosis around evidence rather than the final red status alone.

Conclusion

AI Agent Evaluation Metrics work best as a system of evidence rather than a scoreboard. Start with task success, then explain that result through tool correctness, trajectory quality, repeated-run reliability, safety, latency, cost, and human intervention.

Use deterministic graders for facts your system can prove. Use LLM judges for bounded qualitative criteria, calibrate them with human review, and keep critical policies as hard gates rather than soft averages.

Most importantly, keep evaluation alive after deployment. Production failures should become regression cases, repeated runs should expose unstable behavior, and every major agent change should be compared with a known baseline.

For teams using agents to write or modify Playwright tests, Playwright observability and execution history provide the other half of the picture: the agent can be evaluated on what it produced, and the generated tests can be evaluated on how they behave across real CI runs.

FAQs