Evaluate an AI agent as a stateful system that observes context, chooses tools, changes external state and may hand work to another role. Response quality alone cannot prove that the workflow is safe or useful.
Direct answer
A production decision needs evidence in seven categories:
- business task completion;
- tool and argument correctness;
- state and handoff correctness;
- policy and permission compliance;
- recovery from failure;
- latency and reliability; and
- end-to-end unit cost and human effort.
Anthropic’s engineering guidance notes that agent evaluations are difficult because agents act over multiple turns, call tools, modify state and adapt after intermediate results. It recommends combining evaluation techniques rather than relying on one grader. See Demystifying evals for AI agents.
Build scenarios from real workflow variation
Start with observed or interviewed examples, then remove confidential information. Include:
- straightforward valid requests;
- incomplete and ambiguous requests;
- conflicting source documents;
- denied or revoked permissions;
- unavailable and slow tools;
- duplicate events and retries;
- malicious instructions in retrieved content;
- high-risk actions requiring approval;
- cases that must be declined or escalated.
Every scenario needs an initial state, permitted actions, expected outcome, prohibited outcomes and evidence required for a grader.
Separate outcome grading from trajectory grading
Outcome grading asks whether the requested business result was achieved correctly. Trajectory grading asks whether the workflow used acceptable steps.
For example, an onboarding agent may eventually create the correct ticket but still fail because it queried an unauthorized data source, selected the wrong entitlement tool or bypassed an approval. Grade both result and path.
Use deterministic checks where possible
Do not ask another model to grade facts that software can verify. Deterministic checks can validate:
- tool name and schema;
- required and prohibited calls;
- target record values;
- authorization decision;
- approval presence;
- duplicate side effects;
- timeout and retry limits;
- final workflow state.
Use human or model-based grading for qualities such as reasoning adequacy, explanation, ambiguity handling and evidence use. Calibrate automated graders against human-reviewed examples.
Measure handoffs explicitly
For multi-agent systems, record whether the correct role was selected, whether necessary context was transferred, whether sensitive context was withheld, how many handoffs occurred and whether ownership became ambiguous.
Common failure modes include circular delegation, premature completion, duplicated tool calls, lost state and a specialist acting outside its role.
Evaluate controls, not just intelligence
Test that the workflow cannot convert a model mistake into an unauthorized action. Attempt cross-tenant access, excessive tool arguments, prompt injection, stale approval, role escalation and replay. Verify the policy layer denies the request and produces an audit event.
NIST’s AI Risk Management Framework resources emphasize testing, evaluation, verification and validation as part of managing AI risk. Translate governance expectations into executable scenarios and retained evidence.
Track operating measures
| Measure | Why it matters |
|---|---|
| Task-success rate | Directly connects the agent to the intended outcome |
| Unsafe-action prevention | Shows whether controls stop prohibited behavior |
| Escalation precision | Reveals unnecessary review and missed high-risk cases |
| Tool-call accuracy | Detects wrong functions and malformed arguments |
| End-to-end latency | Includes models, retrieval, tools, approvals and retries |
| Reliability | Measures completion under dependency failures |
| Cost per completed task | Includes model, infrastructure and review cost |
| Human correction time | Prevents automation from hiding manual rework |
Report distributions and failure categories, not only averages. One rare but severe unauthorized action may outweigh a high overall pass rate.
Use release gates and regression sets
Define thresholds before testing. A release gate should identify mandatory scenario groups, minimum success, zero-tolerance failures and the owner who accepts residual risk. Run the suite again when the model, prompt, tool, workflow, policy or knowledge source changes.
Production monitoring should feed new failures back into the evaluation set. Do not tune only to a static benchmark until it stops representing actual work.
Practical lab: produce the evidence pack
The Enterprise AI Agent Integration Lab measures task success, denied actions, latency and cost on a bounded workflow. The full course adds multi-agent handoffs, degraded systems, an automated regression suite and a production-readiness defense.
Frequently asked questions
How many evaluation cases are enough?
There is no universal count. Coverage should reflect workflow variation, business impact and change frequency. Start with the highest-risk and most common paths, then add every material production failure.
Can an LLM grade another agent?
Yes, for selected subjective criteria, but calibrate it against human judgment and combine it with deterministic checks. A model grader should not be the only control for authorization or transaction correctness.
Is a high task-success score enough for release?
No. Review prohibited actions, severe failure cases, reliability, human workload, cost and monitoring readiness. Aggregate success can hide unacceptable risk.
Explore Enterprise AI Agent Engineering or enquire about a private evaluation lab.