Agent testing · 4 min read
Test the workflow. Not just the answer.
A practical perspective on evaluating tool use, recovery, and human handoffs.
Evaluate the path to the result
An agent can reach a plausible answer through an inappropriate action. Define which tools and data it may use, which decisions need approval, and which actions must never occur. Evaluate both the outcome and the sequence of actions that produced it.
Design for interruption and uncertainty
Include missing information, unavailable tools, contradictory inputs, and interrupted sessions. Look for sensible recovery: asking a targeted question, preserving relevant context, and stopping when an action requires authorization. More activity is not necessarily better performance.
Keep the environment controlled
Use test accounts and synthetic or approved data. Isolate experiments from production systems and observe tool calls and side effects. Combine repeatable scenarios with expert review, and distinguish measured results from assumptions about performance in the real world.