To test an AI agent before it goes live, run it against a set of checked tasks, grade what it actually changed rather than what it said, make every write safe to retry, run it in shadow beside the current process, and require approval for risky actions until the results earn trust. Then keep logging and re-testing after launch.

Picture an agent that handles refund requests for an online shop. In the demo it reads the ticket, checks the order, issues the refund and replies to the customer. Two weeks after launch, finance finds a handful of customers refunded twice. The payment provider timed out, the agent retried, and both calls went through. Nothing in the demo could have shown this, because nothing in the demo ever timed out.

What does testing an AI agent involve that normal software testing doesn't?

Testing an agent means testing decisions as well as code: the same input can lead to different tool calls on different runs, so one passing run proves little. You need many runs per task, graders that check the result, and controls that limit the damage when a run goes wrong.

Three things make agents different:

• They are nondeterministic. The model can pick a different path each time, so a test has to be repeated.

• They act on other systems. A wrong answer in a chatbot is text; a wrong action in an agent is a refund, an email or a changed record.

• They read untrusted input. Tickets, emails and web pages can contain instructions the agent should ignore.

If you need a refresher on where agents end and workflows begin, see What Is an AI Agent, Really?.

How do you build an evaluation set for an AI agent?

An evaluation set is a list of realistic tasks, each with a known correct outcome and a way to grade it automatically. Anthropic's engineering team defines an eval as "a test for an AI system: give an AI an input, then apply grading logic to its output to measure success."

Where the tasks come from

Start from real work, not invented prompts. Pull tickets, requests or cases from the process the agent will take over, including the awkward ones: missing data, angry customers, requests that should be refused. Anthropic's guide on evaluating agents says "20-50 simple tasks drawn from real failures is a great start."

Grade the outcome, not the transcript

A transcript is the full record of what the agent did in one run; the outcome is the state of the systems afterwards. Anthropic's example is a flight-booking agent that says "Your flight has been booked", where "the outcome is whether a reservation exists in the environment's SQL database." Check the database, the CRM record or the payment log, not the agent's final message.

Run each task more than once

Because runs vary, run each task several times. Anthropic describes two measures: pass@k, the chance that at least one of k attempts succeeds, and pass^k, the chance that all k succeed. For an agent that acts for clients, pass^k is the one that matters: it has to work every time, not once in a while.

Keep the set and re-run it after every change to the prompt, the model, the tools or the data. Anthropic calls these regression evals: they ask whether the agent "still handle[s] all the tasks it used to".

How do you stop an agent from taking the same action twice?

Make every write idempotent: give each action a unique key so that a retry returns the first result instead of acting again. Idempotency means doing something twice has the same effect as doing it once.

Duplicates come from ordinary failures: a timeout, a crashed worker, a model that calls the same tool twice in one plan. The fix sits in the tools, not the prompt.

• Use the target system's idempotency support. Stripe's API, for example, accepts an idempotency key on requests that create or update objects, so that "you can safely repeat the request without risk of creating a second object or performing the update twice."

• Derive the key from the business event. Use the ticket ID plus the action, not a random value per attempt, so a second attempt at the same refund carries the same key.

• Check before writing where no key exists. Look up whether the email was sent or the record already updated before acting.

• Test it on purpose. Add tasks to the evaluation set that force a timeout or a retry, and check that exactly one action landed.

What is a shadow run, and how long should it last?

A shadow run is a period where the agent processes real inputs but its actions are recorded, not executed, while people keep doing the work. You then compare what the agent would have done with what people actually did.

A shadow run catches what an evaluation set misses: the cases nobody thought to write down. Run it until it has covered the normal spread of work, including a busy period and the month-end or weekly cycle if the process has one. Then review the disagreements one by one. Each one is either a gap in the agent, a new task for the evaluation set or a sign that the people disagree among themselves too.

Which agent actions need human approval before launch?

Actions that are hard to reverse, cost money or reach a client should wait for a person until the agent has a track record. Read-only lookups can usually run freely from the start.

OpenAI's practical guide to building agents suggests rating each tool "low, medium, or high" risk "based on factors like read-only vs. write access, reversibility, required account permissions, and financial impact", and using those ratings to pause for checks or escalate to a human.

A simple way to apply it:

• Low risk: Example: look up an order. Before launch: runs automatically.

• Medium risk: Example: draft a reply or update an internal field. Before launch: runs, with a person reviewing a sample each day.

• High risk: Example: issue a refund, send an email to a client or delete a record. Before launch: a person approves each one.

Loosen a tier only when the logs show the approvals were rubber stamps. Our post on designing for AI agents covers how to make those approval screens quick to use.

What should you monitor after an AI agent goes live?

Monitor every tool call with its inputs, result and the identity it ran under, and watch for the failure patterns your tests found. The UK's National Cyber Security Centre (NCSC), writing about the joint guidance Careful adoption of agentic AI services, puts the bar plainly: "If you cannot understand, monitor or contain an agent's actions, it is not ready for deployment."

What to track

• Every action taken, with a link back to the input that caused it.

• Approval decisions, and how often people change or reject the agent's proposal.

• Errors, retries and escalations to a human.

• A weekly sample of transcripts, read by someone who knows the process.

The same NCSC post advises organisations to "deploy agentic AI incrementally, starting with tightly bounded pilots using clearly defined tasks", and to "build confidence in the system before you expand the scope". In practice that means one process, one team and a small set of tools first.

How we approach it at Vizio AI

We write the evaluation set before we write the agent. With the people who do the work today, we collect real cases, agree what a correct outcome looks like for each, and decide which actions are low, medium or high risk.

We then build the tools so writes are idempotent and scoped to the agent's own service account, run the agent in shadow beside the team, and review every disagreement with them. Approval stays on high-risk actions until the logs justify removing it, and the evaluation set keeps running after handover, so a prompt or model change that breaks a case fails a test instead of reaching a client.

This is part of our DX Studio work. If the agent connects to your systems through MCP, read Connecting an AI Agent to Your CRM? Secure the MCP Server First as well.

Sources

• Anthropic, Demystifying evals for AI agents (9 January 2026): https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

• OpenAI, A practical guide to building agents: https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf

• NCSC, Thinking carefully before adopting agentic AI (15 May 2026): https://www.ncsc.gov.uk/blogs/thinking-carefully-before-adopting-agentic-ai

• ASD's ACSC, CISA, NSA, Canadian Centre for Cyber Security, NCSC-NZ and NCSC UK, Careful adoption of agentic AI services (1 May 2026): https://www.cyber.gov.au/business-government/secure-design/artificial-intelligence/careful-adoption-of-agentic-ai-services

• Stripe, Idempotent requests: https://docs.stripe.com/api/idempotent_requests