The one failure that outweighs 99 passing AI checks

Decide which failure can stop the release

Imagine a test run with 99 acceptable answers and one answer that reveals a restricted customer record. Calling it “99% accurate” hides the decision a release owner actually faces. The restricted disclosure is not interchangeable with a slightly awkward sentence.

The figures in this article are hypothetical. They illustrate how to structure a release review, not the results of a real product test. Define critical failures, acceptance rules and response owners before running the evaluation. Otherwise, a high average can become an excuse to explain away the one case that contradicts the system’s basic promise.

A release table with three separate conclusions

Keep ordinary quality, critical failures and coverage limits in different fields. They answer different questions. A small clean test set can establish that those examples passed, while leaving many user groups or operating conditions untested.

Scroll the table sideways to see every column.

Keep the denominator visible

“One failure” means little without the number and kind of opportunities. Record failures divided by evaluated cases, but also describe how the cases were chosen. One hundred easy questions copied from a demonstration cannot represent all the work users will bring.

Keep a held-out set that was not used to tune the prompt. Add cases from actual reported problems where lawful and appropriate, removing unnecessary personal details. Repeated runs can reveal variability, but repeated copies of one easy question do not replace broader coverage. Report which change each evaluation is meant to assess: model, prompt, tools, retrieval corpus or policy.

Make adversarial cases correspond to real authority

A red-team case should test a failure path the system could actually take. Put a misleading instruction inside a retrieved document and see whether it changes tool authority. Ask for a record belonging to another user. Change a permission after an answer is cached. Use an invalid date or conflicting source to test whether the system admits uncertainty.

Run these checks in an authorized environment. Document the expected behaviour and the observed result, including partial failures. A refusal in the chat window does not prove that no restricted material appeared in logs or previews. Inspect the relevant boundary rather than judging only the final prose.

exact-action approval for AI agentsSpecify the permission boundary before turning it into a release test.

Connect the failed case to a live signal

A pre-release test is useful only if the failure remains visible after deployment. For the fictional disclosure case, define what evidence triggers escalation, who can disable the affected path and how access is contained. Avoid collecting sensitive prompts indiscriminately in the name of monitoring.

Track unsupported claims, access failures, unresolved outcomes and correction effort separately. Sample across the kinds of requests actually received. A sudden change in request mix can make an old score less relevant even when the model version is unchanged. Monitoring needs both an operating measure and a person who will act on it.

Rollback is a procedure, not a button label

Record which model, prompt, retrieval index, tools and configuration form the known working version. A model rollback cannot undo a permission change in a connected service or recover a message already sent. Identify those separate recovery tasks and test the feasible ones before release.

For the example, the decision is stop, investigate the restricted-record path, repair it and rerun relevant regression and access tests. The next review must still state what remains untested. NIST and OWASP provide risk and security guidance; they do not certify our illustrative thresholds or any system that copies this table.