Eight document traps for an AI retrieval system

A citation can faithfully point to the wrong version

Ask for a meal allowance and receive CAD 40 with a link to a travel policy. The link works. The policy really says CAD 40. The answer can still be wrong if the trip falls after the policy’s effective period or the policy applies to a different audience.

Our small executed experiment isolates document selection before any model writes an answer. It uses exact topic metadata, dates and audience fields, not embeddings or a commercial retrieval service. That makes the logic easy to inspect while limiting what the result proves. The exercise is about selecting an applicable source, not establishing the quality or security of a full retrieval-augmented generation system.

The four-record corpus

All policies and amounts below are invented for this experiment. The original corpus has four records; a conflict case adds a fifth. The date in the question describes the trip, not the upload date. A historical question may legitimately require an older document.

Scroll the table sideways to see every column.

Two selectors and an explicit refusal path

The baseline returns the first record with a matching topic. Because T1 appears first, every meal question receives CAD 40. The checked selector validates the calendar date, filters by the supplied audience and effective period, and answers only when exactly one applicable record remains.

Zero matches and multiple matches both produce REVIEW, for different reasons. A missing date needs clarification. A conflict needs a source decision. Treating both as permission to choose whichever passage ranks first would conceal the uncertainty the test was designed to expose.

The recorded result: one of eight versus eight of eight

We ran the code on September 21, 2026. The baseline matched the expected decision only for the June 30 staff query. It returned the old amount at the July boundary, answered without a date and answered for a year outside the corpus. It also ignored the manager scope, unknown audience, contradictory policy and impossible calendar date.

The checked selector matched all eight expected fixture decisions: CAD 40 for the June boundary, CAD 45 for July, CAD 70 for the manager case, and REVIEW for the remaining five. The record below includes the actual outputs, input data, program and instructions.

Eight of eight does not mean production-ready. The rules were designed for these fixtures. The baseline deliberately omits controls and represents no commercial product. Reordering the corpus also changes its first-match behaviour, which is a useful way to expose an accidental dependence on storage order.

The audience field is not authentication

A caller can edit “staff” to “manager” in our local data. The exercise therefore does not enforce real access control. In an application, identity and permission checks must come from trusted systems outside the prompt and apply before restricted material reaches generation.

The index, result snippets and citation previews also need appropriate access treatment. Revoking access to the original document is insufficient if an old cache still serves its text. Our runner has no cache or permission service, so a passing fixture cannot be offered as evidence that those parts are secure.

Test selection separately from the written answer

First ask whether an authorized user received the applicable passage. Then ask whether the generated statement follows from it. A model could receive T2 and still write CAD 40; that is an answer-support failure rather than the selection failure measured here.

We tested that limitation on September 23 using five further cases whose expected decisions were written before the run. Only the missing-topic case returned the expected REVIEW. The selector returned CAD 45 when the document text said CAD 55, when the amount was stored as text, when document text was absent, and when the document's end date was impossible. Those four failures are preserved in the probe data and outputs below. The original eight-of-eight result describes only its original fixtures.

Compare metadata with the source text and validate both before relying on a selected passage. The additional probe set is synthetic and investigator-designed; it is not a measure of a commercial retrieval system. A complete pipeline needs ingestion and content-consistency checks in addition to this date-and-audience logic.

For your own system, retain the question, authenticated scope, source version, selected passage and unresolved conflict. Show readers a permitted passage they can inspect. A citation earns its usefulness from applicability and support, not from the mere presence of a link.

AI answer comparison exerciseAssess the written answer after checking that retrieval selected the right evidence.