AI demo vs pilot vs acceptance test: what a buyer learns

A supplier list answers only the first question

Public Services and Procurement Canada maintains an AI source list as a way for federal buyers to acquire AI goods and services. Being qualified for a procurement route is not proof that every product, configuration or use case from that supplier meets a buyer’s needs. The official description concerns access to a procurement process; it does not report the results of your acceptance test. Treat any broader conclusion as something that needs its own evidence.

This guide starts at that gap. A fictional organization wants an assistant that answers staff questions from internal operating manuals. The demonstration looks convincing. Before the buyer commits, what records would let someone else check the proposed system? Our five-row evidence pack is an editorial exercise for that purchase, not an official federal form or a claim that all Canadian buyers follow the same rules.

Demo, pilot or acceptance test: choose the question first

These three activities answer different questions. The comparison below is our planning framework for the fictional manual assistant, not terminology prescribed by CanadaBuys or a universal contract sequence. Decide what uncertainty you need to resolve before choosing the activity. A small buyer may combine stages, but should still distinguish exploration from a decision to accept delivery.

A demo is useful when you need to understand an interaction. A pilot becomes useful when the uncertainty is whether the workflow works for your staff and documents. Acceptance testing is useful when you need a repeatable yes-or-no decision against agreed requirements. Ask the contracting authority how those activities fit the applicable procurement process; this comparison does not authorize a purchase.

• If the demo succeeds but the pilot leaks a restricted document, keep that failure visible; do not average it away with successful demonstrations.
• If a pilot reveals a new requirement, record the change and follow the applicable process. Do not silently change evaluation rules after seeing competing offers.
• If acceptance passes, record the tested version and the changes that would trigger another check.

Scroll the table sideways to see every column.

Write the task so a supplier can fail it

The sample assistant has one job: answer a staff question using the current manual, or say it cannot find enough evidence. It must not decide employee benefits, browse private personnel files or send messages. This small scope makes several failures observable. An answer from last year’s manual fails even if it sounds sensible. A correct answer obtained from a file the employee cannot open also fails.

The buyer writes those limits before watching a second demonstration. Otherwise the demonstration can quietly define what counts as success: whichever examples the supplier has prepared become the requirements. CanadaBuys guidance on preparing an evaluation asks buyers to check the information required by the solicitation and recommends a requirements checklist. Our practical extension is to attach a piece of inspectable evidence to each requirement.

The five records to request

Each row below asks for something more specific than a reassuring sentence. A diagram can describe an intended design; an execution record can show what happened in a defined test. A contractual commitment has a different purpose again. Keep those evidence types separate rather than awarding all three the same “verified” mark.

Scroll the table sideways to see every column.

A purchase rehearsal with no invented results

Create a tiny pack of invented manuals before involving real internal records. In this exercise, version A says requests go to the service desk, while version B replaces it and names an online form. A third document is restricted to a different test account. Include a question no document answers. Ask the supplier to show the inputs, output and source selection for each case.

We have not run this procurement trial against a vendor. These are proposed acceptance cases, so the result column starts blank. That distinction matters: a template full of green ticks would imply work nobody performed. Save the actual outputs unchanged when a test is run, record the service configuration and have the buyer assess the failure conditions agreed in advance.

A supplier may explain that a requested control requires a different product tier or extra implementation. Record the dependency and its price. Do not count a future feature as a passed test. Likewise, a successful demonstration by an administrator does not establish what a normal staff account can access.

eight-case document retrieval experimentInspect executable selection tests before designing your own supplier trial.

Keep the scoring rule ahead of the evidence

A weighted total can help compare convenience, cost and support. It should not allow enough pleasant features to cancel a disclosure of restricted information. In this fictional purchase, permission leakage is a blocking condition. Response speed is a scored preference. The exact distinction is a buyer decision that belongs in the requirements before offers are assessed, with procurement advice where applicable.

CanadaBuys distinguishes procurement instruments, including supply arrangements and contracts. Choose the applicable route with the contracting authority; do not assume an exercise in this article authorizes a purchase. Federal procedures also do not automatically govern a private business or a provincial institution. The transferable part is the evidence discipline, not a universal legal checklist.

The decision record should outlive the sales meeting

At the end of the fictional exercise, the buyer should be able to hand a colleague one short record: the permitted task, tested configuration, cases and outputs, unresolved exceptions, contract dependencies and the person accepting the residual risk. Put promises in a separate column from observed results. If a record is missing, name it rather than replacing it with a general assurance about responsible AI.

After purchase, repeat the relevant cases when the model, permission setup or document-processing behaviour changes. The acceptance decision applied to a configuration and a use, not to the supplier’s brand forever. A small evidence pack is useful precisely because another person can rerun it and disagree with the conclusion without having attended the original demonstration.