Why evidence beats explanations for AI actions

When an AI system does something consequential, the common instinct is to ask it why. Modern models are fluent at answering. They will produce a clear, confident account of their reasoning. The trouble is that this account is generated after the fact, in the same way the original output was, and it is not bound to what the system actually did. A good explanation and a wrong action can coexist comfortably. For anything that matters, you need evidence instead.

The difference in one line

• An explanation is a story about a decision, produced on request.
• Evidence is a record of what happened, produced by the act of doing it.

A model asked to justify a refund it approved can describe a sensible policy it may never have consulted. A record showing which policy version was checked, what inputs were read, who or what authorised the action, and what changed as a result, does not depend on the model's account being sincere. One is persuasion. The other is fact.

Why explanations are not enough

Explanations fail in the exact situations where you most need to trust them:

• They are reconstructions, not traces. The model narrates a plausible path, not necessarily the one taken.
• They are agreeable. Ask why a decision was right and you will usually be told why it was right.
• They degrade quietly. When a system starts behaving differently, its explanations often keep sounding just as reasonable.

None of this makes explanations useless. They help people understand and debug. But they are a poor foundation for accountability, because they cannot be checked against anything outside the model.

What evidence has to be

Evidence is only worth the name if it holds up when someone hostile, or simply careful, examines it. In practice that means a few properties:

• Inspectable. A person can review what happened without special access or trust in the vendor's summary.
• Tamper-evident. If a record is altered after the fact, that is detectable rather than silent.
• Fail-closed. If the system cannot produce a proper record of an action, the sensible default is to not take the action, not to proceed and hope.
• Replayable. The sequence of inputs, decisions and effects can be walked through again, so a disputed outcome can be examined rather than argued about.

These are ordinary expectations in domains that have handled risk for a long time. Aviation does not run on pilots explaining their reasoning; it runs on recorders. Finance does not run on traders recalling their intent; it runs on ledgers.

The design consequence

Treating evidence as primary changes how a system is built. Evidence cannot be bolted on as a logging feature after the model is wired in, because by then the authority, the inputs and the effects have already been blurred together. It has to be produced at the moment of action, as part of taking it.

This is where "AI proposes, humans decide" becomes practical rather than a slogan. A proposal that arrives with its supporting record attached is something a person can actually weigh. A proposal that arrives with a confident paragraph is something a person can only believe or doubt.

What to ask

When you evaluate an AI system that will act on your behalf, the useful question is not "can it explain itself." It is: after it acts, what can you inspect, and would that record survive someone checking it? If the answer is a well-written narrative, you have explanations. If it is a record you can replay and verify, you have something you can stand behind.