Does our explanation actually reflect what the system did?
Advanced: 22 controls
Who it applies to
Enter for every AI system whose letter is A (this area has only Advanced controls). Go as deep as your system letter, and implement in your mode.
Why it exists
Verify that the explanations you give are faithful, because a confident, plausible explanation that does not reflect what the system actually did is worse than no explanation at all — it manufactures false assurance and misleads the person relying on it. AI systems can generate explanations of their own reasoning that sound compelling and are simply invented after the fact. Do not present an explanation you have not checked reflects reality.
What to actually do
Validate that explanations match the system’s actual behaviour. Where you provide explanations — to users, to affected people, to auditors — confirm that they correspond to what genuinely drove the output, rather than being a plausible story the system or a tool produced. An explanation you have not validated is a claim you cannot stand behind.
Test explanations for the audience that will use them. Confirm the explanation is not only faithful but usable: that the intended audience — often a non-expert — can actually understand and act on it. Faithful but incomprehensible is a failure of the same kind.
Controls
Advanced2 controls
Control 171
Validate that provided explanations reflect the system’s real behaviour.
AdvancedDetectiveall rolesper system
What it is
No unvalidated explanation is presented as fact.
How to implement it · Steps
Sample the explanations each system provides to users, affected people and auditors
Perturbation check: change or remove the stated main factor and confirm the outcome moves as the explanation implies
For explanations the model writes about itself, test consistency across repeated runs and reworded inputs
Where the model allows it, compare with an independent attribution method
Reject or revise explanations that fail, and record the result
Repeat on every model or prompt change
How to start
Take 20 recent explanations from one system and test whether altering the stated main factor changes the outcome as claimed.
Evidence
The validation method, results per sample, and the explanations rejected or revised.
Mode: light / standard / full
light
Commission the check from a specialist once a year and after model changes, with the system owner keeping the results.
standard
The data science team runs the validation on a fixed sample before each release and records failures.
full
An independent model validation function runs automated faithfulness tests on every release and blocks unvalidated explanation formats.
Control 172
Test explanations for comprehension by their intended audience.
AdvancedDetectiveall rolesper system
What it is
Faithfulness and usability both confirmed.
How to implement it · Steps
Define the audience for each explanation: affected people, staff, auditors
Recruit testers who resemble that audience (e.g. 5–10 non-experts for person-facing explanations)
Ask each tester to say why the decision was made and what they would challenge
Score the share who correctly restate the main factors against a set pass mark (e.g. 80 %)
Rewrite and retest explanations that fall short; record each round
How to start
Show one person-facing explanation to five people outside the team and note whether each can say why the decision was made.
Evidence
The comprehension test records, pass marks, and the revisions made after failures.
Mode: light / standard / full
light
Run the test with a few customers or acquaintances who resemble the audience, with the system owner recording results.
standard
The customer-experience team runs the test before launch and after wording changes.
full
A user-research function tests each explanation format with representative panels and tracks comprehension over time.
Evidence
The validation method and results; audience comprehension testing; records of explanations rejected or revised as unfaithful or unusable.