Have we deliberately imagined how this could be attacked or misused — before it ships?
Enhanced: 33 controls
Who it applies to
Enter for every AI system whose letter is E or A (this area has no Baseline controls). Go as deep as your system letter, and implement in your mode.
Why it exists
Before building or deploying, systematically imagine how the system could be attacked, misused, or fail — rather than waiting to find out in production. This deliberate exercise of thinking through the ways a system could be turned against its purpose is cheap before the system exists and expensive afterwards. Complement it by imagining the scenarios in which the system could cause harm even without an attacker — misuse, edge cases, unintended consequences — and planning for them. The point is to meet the failure on paper first.
What to actually do
Work through how the system could be attacked and abused, before it is built. Systematically consider the ways an attacker could manipulate, poison, extract from, or misuse the system — drawing on the specific attack patterns from Technical Security — and design the defences in from the start. Finding an attack path on a whiteboard costs a meeting; finding it in production costs an incident.
Imagine the harmful scenarios that need no attacker. Plan for the ways the system could cause harm through ordinary misuse, unusual inputs, or unintended consequences — the failures that arrive not from malice but from the world being more varied than the design assumed — and mitigate the ones that matter.
Controls
Enhanced3 controls
Control 224
Conduct threat modelling before building or deploying significant AI.
EnhancedPreventiveall rolesper system
What it is
Attack and abuse paths identified and designed against from the start.
How to implement it · Steps
Draw the system: users, inputs, model, data stores, connected tools, outputs and trust boundaries
Per component, list attack paths: injected instructions in inputs or retrieved documents, data poisoning, extraction of model or data, misuse of tools and permissions
Rate each path on likelihood and impact
Decide the defence to design in for each path that matters, and add it to the design requirements
Record the result in the system's threat register; repeat on significant change
How to start
Run a two-hour whiteboard session for the next significant system with whoever builds or configures it, a security person and the business owner.
Evidence
The threat model and the defences it added to the design requirements.
Mode: light / standard / full
light
The configurer and an external adviser run one session and keep a one-page record.
standard
The system owner runs it with the security lead using a standard template.
full
Security architects run it as a mandatory design activity, tracking findings to closure.
Control 225
Plan for harmful scenarios that require no attacker.
EnhancedPreventiveall rolesper system
What it is
Misuse, edge cases, and unintended consequences anticipated and mitigated.
How to implement it · Steps
Ordinary misuse: users relying on outputs beyond the intended purpose or using the system for unapproved tasks
Edge cases: unusual inputs, other languages, missing data, groups under-represented in the data
Unintended consequences: over-reliance by reviewers, people wrongly excluded, extra load on downstream teams
Rate each scenario on likelihood and severity and select those that matter
Set a mitigation for each: guardrail, user guidance, human check or monitoring signal
Test the selected edge cases before launch
How to start
In the threat-modelling session, add 30 minutes on one question: how could this hurt someone with nobody attacking it?
Evidence
The scenario analysis with the mitigations chosen and the edge-case test results.
Mode: light / standard / full
light
Management and the configurer list scenarios in one session, on the same page as the threats.
standard
The system owner runs it with users and support staff, recorded beside the threat model.
full
A cross-functional review including affected-group representatives, with mitigations tracked.
Control 226
Maintain a per-system AI threat register.
EnhancedPreventiveall rolesper system
What it is
Listing the named threats that apply to the system and naming, for each threat that applies the control of Domain 2 that treats it. One page at light mode.
How to implement it · Minimum content
System name and owner
Named threat: e.g. instruction injection, data poisoning, model or data extraction, excessive agent permissions, sensitive-data leakage
Applies: yes or no, with a one-line reason
Treating control: the Domain 2 control, by identifier, for each threat that applies
Control status: in place, planned, or gap
Last review date; review on any change of model, provider, data or connected tools
How to start
Take the named threats from the risk catalogue, mark which apply to the system and write next to each the Domain 2 control that treats it.
Evidence
The current threat register per system and the record of its last review.
Mode: light / standard / full
light
One page listing the threats that apply and the control treating each.
standard
A shared template per system, kept by the owner and checked by the security lead.
full
A maintained register per system in the security tooling, reviewed on every change.
Evidence
The threat models and the defences they drove; the harmful-scenario analyses and their mitigations. The threat register per system and its review on change.