Can we detect an attack on the AI — and recover when one lands?
Enhanced: 3Advanced: 25 controls
Who it applies to
Enter for every AI system whose letter is E or A (this area has no Baseline controls). Go as deep as your system letter, and implement in your mode.
Why it exists
Assume some attacks will get through, and build for that day now — because the two capabilities that decide how bad it gets cannot be improvised: seeing the attack, and getting the model out. Everything earlier in this domain reduces your exposure; nothing eliminates it. What separates a contained AI incident from a prolonged one is whether your security team could see the attack in the systems it already watches, and whether you could pull the model back — quickly, cleanly — the moment you knew. Teams discover in the middle of an incident that they can do neither: the AI’s activity was never wired into monitoring, and there is no tested way to roll back to the previous model. Wire it in, and build the way back, before you need either.
What to actually do
Send the AI’s activity to the people who watch for attacks. What your AI systems log — the requests they receive, the checks that trigger, the sequence of actions an agent takes — is security telemetry, and it belongs in the monitoring your security team already operates, alongside the network and system events they watch today. An attempted manipulation, a burst of probing requests, an agent suddenly acting out of pattern: each is a detectable event, but only if the data reaches the place where someone is watching. Telemetry that stays inside the AI platform protects no one.
Write detections for the attacks AI actually receives. The alert rules your security team runs today were written for network intrusions and malware; none of them will fire on a hidden-instruction attack, a systematic attempt to extract a model, or an agent whose action sequence has turned anomalous. Add detection rules for the AI attack patterns your deployments actually expose — the same patterns your vulnerability testing probes for — so that the monitoring sees what the testing anticipated.
Build and rehearse the way back: rollback, and a kill switch. When a model turns out to be compromised, poisoned, or leaking, you need to act in minutes: revert to the previous pinned version (which your version discipline from the Architecture dimension makes possible), pull the model from the approved list so nothing else deploys it, and revoke the credentials of any agent involved. Build these abilities in calm weather and rehearse them, because a rollback attempted for the first time during an incident is a second incident.
Make the fallback hold the business, not just stop the model. A kill switch that removes the AI and leaves the process dead will not be used — whoever holds it will hesitate, and the hesitation is the damage. Pair every kill switch with the continuity fallback you defined in the Governance security program: the manual path, the previous system, the degraded-but-working mode the process reverts to. And test the fallback itself, not only the switch — the switch is easy; it is the landing that fails.
Keep attacking yourself, on a schedule. Commission recurring exercises in which skilled testers attack your deployed AI with permission — the programme mandated in the Governance security dimension. Route what they find into the detections above, the guardrail tuning of this domain, and the risk process: an exercise that changes nothing was a performance. The attacks against AI evolve continuously; a defence reviewed once a year defends last year.
Controls
Enhanced3 controls
Control 136
Feed AI activity into the security monitoring you already operate.
EnhancedDetectiveall rolesper system
What it is
Requests, triggered checks, and agent action sequences arrive where the watchers are.
How to implement it · Minimum content
Requests: time, caller identity, system, model version, size; content only as your data rules allow
Guardrail events: which check triggered and the action taken
Leakage: output-filter triggers for personal data or credentials
Each rule has a severity, an owner and a response step, and is tested by simulating the pattern
How to start
Write three rules first (repeated injection triggers, bursts of blocked requests, denied agent actions) and test each by simulating the pattern.
Evidence
The detection rules, their test results and the alert and response records.
Mode: light / standard / full
light
A commissioned monitoring service runs the rules, or the provider's built-in alerts are switched on and sent to a named person.
standard
The security team writes and maintains the rules in its existing monitoring, informed by vulnerability testing.
full
A detection-engineering team maintains the AI rules and tunes them with exercise results.
Control 138
Maintain an AI incident playbook including rollback and credential revocation.
EnhancedCorrectiveall rolesper system
What it is
Executed by the cross-functional team defined in the Governance security program.
How to implement it · Minimum structure
Triggers and severity: compromise, poisoning, data leak, harmful output, runaway agent
Roles: the cross-functional team from the Governance security programme, with contacts and deputies
Containment: restrict or disable the system and switch to the fallback
Rollback: revert to the pinned previous version and remove the affected model from the approved list
Credential revocation: agent, integration and provider keys, with rotation
Notification, evidence preservation and post-incident review
How to start
Write a two-page playbook for your highest-stakes AI system and walk the team through it in a one-hour tabletop exercise.
Evidence
The approved playbook and the records of exercises and real incidents handled with it.
Mode: light / standard / full
light
A two-page playbook held by the named AI lead; for services used as delivered, rollback means disabling access and contacting the provider.
standard
The playbook is an annex to the existing incident process, run by the incident team with AI owners on call.
full
AI runbooks sit in the security operations centre's library, partly automated and exercised.
Advanced2 controls
Control 139
Build and rehearse the model kill switch with its business-continuity fallback.
AdvancedCorrectiveall rolesper system
What it is
Reversion to the pinned prior version and the tested fallback path, exercised, with timings recorded.
How to implement it · Steps
Define the switch: who may pull it, the single action that triggers it and what it disables
Define the reversion: the pinned prior version and how to redeploy it
Define the fallback: manual process, previous system or degraded mode, and who runs it
Rehearse on a set schedule: pull the switch, revert, run the process on the fallback
Record timings (time to stop, to revert, to fallback working) against targets; fix gaps and repeat
How to start
Schedule a rehearsal in a quiet window and time how long it takes to stop the model and get the process running on the fallback.
Evidence
The rehearsal records with timings, the gaps found and the fixes made.
Mode: light / standard / full
light
The owner-manager and the person running the AI rehearse switching off the service and working manually, timing it in a spreadsheet.
standard
Operations and the business owner rehearse rollback and fallback together, with results reported to management.
full
Kill switch and rollback are automated and rehearsed in continuity exercises against formal recovery targets.
Control 140
Run the recurring adversarial exercise programme against deployed AI.
AdvancedDetectiveall rolesper system
What it is
Findings routed into detections, guardrails, and the risk process.
How to implement it · Steps
Scope each exercise: systems, attack types (hidden instructions, extraction, data leakage, agent misuse), rules of engagement
Use testers with written permission, internal or commissioned
Set the cadence per system by stakes, and add an exercise after major changes
Rate each finding and assign an owner and deadline
Route each finding into a detection rule, guardrail change or risk entry, and track it to closure
How to start
Commission a first exercise on your highest-stakes deployed system, with a written scope and signed permission.
Evidence
The exercise reports and the record of detections, guardrail changes and risk entries they produced.
Mode: light / standard / full
light
A commissioned specialist runs a scoped exercise and the AI lead tracks findings in a spreadsheet.
standard
The security team plans a yearly programme mixing internal and commissioned testing, with findings in the risk register.
full
An in-house adversarial team runs a continuous programme, with findings feeding detection engineering and guardrail tuning directly.
Evidence
The monitoring integration and its detection rules; alert and response records; the incident playbook; rollback and fallback rehearsal records with timings; exercise reports and the changes they produced.