What can break this AI that would not break ordinary software?
Enhanced: 5Advanced: 16 controls
Who it applies to
Enter only for systems where you integrate, configure or train it (B1), and whose letter is E or A (this area has no Baseline controls). Go as deep as your system letter, and implement in your mode.
Why it exists
Test your AI for the attacks that are actually used against AI — because a conventional security test will pass while the model remains trivially manipulable. The most common real-world attack on deployed AI is disarmingly simple: the attacker hides instructions inside content the AI will read — a document, an email, a web page, a record in a database — and the AI, unable to reliably distinguish the content it is processing from the instructions it should obey, follows the attacker instead of you. This technique, known as prompt injection, appears nowhere in a classical security checklist, and neither do its siblings: coaxing the model past its safety rules, extracting its confidential standing instructions, or abusing the fact that whatever the model outputs is often fed, unexamined, into other systems. Test for these explicitly, or ship untested against the attacks you will actually face.
What to actually do
Test the hidden-instruction attack where it really arrives — inside the content, not the chat box. The dangerous version of this attack is rarely typed by a user into a visible prompt. It arrives embedded in the material your AI reads on someone’s behalf: a clause buried in an uploaded contract, white text in a web page the system summarises, a crafted line in a record it retrieves. Any AI that reads documents, browses, or consults your data stores must be tested against instructions hidden in that content — a test confined to the visible prompt field misses precisely the attack that matters.
Treat the model’s output as untrusted input to whatever consumes it. If what the model produces flows onward — into code that runs, a database query, a web page, another system’s action — then an attacker who can shape the model’s output can attack that destination through it. Test every such path the way you test any untrusted input for injection attacks, because that is exactly what it is: input you did not write, going somewhere that trusts it.
Assume attackers will try to extract the standing instructions — verify there is nothing to gain. Attackers routinely trick deployed models into revealing their confidential instructions and embedded context. Test for it — and, more importantly, verify that success yields nothing: no credentials, no internal addresses, no privileged knowledge (the hygiene mandated in the Architecture dimension). The instructions should be an inconvenience to lose, never a breach.
Re-test when the model changes, not when the calendar says so. A new model version can silently reopen a weakness you had closed and confirmed closed: the manipulation that failed against the old model may succeed against the new one. Tie re-testing to the model-change event — the one your contract now warns you about, and your version pinning makes visible — rather than waiting for an annual test window that certifies a model you may no longer be running.
Test the images and other media the system accepts, not only its text. Where an AI reads images, audio, or mixed documents, an instruction can hide in the non-text part — text laid faintly into an image, a caption crafted to be read as a command — and fire when the model processes the whole. Any system that accepts more than text is tested for instructions hidden in the other modalities, because a test confined to the text field misses them entirely.
Make the tests bite. Wherever you have a build or release process, run these tests as automated adversarial and capability evaluation suites whose failure blocks the release. A test that can only produce a report is a finding; a test that stops a release is a control.
Controls
Enhanced5 controls
Control 98
Test for hidden-instruction attacks, both typed and embedded in content.
EnhancedDetectivedeployer or developer (deployer+)per system
What it is
Every system that reads documents, browses, or consults data stores is tested against instructions planted in that content.
How to implement it · Steps
List every input channel: chat box, uploaded documents, emails, web pages browsed, records retrieved from data stores
Write test cases for direct typed instructions and for instructions planted in each channel (hidden text, comments, metadata)
Set pass criteria: planted instructions ignored, no data disclosed, no unauthorised action
Run before go-live and on significant change; record each case and result
Fix failures, then re-run the failing cases
How to start
Plant a hidden instruction in a test document your system reads (for example, asking it to reveal its instructions or email a summary outside) and check whether it is followed.
Evidence
The test plan, the test cases and reports, and the remediation records.
Mode: light / standard / full
light
Commission an external tester or use a short library of known attack prompts, run by the system owner before go-live.
standard
The security team keeps a test suite per system and runs it at each release.
full
A dedicated AI red team maintains the suites, runs them automatically in the release pipeline, and failures block release.
Control 99
Test every downstream path that consumes model output.
EnhancedDetectivedeployer or developer (deployer+)per system
What it is
Output flowing into code, queries, pages, or actions is tested as untrusted input.
How to implement it · Steps
Map every path where model output goes: code executed, database queries, web pages rendered, API calls, actions in other systems
For each path, run standard injection tests using model output as the attack vector (query, script, command injection)
Check that the consuming system validates, escapes or parameterises the output and never executes it directly
Record findings and fix them in the consuming system, not only in the model's instructions
Re-test whenever a new consumer of model output is added
How to start
Draw the flow from model output to each consuming system and test first the path that executes code or runs queries.
Evidence
The output-path map, the test reports and remediation records.
Mode: light / standard / full
light
Commission a tester to run injection tests on the one or two paths where output is executed or displayed.
standard
Application security includes output paths in its standard testing of each AI release.
full
Output-path tests run automatically in the release pipeline for every consumer, and failures block release.
Control 100
Test for safety-rule evasion and instruction extraction.
EnhancedDetectivedeployer or developer (deployer+)per system
What it is
And verify that extracted instructions yield nothing sensitive.
How to implement it · Steps
Evasion tests: role-play, hypothetical framing, multi-turn pressure, encoded or translated requests aimed at the system's prohibited behaviours
Extraction tests: direct and indirect attempts to make the system reveal its standing instructions and embedded context
Check that anything extracted contains no credentials, internal addresses or privileged knowledge
Set pass criteria and record each case and result
Fix failures and re-test; move any sensitive item found out of the instructions
How to start
Ask the system directly, then indirectly, to repeat its instructions, and check what comes out against your no-secrets rule.
Evidence
The test cases and reports, including the review of extracted content.
Mode: light / standard / full
light
Commission an external tester or run a short set of known evasion and extraction prompts yourself before go-live.
standard
The security team runs evasion and extraction suites per system at each release.
full
A dedicated AI red team maintains and automates the suites, refreshing them as new techniques appear.
Control 101
Trigger a re-test on every model-version change.
EnhancedDetectivedeployer or developer (deployer+)per system
What it is
The change event, not the calendar, drives the test.
How to implement it · Steps
Define the trigger: any change of pinned model version, a provider model-change notice, or a new adaptation
Keep a re-test suite per system: hidden-instruction, evasion, extraction and output-path cases
Run the suite on the new version before it reaches production and compare with the previous results
Block the switch if a previously closed weakness reopens
Link the re-test record to the change record
How to start
Add “re-run the security tests” as a mandatory step in your model-version change procedure.
Evidence
Re-test records tied to each version-change event.
Mode: light / standard / full
light
The system owner re-runs the saved test prompts on the new version before switching, or asks the external tester to do so.
standard
The change process cannot close a version switch without a linked re-test report from the security team.
full
A version change automatically triggers the full suite in the pipeline, and the switch is blocked on any regression.
Control 102
Test multimodal inputs for hidden instructions wherever the system accepts them.
EnhancedDetectivedeployer or developer (deployer+)per system
What it is
Images, audio, and mixed documents are probed for planted instructions alongside text
How to implement it · Steps
Identify every non-text input the system accepts: images, scanned documents, audio, video, mixed files
Build test cases: faint or tiny text in images, instructions in captions, metadata or alt text, spoken commands in audio, embedded objects in documents
Check whether planted instructions change behaviour, disclose data or trigger actions
Record results alongside the text tests; fix and re-test
How to start
Create an image containing a faint written instruction and submit it to the system as an ordinary user would.
Evidence
The multimodal test cases and reports, with remediation records.
Mode: light / standard / full
light
Include a handful of image and document cases in the externally commissioned test.
standard
The security team adds multimodal cases to each system's suite and runs them at each release.
full
The AI red team maintains a multimodal attack library and runs it automatically with the other suites.
Advanced1 controls
Control 103
Test for model and training-data extraction on high-value systems.
AdvancedDetectivedeployer or developer (deployer+)per system
What it is
Where the model itself, or what it learned from, would be worth stealing.
How to implement it · Steps
Decide whether the system is high-value: proprietary or costly-to-build model, or sensitive training or reference data
Model-extraction test: systematic querying to copy the model's behaviour; check whether rate limits and output detail make copying feasible
Data-extraction test: prompts that try to reproduce training or reference records verbatim, and checks of whether known records were used
Set acceptable thresholds and mitigations: rate limiting, less detailed outputs, monitoring of query patterns
Report results and repeat after model changes
How to start
List which systems are high-value and, for the first, try to make it reproduce known records from its training or reference data.
Evidence
The high-value designation, the test plan and the extraction test reports.
Mode: light / standard / full
light
Commission a specialist test for the one high-value system, scoped by its owner.
standard
The security team runs extraction tests on high-value systems at each major release.
full
The AI red team runs extraction testing continuously on high-value systems and monitors production for extraction-like query patterns.
Evidence
The AI-specific test plans and reports; the hidden-instruction and evasion test cases; remediation records; the re-test records tied to version-change events.