How does the model behave under deliberate attack — and can we trust the model file we loaded?
Baseline: 1Enhanced: 1Advanced: 46 controls
Who it applies to
Enter only for systems where you integrate, configure or train it (B1). Go as deep as your system letter, and implement in your mode.
Why it exists
If you build models, adapt them, or download model files to run yourself, two risks land on you that pure buyers largely escape — and both concern the model as an object, not as a service. First, the model can be deliberately fooled or degraded by inputs crafted for that purpose. Second, and less obvious: the model file itself — the artefact you download and load — can have been tampered with before you ever ran it, exactly as a software installer can. A model file is executable-grade material with reach into your systems; treat it with the suspicion you already apply to software you did not write. This dimension applies in full to organisations that build or adapt models and to high-risk deployments; if you only consume AI services, the first measure below — verifying what you download — is the one that applies to you.
What to actually do
Verify the origin and integrity of every model file before loading it — you already do this for software. Teams that would never run an unsigned installer on a production server will download model files from a public hub and load them unverified. Stop the asymmetry: check the file’s integrity against its published fingerprint, prefer files whose origin is cryptographically attested, and record what you verified. An unverified model file is an unaudited program with access to whatever you connect it to.
Sign the models you build, and refuse to deploy what fails verification. For every model your own pipeline produces, generate a cryptographic seal at build time and verify it at load time, automatically. This is the control that catches a model swapped or corrupted anywhere between your build and your production — an attack that is otherwise invisible, because a tampered model looks and mostly behaves like the original.
Screen third-party model files for hidden trigger behaviours. A model obtained from outside can carry a deliberately implanted behaviour that stays dormant — the model performs normally on every ordinary input — until a specific trigger appears, at which point it does what its implanter intended. Where the stakes of the deployment justify it, screen external model files for such implants before adoption; afterwards is too late, because the trigger is chosen by the attacker, not found by you.
Test robustness against the attack that matters to your use case — not as a generic ritual. Match the adversarial testing to the deployment. A model that classifies or detects gets tested against inputs crafted to slip past it. A model adapted on your own data gets tested for whether that adaptation can be poisoned. A model trained on sensitive data gets tested for whether an attacker can determine what — or who — was in the training set. Scoped testing finds the failure that would actually hurt you; generic testing produces a reassuring report about the wrong risk.
Prefer model formats that cannot run code when opened. Some common model files execute embedded code the moment they are loaded — opening the file is enough to run an attacker's payload, before the model has answered anything. Prefer formats designed to hold data only; and where a code-bearing format cannot be avoided, scan it before it is ever loaded, exactly as you would treat an executable of unknown origin.
Controls
Baseline1 controls
Control 104
Verify origin and integrity of every model file before first use.
BaselinePreventivedeployer or developer (deployer+)per system
What it is
Fingerprint checks and attested origin, recorded — the one measure in this dimension that applies to every organisation.
How to implement it · Steps
Download only from the publisher's official source or the publisher's own page on a model hub
Compute the file's fingerprint (hash) and compare it with the one the publisher publishes
Where offered, check the signature or attestation of origin
Record: model, version, source address, fingerprint, result, date, who checked
Do not load a file that fails or cannot be verified without a recorded decision by the system owner
How to start
For each model file already in use, compute its fingerprint, compare it with the publisher's, and record the result.
Evidence
The verification log for every model file in use.
Mode: light / standard / full
light
A named person runs the operating system's hash command and records the result in a spreadsheet.
standard
Verification is a script in the download process, with results stored in the model register.
full
Model files enter only through an internal repository that verifies fingerprints and origin automatically and rejects failures.
Enhanced1 controls
Control 105
Prefer non-executable model formats, and scan any loadable file that can run code on open.
EnhancedPreventivedeployer or developer (deployer+)per system
What it is
Applied to every model file before its first load, alongside the origin and integrity check
How to implement it · Minimum rules
Default: use model formats designed to hold data only, which cannot run code when loaded
Code-bearing formats allowed only with a recorded justification when no data-only version exists
Scan every code-bearing file with a model-file scanner before its first load
Files with scan findings are not loaded
Record the format decision and scan result next to the origin and integrity check
How to start
List the formats of the model files you hold and replace code-bearing ones with data-only versions wherever the publisher offers them.
Evidence
The format decisions and the scan records for code-bearing files.
Mode: light / standard / full
light
The person who downloads models picks data-only formats and, for any exception, runs a scanner and notes the result in the verification spreadsheet.
standard
Format rules sit in the model-management standard, with scanning built into the download process.
full
The internal model repository accepts only data-only formats by default and scans and quarantines any code-bearing file automatically.
Advanced4 controls
Control 106
Sign self-built models and verify automatically at load.
AdvancedPreventivedeveloperper system
What it is
Deployment refuses a model that fails verification.
How to implement it · Minimum rules
The build pipeline signs each model artefact at build time with a key held in the secret store or a signing service
The signature covers the file and its metadata (version, training record reference)
Every environment verifies the signature automatically at load
Deployment refuses a model with a missing or failed signature, and the event is alerted
Signing keys restricted to the pipeline, rotated, and every use logged
How to start
Add a signing step at the end of the training pipeline for one model and a verification check at its serving start-up, then extend to all models.
Evidence
The signing pipeline configuration and the load-time verification records.
Mode: light / standard / full
light
The developer signs each model with a standard signing tool and the serving script refuses to start if verification fails.
standard
The ML engineering team builds signing into the pipeline and verification into every serving environment.
full
A central signing service with protected keys signs all models, and the platform enforces verification on every load.
Control 107
Screen external model files for implanted trigger behaviours.
AdvancedDetectivedeployer or developer (deployer+)per system
What it is
Applied before adoption, proportionate to the stakes of the deployment.
How to implement it · Steps
Decide whether the stakes justify screening: the model influences significant decisions, handles sensitive data, or can act
Check provenance: publisher identity and track record, documentation of training data and method
Run behavioural screening: varied and unusual inputs and candidate trigger patterns, looking for abnormal outputs
Compare behaviour with a trusted reference model where one exists
Record the result and the adoption decision before deployment
How to start
Take the highest-stakes external model you plan to adopt and complete the provenance check and a first behavioural screen before go-live.
Evidence
The screening reports and the recorded adoption decisions.
Mode: light / standard / full
light
Prefer models from well-documented publishers and commission a specialist screen only for the one high-stakes case.
standard
The security team screens external models for high-stakes deployments as part of model approval.
full
A dedicated model-assurance function screens every external model, scaled to stakes, before it enters the internal repository.
Control 108
Run adversarial-robustness testing scoped to the use case.
AdvancedDetectivedeployer or developer (deployer+)per system
What it is
Evasion, poisoning, or training-data exposure — whichever the deployment actually risks.
How to implement it · Steps
Name the attack that matters: evasion for classifiers and detectors, poisoning for models adapted on your own data, training-data exposure for models trained on sensitive data
Write a test plan for that attack with pass thresholds (e.g. maximum accuracy drop under crafted inputs)
Run the tests with crafted inputs or poisoned samples and measure the effect
Fix through hardening, input filtering or data controls, then re-test
Repeat on every model change or new adaptation
How to start
Write one sentence stating which attack would hurt this deployment most, and build the test plan around it.
Evidence
The scoped robustness test plans and results.
Mode: light / standard / full
light
Commission a specialist test scoped to the single attack the owner has named.
standard
The data-science and security teams run scoped robustness tests before each major release.
full
An AI red team maintains scoped robustness suites per use case and runs them automatically in the release pipeline.
Control 109
Assess training-data exposure where the data is sensitive.
AdvancedDetectivedeveloperper system
What it is
Testing whether membership in, or content of, the training data can be inferred from the model.
How to implement it · Steps
Identify which training data is sensitive: personal data, confidential business data
Membership-inference test: can an attacker tell whether a given record was in the training set
Content-extraction test: can the model reproduce training records verbatim or near-verbatim
Set acceptable leakage thresholds and mitigations: deduplication, removal of records, privacy-preserving training, output filtering
Record results and repeat after every retraining
How to start
Take a sample of sensitive records from the training set and test whether the model reproduces them or reveals that they were used.
Evidence
The exposure test plan, results and the mitigations applied.
Mode: light / standard / full
light
Commission a specialist assessment for the one model trained on sensitive data, and keep the report.
standard
The data-science team runs exposure tests before each release, reviewed by the data-protection lead.
full
Exposure tests run automatically after every training run, with release blocked above the agreed thresholds.
Evidence
Verification logs for downloaded model files; the signing pipeline and its load-time verification records; screening reports; the scoped robustness test plans and results. the model-format decisions and the scan records for code-bearing files.