Is the way we build and ship AI secure by construction?
Baseline: 3Enhanced: 6Advanced: 312 controls
Who it applies to
Enter only for systems where you integrate, configure or train it (B1). Go as deep as your system letter, and implement in your mode.
Why it exists
Extend the discipline you already apply to software construction so that it covers what AI construction actually involves — because AI slips into production through doors ordinary software cannot use. AI work begins in exploration: analysts and data specialists experiment in interactive workbooks, on real data, with real credentials close at hand, outside the review and controls that govern your production code — and the successful experiment has a way of becoming the production system without ever passing through a gate. Meanwhile the finished system depends not only on code but on downloaded models and external datasets, dependencies your existing supply-chain checks were never pointed at. Point them there, and close the doors.
What to actually do
Scan the model and data dependencies, not only the code libraries. Your build pipeline already checks the software components a system pulls in for known problems. Extend exactly that check to what AI additionally pulls in: the models it downloads and the datasets it consumes. A model withdrawn by its publisher, or a dataset found to be poisoned or unlawfully sourced, is a supply-chain incident in every sense that a compromised software library is — the bill of materials from the Governance inventory tells you what to scan, and what is affected when the alarm sounds.
Treat the experimental workbooks as the leak surface they are. The interactive workbooks in which AI work is explored routinely accumulate exactly what should never leave controlled ground: pasted credentials, extracts of customer data, unreviewed code. Scan them for embedded secrets as a matter of routine, keep production credentials out of the environments where they run, and — the rule that matters most — require that nothing graduates from workbook to production without passing the same review as any other code. The path from experiment to production must run through the gate, not around it.
Make security sign-off a release gate for high-risk AI. For systems above the risk threshold you defined in Governance, a security review — including adversarial testing where the stakes warrant it — is a condition of release, with the authority to say no. A finding that arrives after launch is an incident; the gate exists so it arrives before.
Generate the bill of materials in the pipeline, as a by-product of building. The record of what went into each model — base model, datasets, components — should be produced automatically by the build itself, not reconstructed by hand afterwards. Records assembled after the fact are incomplete the day they are written and stale the week after; records generated by the pipeline are correct by construction, and they are what turns a future “are we affected?” into a lookup instead of an investigation.
Confirm that the packages the AI suggests are real before you build them in. An AI that writes code will sometimes name a software package that does not exist — and attackers watch for the names it invents, publish malicious packages under exactly those names, and wait for the next developer to install one on the model's advice. Verify that every AI-suggested dependency exists, is the one intended, and comes from a trusted source before it enters a build.
Gate the release on the evaluation. Run the pre-release evaluation as an automated gate in the release process: a model that fails its documented evaluation set does not ship.
Whatever the stakes, check what you built before you rely on it. An integration you configured or a model you trained creates risks that exist only because you built it. Even for a low-stakes use, validate it before production, and for models you train, decide in advance what “good enough” means and keep every change traceable.
Controls
Baseline3 controls
Control 124
Validate the integration before it reaches production.
BaselineDetectivedeployer or developer (deployer+)per system
What it is
A documented check, before go-live and after every change, that the integrated AI behaves as intended on representative inputs, respects its access limits, and fails safely.
How to implement it · Steps
Prepare 20–50 representative inputs with expected results, including typical, edge and deliberately wrong cases
Check outputs are correct and usable for the intended task; mark pass or fail per case
Check access limits: try to make the AI reach data or functions it should not
Record the result with the system owner's sign-off; repeat after any change of model, instructions, configuration or connected data
How to start
Before go-live, write the test cases on a single sheet with the expected result next to each.
Evidence
The dated test sheet with results and owner sign-off for each release.
Mode: light / standard / full
light
The person who set up the integration runs the cases by hand from a spreadsheet and the owner-manager signs.
standard
Test cases live with the integration's code and are run by the delivery team before each release.
full
The cases run automatically in the release pipeline and a quality team reviews and signs the results.
Control 125
Evaluate every model you train or adapt against written acceptance criteria before use.
BaselineDetectivedeveloperper system
What it is
A held-out test set and the criteria the model must meet are fixed before training; the model is used only if it meets them, and the result is recorded.
How to implement it · Steps
Before training, write the acceptance criteria: metric, numeric threshold and minimum performance on key case groups
Hold out a test set that is never used for training or tuning
After training, run the model on the test set and compare against each criterion
Use the model only if every criterion is met; otherwise retrain or reject
Record the result, model version, test-set version and who decided
How to start
For the next model, write three to five criteria with numeric thresholds and get them approved before training starts.
Evidence
The approved criteria and the evaluation record for each model version.
Mode: light / standard / full
light
The person training the model writes the criteria on one page and the owner-manager approves the result.
standard
Criteria and results are stored with each model version and reviewed by someone other than the trainer.
full
An independent evaluation team sets the criteria with the business owner and runs the evaluation.
Control 126
Keep every change to a model you build traceable.
BaselineDetectivedeveloperper system
What it is
Each version records what changed — data, code, parameters — who approved it and why, so that any output can be tied to the version that produced it.
How to implement it · Minimum content
Version identifier for each model put into use
What changed: data version, code version, parameters, base model
Why it changed
Who approved it and when
Evaluation result for that version
Where and when it went live; each output or log entry carries the model version
How to start
Open a change log today with the version currently in use, and record every later change before it is deployed.
Evidence
The change log per model, with approvals and the version tag in output logs.
Mode: light / standard / full
light
The person who builds the model keeps the change log in a spreadsheet and tags outputs with the version manually.
standard
Version control and a model registry record changes, approvals and the version stamped on each output.
full
The machine-learning platform records every change automatically and links each production output to its exact version.
Enhanced6 controls
Control 127
Extend dependency and secret scanning to models, datasets, and experimental workbooks.
EnhancedDetectivedeployer or developer (deployer+)per system
What it is
The supply-chain checks you run on code run on everything AI additionally pulls in.
How to implement it · Minimum rules
Models: origin, integrity (checksum or signature), publisher withdrawal notices, file formats that can run code when loaded
Datasets: source, licence, and any notice of poisoning or withdrawal
Experimental workbooks: embedded passwords, keys, tokens and customer-data extracts, including in saved cell outputs
Scope taken from the bill of materials; rescan when a new warning is published
Action: fail the build or flag, notify the owner, track the fix
How to start
Point your secret scanner at the workbook folders and list the models and datasets each system downloads.
Evidence
Pipeline scan reports covering models and datasets, and the workbook secret-scan results.
Mode: light / standard / full
light
A commissioned service or the code platform's built-in scanning covers workbooks; model sources are checked by hand.
standard
The build pipeline scans code, models and datasets on every build and workbook folders weekly.
full
The application-security team scans all AI assets continuously.
Control 128
Keep production credentials and bulk customer data out of experimentation.
EnhancedPreventivedeployer or developer (deployer+)per system
What it is
Exploration happens on separated data with no production keys; violations are findings.
How to implement it · Minimum rules
Experiment environments are separate from production networks and the production secrets store
No production keys: experiments use dedicated, low-privilege, time-limited credentials
Data: synthetic, anonymised or minimised samples; bulk customer data only by recorded exception with approver and end date
Scan workbooks and experiment storage for production keys and customer-data patterns
Any violation is a finding: logged, credential rotated, data removed, owner informed
How to start
Issue a separate credential set for experimentation and rotate any production key already found in workbooks.
Evidence
The environment-separation configuration, the exception register and the findings log.
Mode: light / standard / full
light
The person responsible for AI keeps experiment keys separate in the provider console and records exceptions in a spreadsheet.
standard
Experiments run in a dedicated environment with its own credentials, and scans run weekly.
full
A managed experimentation platform enforces separation and data masking, with violations raised automatically as security findings.
Control 129
Route everything from experiment to production through code review.
EnhancedPreventivedeployer or developer (deployer+)per system
What it is
No workbook graduates by copy-paste.
How to implement it · Minimum rules
Production code lives only in the version-controlled repository; workbooks are never deployed directly
Every change reaching production is reviewed by someone other than its author
The review covers secrets, data access, dependencies, error handling and tests
The deployment pipeline refuses unreviewed changes and workbook files
Emergency exceptions are recorded and reviewed afterwards within a set period
How to start
Configure the deployment pipeline to reject any change without an approved review, and move any workbook running in production into reviewed code.
Evidence
The review records for code that graduated from experiments, and the pipeline rule that enforces review.
Mode: light / standard / full
light
A second person, internal or commissioned, reviews the code before it goes live and the review is noted in the change record.
standard
Mandatory peer review is enforced in the version-control system for all AI repositories.
full
Branch protection and pipeline controls enforce review centrally, and the security team audits exceptions.
Control 130
Verify that AI-suggested dependencies exist and are trusted before they enter a build.
EnhancedPreventivedeployer or developer (deployer+)per system
What it is
Guarding against installing a malicious package published to match a name the model invented
How to implement it · Minimum rules
Check every package an AI coding assistant suggests before installing it, not after
It exists in an approved registry under exactly the intended name; watch for look-alike spellings
Trust signals: publisher, age, download history, maintenance activity, known vulnerabilities
Install only through an internal mirror or approved registries; new packages need approval
Record each check: package, version, who verified it, date
How to start
Route package installation through an allow-list or internal mirror and tell developers that new AI-suggested packages need a check first.
Evidence
The dependency-verification records and the allow-list or mirror configuration.
Mode: light / standard / full
light
Developers check each new package against the registry page and log it in a shared spreadsheet before installing.
standard
An internal mirror holds approved packages and new ones are requested through a short review.
full
The build system blocks any package not on the approved list, and supply-chain checks run automatically on each request.
Control 131
Set the AI development lifecycle down as a written standard with defined security gates.
EnhancedPreventivedeployer or developer (deployer+)per system
What it is
Naming, for each stage from intake to retirement, the security work required and the point at which release can be refused
How to implement it · Minimum structure
Scope: the AI systems and kinds of work covered (integration, configuration, training)
Stages from intake to retirement: intake, design, data, build, evaluation, release, operation, retirement
Security work per stage: e.g. risk classification, threat review, dependency and secret scans, validation
Gates: where release can be refused, by whom, and the evidence needed to pass
Rules for moving experiments to production and verifying dependencies
Owner, review cadence and exception route
How to start
Trace how your last AI system actually went from idea to production and mark where a security check should have been able to stop it.
Evidence
The approved written standard and the gate records for each system.
Mode: light / standard / full
light
A short standard kept by whoever leads AI use, with one gate before release signed by management.
standard
The standard extends the existing software development policy, with gates built into the release process.
full
A dedicated standard owned by the security architecture team, with gates enforced in the pipeline and audited.
Control 135
Evaluate generative models against a documented evaluation set before release.
EnhancedDetectivedeveloperper system
What it is
Safety, security and task performance measured on a fixed set, with results recorded and compared across versions.
How to implement it · Minimum content
Task cases with expected answers or a scoring guide
Safety cases: harmful requests, toxic output, unsafe advice for your domain
Security cases: hidden-instruction attacks, data leakage, extraction of system instructions
A fixed, versioned set, unchanged between the versions being compared
A threshold per category; results recorded per model version and compared with the previous one
How to start
Build a first set of around 100 cases across task, safety and security, version it, and run it on the current model to set the reference.
Evidence
The evaluation set versions and the results per model version.
Mode: light / standard / full
light
The person training the model keeps the set and results in a spreadsheet and runs it with a simple script.
standard
The set is versioned with the model code and runs before each release.
full
An evaluation team maintains the set, runs it automatically and publishes comparisons across versions.
Advanced3 controls
Control 132
Gate high-risk releases on security sign-off with refusal authority.
AdvancedPreventivedeployer or developer (deployer+)per system
What it is
Adversarial testing included where the stakes warrant.
How to implement it · Steps
Apply the risk threshold set in Governance to decide which releases are gated
Run the security review: threats, guardrails, access, dependencies, validation results
Add adversarial testing where stakes warrant: decisions about people, systems that act, public exposure
Sign-off by a named reviewer independent of the delivery team, with written authority to refuse
Record the decision (approved, approved with conditions, refused) and track the findings
How to start
Name the security reviewer, write their refusal authority into the lifecycle standard and apply it to the next high-risk release.
Evidence
The gate sign-offs and refusals with their review findings.
Mode: light / standard / full
light
A commissioned external reviewer carries out the review and management formally accepts their power to refuse.
standard
The security team holds the gate, with adversarial testing commissioned for the highest-risk releases.
full
A security review board with an in-house adversarial testing team holds the gate, enforced in the release pipeline.
Control 133
Generate the bill of materials automatically in the build pipeline.
AdvancedPreventivedeveloperper system
What it is
Produced by construction, not archaeology.
How to implement it · Minimum content
Base model: name, version, source and checksum
Datasets: identifiers and versions, linked to their lineage records
Code and libraries: versions as resolved by the build
Training configuration: parameters and environment
Evaluation results for the build
Generated by the build step, stored with the model artefact and linked to the AI inventory
How to start
Add a step to the training pipeline that writes the base model, dataset versions and library list to a file stored alongside the model.
Evidence
The pipeline-generated bill of materials for each model version.
Mode: light / standard / full
light
A short script run at the end of each training job writes the record to the model's folder.
standard
The training pipeline generates the bill of materials and stores it in the model registry.
full
The platform generates and signs the bill of materials on every build and feeds it to the inventory and to dependency scanning.
Control 134
Run safety and security regression checks before every deployment.
AdvancedDetectivedeployer or developer (deployer+)per system
What it is
A model that newly fails a previously passing check does not ship.
Run the suite automatically on every change of model, instructions, configuration or guardrails
Compare with the last passing version; any newly failed case blocks the release
Record results per version; any override needs documented approval
How to start
Turn the findings of your last tests and incidents into a first set of regression cases and run it against the current version.
Evidence
The regression-check results for every deployment.
Mode: light / standard / full
light
A commissioned service or a named person runs the suite from a script before each deployment and keeps the results.
standard
The suite runs in the release pipeline and blocks failing deployments.
full
A quality team maintains a large versioned suite run on every change, with trend reporting across versions.
Evidence
Integration validation records; acceptance criteria and evaluation results per model version; the change log per model; Pipeline scan reports covering models and datasets; workbook secret-scan results; review records for graduated code; gate sign-offs and refusals; the pipeline-generated bills of materials; regression-check results. the dependency-verification records and the written development-lifecycle standard. Pre-release evaluation sets and results per version.