Apply the Techniques That Make Models Cheaper to Run
Could this model do the same job while consuming far less?
Advanced: 11 controls
Who it applies to
Enter only for systems where you train or adapt the model (B1), and whose letter is A (this area has only Advanced controls). Go as deep as your system letter, and implement in your mode.
Why it exists
Where you build or run your own models, apply the established techniques that shrink what a model costs to operate without meaningfully reducing what it can do. Several exist, and they are worth naming plainly: one removes the parts of a model that contribute little, leaving a smaller model that performs almost as well; another stores the model’s numbers in a coarser, cheaper form that runs faster and lighter; another trains a small model to imitate a large one, capturing most of the capability at a fraction of the running cost. These are not exotic research; they are standard practice that routinely cuts running cost and energy substantially with little loss of accuracy — and they are left on the table only when no one is asked to look.
What to actually do
Use efficiency techniques where they reduce cost without unacceptable loss. For models you run at scale, apply the techniques that make them smaller and cheaper to operate — trimming, coarser representation, or training a compact model to imitate a large one — wherever the accuracy trade is acceptable. Measure the trade rather than assuming it; the point is meaningful saving at little cost, and you confirm that by checking.
Controls
Advanced1 controls
Control 217
Apply model-efficiency techniques to models run at scale, where the accuracy trade is acceptable.
AdvancedPreventivedeveloperper system
What it is
The trade measured, not assumed.
How to implement it · Steps
Identify models run at scale: e.g. those with the highest inference volume or monthly running cost
Set the accepted accuracy loss per model in advance (e.g. at most 1 point on the evaluation set)
Try the techniques: pruning low-contribution parts, quantisation to lower numeric precision, distillation into a smaller model
Measure on the same evaluation set: accuracy, latency, cost per 1,000 requests and energy per request
Re-run the model's standard release tests before swapping the compressed version in
Deploy only within tolerance; record the measured trade either way
How to start
Rank the models you run by monthly inference cost and trial quantisation on the top one, as it usually needs no retraining.
Evidence
Per model, the technique applied and the measured accuracy, cost and energy before and after.
Mode: light / standard / full
light
The developer quantises the one model at scale and notes before-and-after figures.
standard
The model team evaluates the techniques as part of its release checklist.
full
A platform team runs an efficiency pipeline across the estate and reports savings per model.
Evidence
The efficiency techniques applied and where; the measured cost, energy, and accuracy trade for each.