Fairness and Non-Discrimination in Person-Affecting Uses
Could this system treat people differently because of who they are — and would we know?
Baseline: 7Enhanced: 2Advanced: 110 controls
Who it applies to
Enter only for systems where it decides about people (B2). Go as deep as your system letter, and implement in your mode.
Why it exists
Stop treating fairness as a value in the policy and an item on a committee’s agenda. The most frequently reported class of AI harm in small organisations requires no attacker and no technical sophistication: an assistant is given a handful of past outcomes as examples and asked to rank new people, and it reproduces whatever those outcomes had in common — the last four hires were men, so the women are discarded. Every governance control in this framework can be in place and this still happens, because the human in the loop is approving a ranking that already encodes the pattern, and nothing has asked for the pattern to be looked for. This dimension exists so that the person operating the tool, with no structure behind them, looks for it. It is activated wherever a use scores, ranks, selects or decides about people, and its Baseline is deliberately achievable in hours.
What to actually do
Write the criteria down before you use the AI, and put them in the prompt. State what a good candidate, a valid claim, an eligible applicant is, in words, and instruct the system to apply those criteria. Never ask it to infer the criteria from examples of who was chosen before: examples of past outcomes are a model of past decisions, including their mistakes.
Do not feed it past decisions about people as examples — or check them first. Where examples are unavoidable, check that they are balanced across the protected groups you can observe before they are used. A few-shot prompt built from an unbalanced history is an unbalanced model.
Take out who the person is before the AI sees them. Remove names, gender markers, age, photographs, addresses and other obvious proxies from the input — with the data-loss-prevention boundary of Domain 3, not by asking staff to remember. The AI cannot discriminate on what it does not receive.
Run the swap test before the first real use. The same input with only the protected attribute changed must produce the same output. A difference is a failure that blocks use until the criteria are fixed. Repeat the test whenever the model or the prompt changes.
Review the rejected, not only the recommended. Automation bias concentrates on what the system did not surface. The reviewer looks at the bottom of the list as well as the top.
Count outcomes by group, and treat a gap as a finding. Each period, compare selection or approval rates across the groups you can observe. In the absence of a more specific threshold, use the four-fifths rule (a rate below 80 % of the most favoured group’s rate) as the trigger for investigation. A spreadsheet is enough.
Tell the people affected. That AI was used, what criteria it applied, and how to have the decision reviewed by a person.
Measure fairness properly once you can. For systems whose outputs and groups can be observed, compute a fairness metric — demographic parity, equalised odds or calibration, chosen for the use case — on a representative evaluation set before deployment and on outcomes in operation; report performance per group in the factsheet; record the choice of metric as a trade-off decision under Domain 9, because the metrics cannot all be satisfied at once. For LLM-based systems, where feature attribution is not meaningful, the swap test and group-wise outcome rates are the counterpart.
Look at combined groups as well. A system can treat women and older applicants fairly as separate groups and still disadvantage older women. Where the number of people allows, compute rates and metrics for combinations of groups as well as for each group alone.
Have it audited independently where the stakes or the law require. An annual independent audit of each high-risk system that decides about people, comparing outcomes across groups, with a summary published where required.
Controls
Baseline7 controls
Control 173
Write the decision criteria before use and state them explicitly in the prompt or configuration.
Selection or impact ratios compared across groups by an auditor independent of the system; summary published where required.
How to implement it · Steps
Scope: each high-risk system that decides about people
Auditor: external, or internal audit independent of the team that builds or runs the system
Inputs: outcomes per group for the period, the written criteria, the configuration and prior findings
Analysis: selection or impact ratios compared across groups, including combined groups where numbers allow
Report: findings with remediation owners; summary published where required
Repeat every year
How to start
List the high-risk person-affecting systems and engage an independent auditor for the first, giving them a year of outcome data by group.
Evidence
The audit reports, remediation tracking and any published summary.
Mode: light / standard / full
light
An annual commissioned audit of the one system, or the provider's audit where the tool is unmodified and the audit covers your use.
standard
An external auditor is engaged annually per high-risk system and the system owner tracks remediation.
full
Internal audit runs the programme across all high-risk systems, using external auditors where independence or expertise requires.
Evidence
The written criteria and the prompt or configuration; the example-balance checks; the DLP policy covering protected attributes; swap-test records; the oversight procedure naming review of rejects; the periodic rate checks; the notices to affected people; metric reports and mitigation records; audit reports.
Light mode
How a small organisation meets it.
Swap test plus a quarterly rate check on a spreadsheet, kept as evidence; an annual commissioned audit of the one system, or the provider’s audit where the tool is unmodified and the audit covers your use.