AI Security and Assurance
Last updated:
AI security protects AI systems, and the data and decisions they touch, from misuse, manipulation, and leakage. AI assurance is the evidence that those protections work. In production systems the exposure is rarely inside the model: it sits in what the system is allowed to read and what it is permitted to do.
The AI-specific threat classes
| Threat | What it looks like | Practical defence |
|---|---|---|
| Prompt injection | Retrieved content carries instructions the model follows | Assume it succeeds; constrain tools and scope credentials so it reaches nothing |
| Data leakage | Answers restate material the user is not entitled to see | Carry document permissions into retrieval; filter per user, not per corpus |
| Poisoning | Corrupted training or retrieval data degrades answers quietly | Control write access to the knowledge base; monitor answer quality over time |
| Over-permissioned agents | An agent acts beyond its mandate because its credentials allow it | Least privilege per tool; human confirmation at consequential actions |
| Vendor supply chain | AI arrives inside purchased software with uninspected controls | AI clauses in procurement; written answers on data handling and testing |
What Is AI Security and Assurance?
AI security is the practice of protecting AI systems, and the data and decisions they touch, from misuse, manipulation, and leakage. AI assurance is the evidence that those protections work: not the claim that a system is safe, but the artefacts that let someone else verify it. The distinction matters, because most organisations produce assertions and call them assurance.
Classic security governance still applies. Access control, least privilege, logging, incident response, and supplier due diligence are not made obsolete by machine learning. What changes is where the exposure sits and what an attacker can do with a system that reasons over text and takes actions on your behalf.
The Exposure Is in the Data Path and the Agent's Authority
Teams instinctively look for the risk inside the model. In production systems the model is rarely the weakest point. The exposure is in what the system can read and what it is permitted to do.
A retrieval system that indexes documents without carrying their permissions will answer questions using material the person asking should never see. That is not a model failure; it is an access-control failure with a language interface in front of it. An agent granted broad credentials so that it can complete its task will, sooner or later, take an action nobody intended, because the instruction it followed arrived inside content it retrieved. Separate the model from the authority: the model may propose, but the permissions decide what it can actually touch.
The Threat Classes That Are Specific to AI
Prompt injection is the one to design around rather than solve. Any content the system reads, a document, a web page, an email, can contain instructions, and no filter reliably distinguishes data from directive. The durable defence is architectural: assume injection succeeds and ensure that what it reaches is worthless. Constrain tools, scope credentials narrowly, and require confirmation for consequential actions.
Data leakage through outputs is the second: models restate what they were given, so anything in the context window is potentially in the answer. Training and retrieval data poisoning is the third, and the most patient, because a corrupted knowledge base degrades answers quietly for months. Supply chain is the fourth and the largest by volume: most AI in an organisation arrives inside purchased software, which means your exposure is set by vendors whose controls you have not inspected.
Assurance Means Evidence, Not Assertion
An assurance case answers a specific question: what would convince a reasonable outsider that this system is safe enough to operate? Four artefacts carry most of the weight. Access control evidence showing that retrieval respects the permissions of the person asking. Adversarial testing results, including the attempts that succeeded. Output monitoring in production, because the tests you ran before launch describe a system that no longer exists once data drifts. And an audit trail that can reconstruct who approved what, on what evidence, and who was authorised to switch it off.
This maps directly onto the Measure and Manage functions of the NIST AI Risk Management Framework. The framework does not tell you which rules apply in your jurisdiction, but it gives a defensible structure for showing that the rules you adopted are actually being followed.
Control Effort in Proportion to Consequence
Applying the same scrutiny to every AI system guarantees that the important ones receive too little. Tier by what happens when the system is wrong: who is affected, whether they can appeal, and whether the effect can be reversed. A drafting assistant whose output a person reads before use is not in the same category as a system that influences eligibility for a service.
In humanitarian and public-sector contexts the tiering question has a sharper edge, because the people affected often cannot appeal, cannot switch providers, and may face real consequences from a mistake. That is the reason protection decisions belong in the design phase rather than the compliance review, and why the resolution of the data you produce is itself a security decision.
Most of Your AI Arrives From Vendors
Security programmes that inspect only internally built models cover a small and shrinking share of actual exposure. The practical response is to put AI questions into procurement rather than to build a parallel review process: where is our data processed and under whose law, is it used for training, what is logged and for how long, what happens to our data when the contract ends, how are model changes communicated, and what has independent testing found.
A vendor unable to answer those questions in writing is not necessarily insecure, but they are unassured, and you will inherit the difference. Ask for two references from organisations of comparable size and sector; if none exist, you are the pilot and should be priced accordingly.
Frequently asked questions
What is AI assurance?
AI assurance is the evidence that an AI system is secure and performing as claimed, rather than the assertion that it is. In practice it consists of access-control evidence, adversarial testing results including failures, production output monitoring, and an audit trail of approvals and stopping authority.
Can prompt injection be fixed?
Not reliably by filtering, because no filter dependably separates data from instruction in natural language. Treat it as unsolved and design so that a successful injection reaches nothing of value: constrain the tools an agent can call, scope its credentials narrowly, and require human confirmation for consequential actions.
What is the most common AI security mistake?
Indexing documents into a retrieval system without carrying their access permissions. The result is a system that answers questions using material the person asking is not entitled to see. It presents as an AI problem but it is an access-control failure with a language interface in front of it.
How does AI security relate to the NIST AI RMF?
The assurance artefacts map onto the framework's Measure and Manage functions: testing and monitoring provide the measurement, while stopping conditions, incident response, and review cadence provide the management. The framework supplies the structure; your jurisdiction and sector supply the rules.
Where to go next
The working version of this material, with a vendor assurance question set, an AI control checklist by consequence tier, and an incident triage sheet, is in the AI Security and Assurance Playbook. The tiering scheme it references is defined in the AI Governance Playbook.
For the institutional context, see AI governance in the United Nations and the NIST AI RMF playbook. These controls are written to be delivered under the constraints described in the Last-Mile AI Framework. Assurance is what a security programme produces; the independent examination of it is a separate discipline, set out on the IT audit of AI systems hub. This practice draws on CISM and CISSP-grade security governance applied across United Nations data systems.
Get the next essay by email
One practical essay a month on AI governance, agentic AI, and digital delivery in the UN system. No marketing, no forwarding of your address.