AI Security and Assurance
Last updated: 2026-08-14
AI security and assurance is the practice of proving an AI system behaves safely before and after it reaches production, covering vendor assurance, model and data controls, evaluation, and incident response. AI security differs from traditional application security because the model itself is an untrusted input path.
AI security protects AI systems, and the data and decisions they touch, from misuse, manipulation, and leakage. AI assurance is the evidence that those protections work. In production systems the exposure is rarely inside the model: it sits in what the system is allowed to read and what it is permitted to do.
The AI-specific threat classes
| Threat | What it looks like | Practical defence |
|---|---|---|
| Prompt injection | Retrieved content carries instructions the model follows | Assume it succeeds; constrain tools and scope credentials so it reaches nothing |
| Data leakage | Answers restate material the user is not entitled to see | Carry document permissions into retrieval; filter per user, not per corpus |
| Poisoning | Corrupted training or retrieval data degrades answers quietly | Control write access to the knowledge base; monitor answer quality over time |
| Over-permissioned agents | An agent acts beyond its mandate because its credentials allow it | Least privilege per tool; human confirmation at consequential actions |
| Vendor supply chain | AI arrives inside purchased software with uninspected controls | AI clauses in procurement; written answers on data handling and testing |
What Is AI Security and Assurance?
AI security is the practice of protecting AI systems, and the data and decisions they touch, from misuse, manipulation, and leakage. AI assurance is the evidence that those protections work: not the claim that a system is safe, but the artefacts that let someone else verify it. Most organisations produce assertions and call them assurance.
Classic security governance still applies. Access control, least privilege, logging, incident response, and supplier due diligence are not made obsolete by machine learning. What changes is where the exposure sits and what an attacker can do with a system that reasons over text and takes actions on your behalf.
The Exposure Is in the Data Path and the Agent's Authority
Teams instinctively look for the risk inside the model. In production systems the model is rarely the weakest point. The exposure is in what the system can read and what it is permitted to do.
A retrieval system that indexes documents without carrying their permissions will answer questions using material the person asking should never see. That is not a model failure; it is an access-control failure with a language interface in front of it. An agent granted broad credentials will, sooner or later, take an action nobody intended, because the instruction it followed arrived inside content it retrieved. Separate the model from the authority: the model may propose, but the permissions decide what it can actually touch.
Assurance Means Evidence, Not Assertion
An assurance case answers one question: what would convince a reasonable outsider that this system is safe enough to operate? Four artefacts carry most of the weight. Access control evidence showing that retrieval respects the permissions of the person asking. Adversarial testing results, including the attempts that succeeded. Output monitoring in production, because the tests you ran before launch describe a system that no longer exists once data drifts. And an audit trail that can reconstruct who approved what, on what evidence, and who was authorised to switch it off.
This maps onto the Measure and Manage functions of the NIST AI Risk Management Framework, which supplies the structure while your jurisdiction and sector supply the rules.
Control Effort in Proportion to Consequence
Applying the same scrutiny to every AI system guarantees the important ones receive too little. Tier by what happens when the system is wrong: who is affected, whether they can appeal, and whether the effect can be reversed. In humanitarian and public-sector contexts that question has a sharper edge, because the people affected often cannot appeal, cannot switch providers, and may face real consequences from a mistake.
Most of Your AI Arrives From Vendors
Security programmes that inspect only internally built models cover a small and shrinking share of actual exposure. Put AI questions into procurement: where is our data processed and under whose law, is it used for training, what is logged and for how long, what happens to our data when the contract ends, how are model changes communicated, and what has independent testing found. A vendor unable to answer in writing is not necessarily insecure, but they are unassured, and you will inherit the difference.
Frequently asked questions
What is AI assurance?
AI assurance is the evidence that an AI system is secure and performing as claimed, rather than the assertion that it is: access-control evidence, adversarial testing results including failures, production output monitoring, and an audit trail of approvals and stopping authority.
Can prompt injection be fixed?
Not reliably by filtering, because no filter dependably separates data from instruction in natural language. Treat it as unsolved and design so that a successful injection reaches nothing of value: constrain the tools an agent can call, scope its credentials narrowly, and require human confirmation for consequential actions.
What is the most common AI security mistake?
Indexing documents into a retrieval system without carrying their access permissions, so the system answers using material the person asking is not entitled to see. It presents as an AI problem but it is an access-control failure with a language interface in front of it.
The working version of this material, with a vendor assurance question set, an AI control checklist by consequence tier, and an incident triage sheet, is in the AI Security and Assurance Playbook. The tiering scheme is defined in the AI Governance Playbook. For institutional context see AI governance in the United Nations. This practice draws on CISM and CISSP-grade security governance applied across United Nations data systems.