AI Security and Assurance: A Practical Playbook

Last updated: 2026-08-14

Your organisation is deploying AI systems faster than your security function can review them. You need to know where to spend limited assurance capacity.

AI Security and Assurance: A Practical Playbook

The decision this page helps you make

Your organisation is deploying AI systems faster than your security function can review them. You need to know where to spend limited assurance capacity.

The answer is rarely the model. Most real incidents involve data reaching somewhere it should not, or a system taking an action it should not have been permitted to take. Adversarial machine learning is a genuine field and a rare cause of loss. Access control and egress remain the main event.

Rules of thumb

1. Threat model the data path before the model. Draw where data enters, where it is stored, where it is sent, who can read the logs, and what leaves the boundary. Most of your findings will come from this diagram. The prompts and completions are usually the least protected data in the system, and they routinely contain the most sensitive content the organisation holds.

2. Separate the model from the authority. The security question is not what the model can say. It is what the surrounding system can do. Give the model no credentials of its own. Route every consequential action through a service that applies its own authorisation, independently of what the model requested. A compromised model should be able to produce bad text and nothing else.

3. Treat prompt injection as unsolved. Filters and system prompt hardening reduce the rate. They do not close the class. Design so that a successful injection reaches nothing worth reaching. If your architecture only holds because injection fails, your architecture does not hold.

4. Assume any content the model reads is hostile input. Retrieved documents, web pages, emails, uploaded files, and tool outputs are all untrusted. The moment a system reads content and then acts, it has an injection path. Mark provenance and never let retrieved content escalate the permissions of the session.

5. Give agents narrow, expiring, and observable permissions. Scope credentials to the task. Time bound them. Log every call. Require confirmation for anything irreversible, anything that moves money, and anything that sends to an external recipient. The default agent permission set is the one that survives being turned against you.

6. Ask vendors two questions that reveal more than a certification report. First, provide the current sub-processor list, including the model providers behind the product. Second, state the notification policy for model changes, including deprecations and silent version updates. A vendor who cannot answer either does not control their own supply chain, and you have inherited that.

7. Contract for change notification, not just for security posture. Your assurance evidence expires when the underlying model changes. Put change notice periods, deprecation windows, and the right to re-test into the contract. A control assessed against a model that no longer serves your traffic is a historical document.

8. Red team against business abuse, not academic attacks. Run the scenarios that would actually hurt: extracting another customer's record, getting the system to authorise an exception, causing a defamatory or discriminatory output that reaches a member of the public, driving cost through unbounded usage. Publish the scenarios internally so builders design against them.

9. Cap consumption as a security control. Rate limits, token ceilings, and spend alerts are availability and cost controls, and they are also the fastest detection you have. An abuse pattern usually appears as a consumption anomaly days before anyone reports a functional problem.

10. Log prompts and completions, then protect the log as a crown jewel. Without this record you cannot investigate anything. With it, you have created a high value store of sensitive content that frequently sits outside the classification scheme. Apply retention limits, restrict access, and include the log in your data protection assessment.

11. Write the AI incident playbook before you need it. Define who declares, how you take a model out of service without taking the business down, what the fallback process is, and who notifies. Test the fallback. A rollback path that has never been exercised is a plan, not a control.

12. Include the failure mode in which the system is confidently wrong. Traditional security assumes availability, integrity, and confidentiality. AI adds a fourth concern: a system that runs perfectly and produces harmful output that nobody questions because it is fluent. Assurance has to cover output quality under live conditions, not only whether the system is up.

The artefact: vendor assurance question set

Send this before contract, not after. Score each answer as answered, partial, or refused.

Supply chain 1. List all sub-processors, including underlying model providers and inference hosts. 2. Where is inference physically performed, by jurisdiction? 3. Which sub-processors can access customer content, and under what controls?

Data handling 4. Is customer content used for training, fine tuning, or evaluation? State the default and how to opt out. 5. What is the retention period for prompts, completions, and logs? 6. Is content isolated per tenant at rest and in transit? Describe the mechanism. 7. Which of your staff can read customer content, under what approval, and is that access logged and reviewable by us?

Change management 8. What is your notice period for a change to the underlying model? 9. What is your deprecation policy and minimum support window for a pinned version? 10. Do we have the contractual right to re-test after a model change?

Security controls 11. Provide the results of your most recent independent security assessment and the remediation status of open findings. 12. Describe your controls against prompt injection, and state which residual risks you consider open. 13. What permissions does the product require in our environment, and what is the minimum viable set?

Incident and exit 14. What is your incident notification commitment, in hours, and what triggers it? 15. On termination, how is our content deleted, on what timeline, and how is deletion evidenced?

The artefact: AI control checklist

Apply proportionally to consequence tier.

The artefact: incident triage sheet

When something goes wrong, answer these in order.

1. Is data leaving the boundary right now? If yes, cut the egress before anything else.
2. Can the system take actions? If yes, revoke its credentials before investigating.
3. What is the fallback process, and is it running?
4. How many outputs are affected, and can you enumerate the recipients?
5. Is the cause input, model, integration, or permission? Investigate in that order, because that is the order of frequency.
6. Does this trigger a notification obligation? Start the clock from detection, not from confirmation.

Failure signals

• The security review happens after the pilot is already handling live data.
• Nobody can produce the list of AI systems with access to production data.
• The agent runs with a service account that predates the project.
• Prompt logs exist but sit outside the data classification scheme.
• Assurance evidence references a model version no longer in service.
• The red team exercise tested jailbreak prompts and did not test whether one user could reach another user's data.
• The vendor answered the security questionnaire and could not name their sub-processors.

What this does not cover

This page addresses the security and assurance of AI systems in operation. It does not cover securing the training infrastructure for organisations building foundation models, model theft and extraction defence at research scale, or the statutory breach notification thresholds that apply in your jurisdiction. Confirm the last of these with counsel and encode the timelines into the triage sheet above.

Download this playbook as Markdown · All playbooks