Prompt injection and output attacks

Threat 8: prompt injection

Instructions hidden in input override the behaviour the system was designed to have. In direct injection the user types them. In indirect injection they arrive inside content the assistant reads: a web page, an email, a PDF, a calendar entry, a retrieved document. The asset hit is the service and its input; the objective is integrity; the threat concerns generative systems. The OWASP Top 10 for LLM applications ranks prompt injection first.

With agents and tools the consequences escalate. An assistant that can only answer questions is limited to saying the wrong thing. An agent that can send email, query a database or approve a request can be made to act. Policy bypass, data exfiltration through the assistant, unauthorised actions through connected tools and reputational harm from manipulated answers are the consequences the standard lists.

The standard’s message is that trusting external data sources is the risk. No filter reliably distinguishes data from directive, so the durable defence is architectural: assume injection succeeds and make sure what it reaches is worthless.

• Detect: monitor inputs and outputs for instruction-like content

• Detect: canary tokens in system prompts that reveal when a prompt has been exfiltrated

• Detect: log prompt template versions with every request

• Detect: red-team with indirect injection scenarios, not only typed attacks

• Mitigate: treat all external content as untrusted data, never as instructions

• Mitigate: separate the instruction context from the data context

• Mitigate: least-privilege tool permissions; human approval for high-risk actions

• Mitigate: output filtering and allow-lists on what the system may do

Threat 9: output containing injection attacks

This is the reverse direction. Model output carries a payload that a downstream system executes: script embedded in a web page, a database command, an operating system command. The model becomes a delivery channel into other applications, which may have nothing to do with AI. The asset hit is the output; the objective is the integrity of the consuming systems.

The consequence is compromise of applications that trusted the model’s output as if it were safe. Detection validates output against expected formats, security-tests every integration that consumes output, and monitors downstream systems for anomalous commands.

The fix is classic secure development: treat model output as untrusted input in every consuming system. Encode and sanitise output, use parameterised queries, sandbox any code execution, restrict what output can trigger, and apply the organization’s existing secure coding rules to AI integrations.

Denial of service and supply-chain compromise

Two further Annex B threats complete the picture of attacks on the service. Denial of service on inference uses flooding or resource-heavy inputs to make the model unavailable or expensive to run; rate limits, quotas, input size limits and capacity planning answer it. Supply-chain compromise introduces malicious or vulnerable models, data sets, libraries, notebook platforms and MLOps tools; supplier assessment, an AI bill of materials, signed artefacts and patching answer it.

Both are conventional attacks with AI-specific targets, which is why they sit in the annex. Their controls exist in ISO/IEC 27002 already; the work is extending supplier management and capacity management to cover the AI tool chain.

Controls for agents that act

Because injection reaches furthest when a system can act, the standard’s guidance converges with the practice of agentic AI governance: separate what an agent may propose from what it may do, scope tool authority narrowly, log the actions rather than only the outcomes, and place a human approval point before any action with financial, access or safety consequence.

Lesson 8 quiz

Five questions. Score 4 or more to mark the lesson complete. Each answer shows an explanation, so a wrong answer teaches as much as a right one.