LLM Security in Public Institutions
Last updated: 2026-08-15
Large language model security differs from ordinary application security in one structural way: the model itself is an untrusted input path. Text arriving from a user, a document, or a retrieved web page can change what the system does, so the boundary that matters is not only the network perimeter but every place text enters the model.
How to manage LLM security risk in a public institution: the staff data-leakage problem, the model as an untrusted input path, and when on-premises deployment is justified.
LLM Security in Public Institutions
Large language model security differs from ordinary application security in one structural way. In a conventional system, data is data and code is code. In a system built on a language model, text arriving from a user, an uploaded document, or a retrieved web page can change what the system does. The model treats instruction and content identically because it cannot reliably tell them apart. That single property produces most of the risk.
The problem that arrives first: staff, not attackers
Long before anyone attacks an institution's AI system, its own staff will paste sensitive material into a commercial chat interface. This is not misconduct. It is a rational response to a tool that saves an hour on a task, offered to people under delivery pressure with no approved alternative.
Prohibition fails predictably. A ban with no sanctioned tool moves the activity from a channel the institution can see to personal devices and personal accounts, which is strictly worse: the same data leaves, and now there is no record of it.
The measures that work are dull.
Provide an approved tool. The single most effective control is a sanctioned interface with an enterprise agreement that excludes the institution's data from training and defines retention. Adoption of the approved route is what shrinks the unapproved route.
Say what must never be entered, concretely. "Do not paste sensitive data" is unusable guidance because everyone believes their case is the exception. Name the categories: personal data about beneficiaries or staff, credentials, unpublished procurement material, legal advice, security assessments. A list of five concrete prohibitions outperforms a paragraph of principle.
Log at the gateway, not on the endpoint. Routing approved use through an institutional gateway gives an audit trail and lets policy be enforced in one place, rather than depending on every user remembering a rule.
The model as an untrusted input path
Once a system does more than answer questions, the input problem becomes an execution problem. Prompt injection is the general case: text that reaches the model carries instructions the model then follows. The text can come from a user, but it can equally come from a document the system was asked to summarise, an email in a mailbox it can read, or a page it retrieved.
The defences are architectural rather than linguistic. You cannot instruct your way out of this: an instruction telling a model to ignore malicious instructions is itself just more text in the same channel.
- Separate retrieval from action. A system that reads untrusted content should not also hold credentials to act on the institution's behalf without a human between the two.
- Constrain tools at the tool, not in the prompt. If an agent should only read from one directory, enforce that in the tool's permissions. A prompt saying "only read from this directory" is a request, not a control.
- Treat model output as untrusted input to whatever consumes it. Output rendered into a page, passed to a shell, or inserted into a query needs the same escaping and validation as any user-supplied string.
- Keep irreversible actions behind a checkpoint. Sending, deleting, publishing, paying: these belong to a human confirmation, and that requirement is set out further in the agentic AI governance work.
When on-premises or offline deployment is justified
Running a model inside your own boundary is often proposed as the answer to all of the above. It resolves one question decisively and leaves the others untouched.
What it resolves: where inference happens, and under whose law. If prompts contain data that must not leave a jurisdiction, or that would be catastrophic to expose, local inference is the control that settles it. For institutions handling refugee or protection data, this is sometimes the deciding factor, and it is a legitimate one.
What it does not resolve: prompt injection, over-broad tool permissions, weak output handling, and the absence of a stopping condition all behave identically on a self-hosted model. A local deployment with poor architecture is less safe than a well-architected commercial one.
What it costs: hardware, the engineers who keep it running, an evaluation capability you now own entirely, and a capability gap against frontier models that widens between refresh cycles. That trade is examined in full on the sovereign AI page.
The workable middle path in most public institutions is a split by sensitivity: commercial models under contract for non-sensitive work, local or private deployment for the workloads where data residency genuinely decides the question, and clear rules for which is which.
Where to start
Order matters more than completeness. Provide a sanctioned tool, name the five things that must never be entered, route approved use through a gateway that logs, and only then work through the architectural controls for systems that take actions. An institution that begins with an architecture review and no approved tool will find, six months later, that the interesting risk was never in the architecture.
The vendor questions to ask before any of this is procured are in the AI vendor assurance question set, the controls proportional to consequence are in the AI control checklist, and what to do when something goes wrong is in the incident triage sheet.
Written by Shahzad Asghar, Head of Data and Digital Solutions at UNESCWA. See all articles, the playbooks, and the templates.