RAG or Fine-Tuning: The Governance Questions That Decide It

The choice is usually argued on accuracy and cost. In an institution it is settled earlier, by who can correct a wrong answer, whether personal data can be deleted afterwards, and what an auditor can be shown.

Published 2026-09-04 · By Shahzad Asghar

The choice between retrieval and fine-tuning is usually presented as a technical one: which is more accurate, which costs less, which is faster to serve. In an institution those are the second set of questions. The first set is about who can correct a mistake, who can explain a decision, and what happens to a person's data once it has been used.

Answer the technical questions first and you will frequently arrive at the wrong architecture — one that works in a demonstration and cannot pass a review.

This is the governance version of the comparison. For the mechanics of what weights are and how retrieval differs from training, the plain-language guide covers that ground.

The question that decides most cases

When the system produces a wrong answer, what does it take to fix it, and how long does it take?

With retrieval, a wrong answer usually traces to a wrong or outdated document. Someone edits the source, and the next query is correct. The correction is visible, attributable to a person, and reversible.

With fine-tuning, a wrong answer is distributed across billions of weights. There is no line to edit. Correcting it means assembling new training data, retraining, evaluating, and redeploying — and there is no guarantee the specific error disappears rather than moving. In the meantime the system keeps producing it.

For a customer-facing product, that difference is an inconvenience. For an institution answering questions about eligibility, entitlements, or safety, it is the difference between an error that lasts an afternoon and one that lasts a quarter.

Personal data does not come back out

This is the constraint that most technical comparisons omit, and it is the one with legal weight.

If personal data goes into a retrieval index, it can be deleted. The record comes out, the index is rebuilt, and the system can no longer surface it. Deletion is demonstrable — you can show that the document is gone.

If personal data goes into fine-tuning, it is diffused into the weights. There is no location to delete. Removing a specific person's information from a trained model is an unsolved research problem, not an operational procedure, and the practical remedy is to retrain the model without that data — which means keeping the original training corpus, which is its own exposure.

Any organisation subject to data-protection obligations, and certainly any handling data about people who cannot easily contest a decision, should treat this as close to decisive. Fine-tuning on personal data creates a deletion obligation you may not be able to satisfy. Retrieval does not.

The corollary is practical rather than absolute: fine-tune on patterns, formats and classifications. Retrieve for anything describing an identifiable person.

What an auditor asks

An internal auditor reviewing an AI-assisted process will ask a version of one question: *why did the system say that?*

A retrieval system can answer it. The passage that produced the answer can be shown, dated, and traced to an owner. That is an evidence trail, and it is the same kind of trail the rest of the control environment already runs on.

A fine-tuned model cannot answer it in any way that survives scrutiny. The honest response is that the output reflects patterns learned across the training set, which is true and useless as evidence. You can show evaluation results in aggregate, but not the basis for the specific answer in front of the auditor.

That does not make fine-tuning unauditable — it makes the audit target different. You are no longer auditing answers; you are auditing the process that produced the model: what data went in, who approved it, what was measured before release, and what triggers a retrain. That is a real methodology, and it is the shift I have written about in information technology audit in the age of AI.

The mistake is assuming an answer-level explanation exists when it does not.

Where the data goes, and where the compute happens

Retrieval and fine-tuning place different demands on data residency.

A retrieval system can keep documents inside your own environment. The model reads the passages you supply and returns text. If the model runs locally as well, nothing leaves at all — the approach set out in how to run AI locally without a GPU applies at institutional scale too.

Fine-tuning means moving a training corpus to wherever the training happens. If that is a managed service in another jurisdiction, your entire curated dataset — which by definition is the sensitive material worth training on — has crossed a border and been processed by a third party. The resulting weights may also be governed by the provider's terms rather than yours.

For an organisation with a mandate constraining where data may be processed, that is often the end of the discussion before accuracy is ever considered. It is the same reasoning set out under sovereign AI.

What you are actually buying

Procurement makes the difference concrete.

With retrieval, the substantive asset is your document corpus and the index built from it. Both stay yours. Swapping the model underneath is a configuration change. Exit costs are low, because the thing of value never moved.

With fine-tuning, the asset is a set of weights derived from your data. Ownership depends entirely on the contract. Ask directly: who owns the resulting weights, may the provider use your training data to improve their own models, what happens to the weights if you leave, and can you export them in a usable format.

If those questions have unsatisfactory answers, fine-tuning is a lock-in decision wearing a technical costume. The AI vendor assurance questions template covers this ground in a form you can put in front of a supplier.

When fine-tuning is the right institutional choice

None of the above makes fine-tuning wrong. It makes it a narrower tool than vendors present.

Fine-tuning is the right call when the requirement is behavioural, stable, and free of personal data:

• Enforcing a consistent output format that prompt instructions keep drifting away from
• Classification into a fixed taxonomy that changes rarely — ticket routing, document typing, triage categories
• Extracting structured fields from consistently messy inputs
• Holding a register or tone across a large team so output does not vary by whoever wrote the prompt

In each case you are teaching a repeatable procedure, not a fact about anyone. The training data can be synthetic or anonymised. The failure mode is a wrong format rather than a wrong claim about a person, and the correction cycle being slow matters much less.

A sequence that survives review

Start with retrieval. It is cheaper, reversible, and answers the most common complaint — that the assistant does not know enough about the organisation. Most apparent knowledge problems are retrieval problems.

Instrument before you decide anything else. Log what was retrieved and what was answered. Without that, the fine-tuning conversation is speculation.

Name the specific failure before fine-tuning. "The format drifts on long tickets" is a case. "It should sound more like us" is not yet one. If you cannot state what would count as fixed, you cannot evaluate whether training worked.

Check the data before training on it. If the corpus contains personal data, either remove it, synthesise around it, or do not train. This is the step that gets skipped, and it is the one that cannot be undone afterwards.

Keep the retrieval layer after fine-tuning. They solve different problems and the finished system usually needs both: retrieval for what is true today, fine-tuning for how the answer is shaped.

The decision, restated

The technical comparison matters, and there are cases where fine-tuning wins it decisively. But in a public institution, in humanitarian delivery, or anywhere the people described in the data cannot easily appeal a decision, the governance column decides first — and it usually points the same way.

The wider framing of these questions sits under AI governance in the United Nations, and the delivery constraints behind them under the Last-Mile AI Framework.

Frequently asked questions

Is RAG safer than fine-tuning for personal data?

In general yes, because retrieved data can be deleted and trained-in data cannot. If a person exercises a right to erasure, removing their record from a retrieval index is a straightforward operation. Removing their information from model weights is not a solved problem, and the practical remedy is retraining without that data.

Can you explain why a fine-tuned model gave a particular answer?

Not at the level of a specific answer. The output reflects patterns distributed across billions of weights with no traceable path back to a single source. You can evidence the process — training data, approvals, evaluation results — but not the basis for one response. Retrieval systems can cite the passage they used.

Does fine-tuning create data protection obligations?

It can. Training on personal data means that data has been processed and is now reflected in an artefact you cannot easily edit. Before fine-tuning on any corpus containing information about identifiable people, establish the lawful basis, whether deletion requests can be honoured, and whether the training itself moves data across a jurisdiction.

Who owns the weights of a fine-tuned model?

That depends entirely on the contract, which is why it must be asked before signing. Establish who owns the resulting weights, whether your training data may be used to improve the provider's own models, whether you can export the weights, and what happens to them if you leave.

Should a public-sector organisation ever fine-tune?

Yes, for behavioural requirements that carry no personal data: output formats, fixed classification taxonomies, structured extraction, consistent tone. The caution is about training on records describing people, not about the technique itself.

What should be built first?

Retrieval, in almost every case. It is cheaper, reversible, and resolves the most common complaint. Fine-tune later, against a specific named failure, once you have logs showing what the system actually gets wrong.

← All articles