How LLMs Actually Work: Weights, Training and RAG in Plain Language

What a weight actually is, what training does to it, and why RAG is a completely different tool. Includes a decision matrix for when to retrieve, when to fine-tune, and when to do both.

Published 2026-09-04 · By Shahzad Asghar

If you have asked ChatGPT, Claude, or a local model a question and wondered what is actually happening inside it, this is written for you. No computer science background required.

By the end you will understand what a large language model is, what people mean by "weights", the difference between training a model and giving it RAG, when to use each, what an open weight model is, and which method fits which real problem.

What is an LLM, in plain language?

A large language model is a program that has read an enormous amount of text — books, articles, code, conversations — and learned the patterns of language well enough to predict, one word at a time, what should come next.

That is genuinely the core trick. Ask it "How do I fix VPN error 809?" and it is not looking up an answer in a filing cabinet. It is generating, word by word, the response its training has taught it is most likely to be correct.

The analogy: a very well-read assistant with no filing cabinet. Imagine someone who has read millions of manuals, forums and support tickets, but instead of keeping searchable notes, has internalised the patterns — what kind of answer usually follows what kind of question. Ask them something and they compose a fresh answer on the spot. They are not turning pages. They know how to talk about it because of everything they have absorbed.

That compression of everything-read-into-instinct is what a language model is.

What does "weights" mean?

This is the most misunderstood term in the field, so it is worth slowing down.

A weight is just a number. It is one of billions of numbers stored inside the model that determine how it responds. There is no code saying "if the question is about VPNs, say this." There is only a large collection of numbers, and the answer emerges from arithmetic performed on them.

The recipe card

Think of a recipe: 350g flour, 4 eggs, 12g salt, 180°C for 35 minutes. Every number on that card is a weight — a value that shapes the outcome. A language model is the same idea with billions of numbers instead of six. When you see a model described as "8B", that means the recipe card carries eight billion numbers.

The piano

Picture a piano where every string has been tuned to an exact tension. Playing is just pressing keys and letting the pre-tuned strings do their work. Training is the tuning. Once it is finished the tensions are locked in, and those locked-in values are the weights. Nobody retunes the piano each time somebody plays.

The combination lock

Imagine a lock with a billion tiny dials that only opens when every dial sits in the right position. Training is the slow process of nudging each dial toward its correct spot using millions of examples as feedback. The weights file is a snapshot of where every dial ended up.

The point: weights are not a database, a rule set, or a lookup table. They are the model's entire knowledge and behaviour encoded as numbers. Nothing else is inside — no notes, no comments, no logic. Numbers, and the arithmetic that runs on them.

How a question actually travels through the model

1. Your words become numbers. Each word or word-fragment is converted into a list of numbers, rather like a barcode. That conversion table is itself made of weights.
2. The numbers pass through layers of attention. Picture a long line of editors. Each one looks at every other word in your sentence and decides how much attention to pay to it in order to understand the word in front of them. How much attention is controlled entirely by weights, and each editor holds a private sheet of numbers.
3. A next-word scorecard comes out. After every layer, the model produces a probability for every possible next word. "VPN error 809 is usually caused by…" might score 87% for "blocked", 4% for "a", and so on down a very long list.
4. It picks a word, appends it, and repeats the whole pass to produce the next one. That is why longer answers take longer: the model runs the full computation again for every word.

Nothing is cached and nothing is retrieved from a drawer. Every response is recomputed from scratch by running your text through this chain of weighted arithmetic. If you want to see that machinery laid out visually, the LLM architecture explainer walks through it step by step, and the free Learn LLMs course builds the same picture from the ground up.

What training actually does

Training nudges every one of those billions of numbers, in tiny steps, until the model gets better at predicting the next word. It sees an example, guesses, is told how wrong it was, and every weight shifts slightly to reduce that error. Repeat across a vast quantity of text and the numbers converge into something that produces fluent, useful answers.

When training stops, the numbers freeze. That frozen snapshot is the model. There is no separate brain and instruction manual to ship — the weights file is the whole thing.

This matters because it explains the key limitation: training changes what the model *is*, permanently, until you train it again.

What RAG means

RAG — retrieval-augmented generation — is a far simpler idea, and it solves a different problem. Rather than baking new facts into the weights, it gives the model a lookup step before it answers.

In practice: your documents go into a searchable store, a user asks a question, the system finds the relevant passages, and the model reads those passages and writes its answer from them.

A concrete example. The user asks how to fix VPN error 809. Retrieval searches old tickets and finds a note that error 809 is usually caused by blocked UDP ports 500 and 4500. The model then answers that the firewall is likely blocking IPsec traffic, and to check UDP 500 and 4500, confirm NAT-T is enabled, and restart the client.

The model never memorised that fact. It looked it up, the way you would check your own notes before answering a colleague.

Why the distinction matters: retrieved knowledge can be corrected, updated or deleted the same day it changes. Trained-in knowledge cannot — fixing it means retraining. That single difference settles most decisions between the two.

RAG or fine-tuning: the simple rule

RAG is for knowledge. Training is for behaviour.

Use RAG when the assistant needs to look something up: a fact, a policy, a past case, anything that might be different tomorrow.

Use fine-tuning when the assistant needs to learn a way of doing something: a tone, a format, a classification, a decision rule that repeats identically every time.

Where RAG fits

Anywhere information changes or has to be exactly right — support tickets, internal knowledge bases, HR policies, troubleshooting guides, product manuals, legal and pricing documents, clinical protocols, procurement rules, field manuals, contracts, standard operating procedures.

The shape of the question is the giveaway. "According to our policy, what should I do?" "What was the fix for this error before?" "Summarise this customer's previous issues." "Find similar incidents from last year." All retrieval.

Where fine-tuning fits

When you want to change how the model behaves rather than what it knows: a house tone of voice, a fixed answer format, a repeated workflow, ticket classification, routing decisions, pulling structured fields out of messy text.

Classification. Given "User cannot access email after password reset", the model should return a consistent object:

``json { "category": "Email Access", "priority": "Medium", "team": "IT Support" } ``

That is a pattern fine-tuning can teach, so the same logic applies every time without being re-explained.

House style. Given "Customer is angry because service is down", it should reply in your organisation's register: acknowledging the urgency, stating that the team is investigating, and committing to an update within a set time.

A fixed skeleton. If every answer should carry the same structure — issue, cause, steps, escalation, final response — fine-tuning holds that shape reliably, where a one-off instruction in a prompt tends to drift.

Decision matrix

What most teams actually need

For a real support system the answer is usually both, doing different jobs. Retrieval supplies the knowledge, putting the right ticket history or policy page in front of the model at the moment of the question. Fine-tuning supplies the manner, so the answer comes back sounding like your team in your format.

Build retrieval first. It is cheaper, fully reversible, and most problems that present as "the model does not know enough" turn out to be retrieval problems rather than training problems.

What "open weight model" means

Now that weights make sense, this term is straightforward. An open weight model is one where the organisation publishes the actual weights file for anyone to download. You get the recipe card itself, not access to somebody else's kitchen.

Qwen, Llama, Mistral, Gemma, Phi and the open DeepSeek releases all ship this way, typically as a single file such as model.gguf or model.safetensors. A file named something like Qwen3-4B-Q4_K_M.gguf is exactly that: one file holding all the weights, small enough to run on a laptop, and fully functional offline once downloaded.

Closed models are reached through an API or a website. The weights stay with the provider, and your questions and documents travel to their servers to be answered.

Open weight models run on your own hardware, work with no network connection, can often be fine-tuned further depending on the licence, and keep your data on your machine.

The practical difference: asking a cloud assistant what the fix was for ticket 1021 sends that ticket data to somebody else's servers. Running an open weight model locally against your own knowledge base sends nothing anywhere. If you want to try that, how to run AI locally without a GPU is the step-by-step version.

One caveat worth taking seriously: open weight does not mean unrestricted. Licences vary considerably — some permit personal use only, some research, some full commercial use, and some carry real conditions. Check the specific licence before building anything on top of a model.

Matching the method to the problem

The rule worth remembering

RAG is for knowledge — what the assistant can look up. Training is for behaviour — how the assistant answers. An open weight model is the file you run yourself, where both of those happen.

If you are building a support or knowledge assistant, start with an open weight model and RAG. Add fine-tuning later, and only for a specific named problem: the format keeps slipping, classification accuracy is too low, or the tone is wrong. Do not fine-tune first and hope it fixes a knowledge gap. It will not, because fine-tuning changes behaviour rather than facts.

For how these choices land inside an institution rather than a side project — who owns the model, what data it may see, and what happens when a vendor changes terms — see sovereign AI and AI governance in the United Nations.

Frequently asked questions

What is the difference between RAG and fine-tuning?

RAG gives a model information to look up at the moment it is asked. Fine-tuning changes the model's permanent behaviour by retraining its weights. RAG is for facts that may change; fine-tuning is for patterns that should stay constant.

Do I need to fine-tune a model to use my own company data?

Usually not. If the goal is answering questions from your documents, retrieval is almost always the right starting point. It is faster to set up, cheaper to run, and mistakes can be corrected by editing a document rather than retraining a model.

What does "weights" mean in a large language model?

Weights are the billions of numeric values inside a model that were adjusted during training and now determine how it responds. They are the model's entire knowledge and behaviour, stored purely as numbers. There is no separate rule set or database alongside them.

What is an open weight model?

A model whose weights file is published for download, so it can be run on your own hardware rather than only through somebody else's API. Qwen, Llama, Mistral and Gemma are common examples.

Can I run a large language model without an internet connection?

Yes, if it is an open weight model. Download the weights file, such as a .gguf, and run it locally with a compatible runner. After the download, no network connection is needed.

Is RAG just fine-tuning with extra steps?

No. Fine-tuning permanently alters the model's internal weights. RAG leaves the model completely unchanged and simply supplies relevant text for it to read before answering.

Which should I build first, RAG or fine-tuning?

RAG. It is cheaper, reversible, and addresses the most common complaint, which is that the assistant does not know enough about your organisation. Fine-tuning is worth adding once you have a specific behavioural problem to solve.

← All articles