How to Run AI Locally Without a GPU: A Beginner's Guide

You do not need a graphics card or a cloud subscription to run a language model. This is the whole setup on an ordinary Windows machine: the llamafile runner, a quantized GGUF model, and one batch file. It runs offline, and it runs from a USB drive.

Published 2026-09-04 · By Shahzad Asghar

Most people assume running AI means paying for a cloud subscription or buying an expensive graphics card. Neither is true any more. This guide covers how to run AI locally without a GPU on an ordinary Windows machine, an external drive, or a USB stick, using free and open tools.

The model runs on your computer. Your prompts never leave it.

What local AI actually means

Local AI means the language model executes on your own hardware. There is no API call, no account, and no third party receiving the text you type. Once the runner and the model file are downloaded, the whole thing works with the network switched off.

This is the practical form of what is often called sovereign AI: you control the model, the data, the cost, and the machine it runs on. The same principle applies whether you are one person protecting personal notes or an institution that cannot send case files to an external provider. I write about the institutional version of this question under sovereign AI.

Why bother, when cloud AI is faster

Cloud AI is faster and more capable. That is not in dispute. Local AI is worth the trade in specific situations:

• Your prompts and documents stay on your own disk
• No monthly bill for ordinary personal use
• It works offline once installed
• It is a safe way to learn how these models behave
• It runs from a portable drive, so the setup travels with you
• It removes dependence on a single vendor's pricing and availability

The honest limitation: without a GPU it is slower, and a large model on a modest CPU can be slow enough to be annoying. Choosing the right model size is most of the battle, and I come back to that below.

What you need

• A Windows PC
• 8 GB RAM minimum, 16 GB or more if you want a larger model
• Roughly 3 GB free storage for a 4B quantized model, more for bigger ones
• A model file in GGUF format
• The llamafile runner from Mozilla

A GPU is optional. Everything here works on CPU alone.

The tools, and why these ones

llamafile is a single-file runner from Mozilla that wraps llama.cpp. It needs no installer, no Python environment, and no system changes. That matters on a locked-down work laptop where you cannot install software.

GGUF is the model file format llama.cpp uses. Quantized GGUF models are compressed so they fit in ordinary RAM: a 4B model at Q4 quantization is around 2.5 GB rather than 16 GB.

Official sources, all verified:

• Mozilla llamafile
• llamafile releases on Hugging Face
• llamafile on GitHub
• llama.cpp on GitHub
• Qwen3-4B GGUF
• Ollama for Windows, a friendlier alternative if you would rather not touch a batch file

Step 1: Make a folder

Create a folder anywhere:

``text D:\LocalAI ``

A USB drive or external SSD works too:

``text E:\LocalAI ``

Use an external SSD rather than a cheap flash drive if you have one. Model files are large and read speed is the difference between a model that loads in seconds and one that takes minutes.

Step 2: Download the llamafile runner

Open PowerShell in that folder and run:

``powershell curl.exe -L -o llamafile.exe https://huggingface.co/mozilla-ai/llamafile_0.10/resolve/main/llamafile-0.10.5-thin ``

That downloads roughly 42 MB and saves it as llamafile.exe. The "thin" build is the runner on its own, without a model baked in, which is what you want when you plan to swap models.

Step 3: Download a GGUF model

Model size is the single decision that determines whether this feels usable. On CPU:

Start at 4B. Move up only once you know the setup works.

Download a .gguf file from the Qwen3-4B GGUF page and put it in the same folder as llamafile.exe. A typical filename looks like:

``text Qwen3-4B-Q4_K_M.gguf ``

Q4_K_M describes the quantization. It is a good default: roughly a quarter of the original size with modest quality loss.

Step 4: Write the batch file

Create a file called runlocalai.bat in the same folder, containing:

``bat @echo off cd /d "%~dp0" ".\llamafile.exe" -m ".\Qwen3-4B-Q4_K_M.gguf" --server -c 4096 -np 1 ``

Replace Qwen3-4B-Q4_K_M.gguf with your actual filename. It must match exactly, including capitalisation.

What the flags do: -m points at the model, --server starts the local web interface, -c 4096 sets the context window to 4096 tokens, and -np 1 handles one request at a time, which is right for a single user on CPU.

Step 5: Run it

Double-click runlocalai.bat. The first load takes a while on CPU because the whole model is read into RAM.

When it is ready, open:

``text http://127.0.0.1:8080 ``

That is a local address. Nothing is being served to the internet.

Step 6: Confirm it is working

Open:

``text http://127.0.0.1:8080/health ``

A working server returns:

``json {"status":"ok"} ``

Running it from a USB drive

This works, provided all three files sit in the same folder:

``text runlocalai.bat llamafile.exe Qwen3-4B-Q4_K_M.gguf ``

The batch file uses cd /d "%~dp0", which means it runs from wherever it happens to be. The drive letter can change between machines and it will still work.

An external SSD is worth the money here. A slow USB 2.0 stick can turn a thirty-second model load into several minutes.

Does it work offline?

Yes. The network is needed once, to download the runner and the model. After that you can disconnect entirely. This is the property that makes local AI useful in places with unreliable connectivity, which is a constraint I have written about at length in the Last-Mile AI Framework.

How slow is it really?

Speed depends on CPU cores, RAM speed, model size, and quantization. Rough expectations on a mid-range laptop with no GPU:

• A 1B model responds quickly enough for conversation
• A 4B model produces a steady few words per second
• An 8B model may take a noticeable pause before each reply

If it is too slow, reduce the model size before anything else. That single change has more effect than any flag.

Troubleshooting

The window opens and closes immediately. The model filename in the batch file does not match the file on disk. Check it character by character.

"llamafile.exe is not recognized". The download did not complete, or it saved under a different name. Check the file is present and around 42 MB.

The computer freezes or swaps to disk. The model is too large for your RAM. Drop to a smaller one.

The browser shows nothing at 127.0.0.1:8080. Give it longer. On CPU, loading an 8B model can take several minutes before the server answers.

Windows SmartScreen blocks it. Expected for an unsigned executable downloaded from the internet. Verify you downloaded from the Hugging Face link above before allowing it.

Where this fits

For an individual, local AI is a privacy decision. For an institution, it is a governance one: which data may leave the building, who is accountable when a model is wrong, and what happens when a vendor changes terms. Those questions do not disappear because the model is small — they change shape. I cover that side under AI governance in the United Nations and, for delivery in difficult environments, humanitarian AI.

If you want to understand what is actually happening inside the model you have just downloaded, the free Learn LLMs course works up from next-token prediction to the Transformer.

Frequently asked questions

Can you run AI locally without a GPU?

Yes. A quantized model in GGUF format runs on CPU alone using a runner such as llamafile. A 4B model needs roughly 3 GB of disk and 8 GB of RAM. It is slower than a GPU or a cloud service, but it works, and everything stays on your machine.

How much RAM do you need to run a local AI model?

Around 8 GB for a 4B quantized model, and 16 GB if you want to run 7B or 8B comfortably. The rule of thumb is that the model needs slightly more RAM than the size of its file on disk, plus headroom for the context window.

What is the best model size for a computer without a GPU?

Between 2B and 4B parameters at Q4 quantization. Smaller models answer quickly but reason poorly; 7B and 8B models answer better but often too slowly to be pleasant on CPU. Start at 4B and adjust from there.

Can local AI run from a USB drive?

Yes, provided the runner, the model file, and the batch file are all in the same folder. Use an external SSD rather than a flash drive: model files are large and read speed determines how long loading takes.

Is local AI private?

Your prompts stay on your own machine, and after installation no network connection is required. That removes the third party from the equation. It does not make the machine itself secure, so full-disk encryption and access control still matter if the data is sensitive.

What is sovereign AI?

Sovereign AI means controlling where a model runs, what data it sees, which model is used, and what it costs, rather than depending entirely on an external provider. Running a model locally is the simplest form of it. At an institutional level it extends to procurement, hosting, and the ability to keep operating if a vendor withdraws.

← All articles