How to Run an
LLM Locally

How Do You Run an LLM Locally?

You run an LLM locally by installing a runtime (Ollama, LM Studio, or llama.cpp), downloading a quantized open-weight model that fits your memory, and then running that model directly on your CPU or GPU — with no cloud, no API key, and no internet after the initial download. The entire workflow is free and open source, and a capable laptop is enough to start.

The reason interest has exploded is simple: privacy plus capability. Self-hosting AI is now a mainstream choice rather than a niche one — in Stack Overflow's 2024 developer survey, 76% of developers reported using or planning to use AI tools, and a fast-growing share run models locally to avoid sending code and data to third-party APIs (Stack Overflow Developer Survey, 2024). Open-weight models have closed much of the quality gap with proprietary cloud models, so a 7B–14B model on your own machine is genuinely useful for chat, coding help, summarization, and document Q&A.

What Do You Need to Run an LLM Locally? (Hardware Checklist)

You need three things: a local LLM tool, a quantized model file, and enough RAM or VRAM to hold it. Memory is the single most important constraint. A practical rule of thumb is that a 4-bit quantized model uses roughly 0.6–0.7 GB of memory per billion parameters, so a 7B model fits in about 5–6 GB and a 13B in about 9–10 GB, with a few gigabytes of headroom for the operating system and the context window.

• RAM / VRAM: 16 GB total handles 7B–13B models well. 8 GB is enough for small (3B–7B) models. 24 GB+ of GPU VRAM or 32 GB+ of system RAM opens up 30B–70B models.
• GPU (optional but faster): an NVIDIA card with 8–24 GB VRAM gives the best speed; Apple Silicon (M-series) Macs are excellent because the GPU shares unified memory.
• CPU: any modern multi-core CPU works. Newer Intel Core Ultra and AMD chips include an NPU that accelerates on-device AI without a discrete GPU.
• Disk: 5–50 GB free per model. Quantized 7B files are ~4–6 GB each; larger models grow accordingly.
• OS: Windows, macOS, and Linux are all fully supported by the three tools below.

That checklist sizes a single desktop or laptop. If you are stepping up to serving several people at once, the hardware sizing guide works through which GPU to buy for AI workloads and how much memory each tier of model needs.

Quantization is what makes this feasible on consumer hardware. It compresses model weights from 16-bit to 4-bit (or 5/6/8-bit), cutting memory use by roughly 4x with only a small quality loss. Most local models you download are already quantized in the GGUF format that Ollama, LM Studio, and llama.cpp all read. For a fuller definition of a local LLM and help choosing which model family to run, see that guide. For more information on which models are supported and how to bring your own, visit the Supported LLM Models and Bring Your Own page.

1 Step 1: Pick a Tool — Ollama vs LM Studio vs llama.cpp

Choose LM Studio if you want a one-click graphical app, Ollama if you want a clean command line with a built-in API, or llama.cpp if you want maximum control and the leanest possible footprint. All three are free, open source, run the same GGUF models, and work on Windows, macOS, and Linux. In fact, Ollama and LM Studio are both built on top of the llama.cpp engine — so picking is really about the interface you prefer.

All three are open-source projects: Ollama, LM Studio, llama.cpp.

Our recommendation for most readers: start with LM Studio if you have never run a model before — it has a built-in model catalog, automatic hardware detection, and a chat window. Move to Ollama the moment you want to script the model, plug it into an editor, or expose a local API. Reach for raw llama.cpp only when you need custom compilation flags or are embedding inference into another application. If you are weighing Ollama alternatives for a team rather than a single machine, the enterprise roundup compares the supported options side by side.

2 Step 2: Choose a Model That Fits Your Hardware

Pick the largest open-weight model your memory can hold at 4-bit quantization — for most laptops that means a 7B–14B model such as Llama 3.x, Qwen 2.5, Gemma 2, or Mistral. Bigger is generally smarter, but only if it fits in memory without spilling to disk, which collapses speed. Match the model to your RAM first, then to the task.

Open models have become remarkably capable. Meta has reported that its Llama family surpassed 1 billion downloads, underscoring how mature the open-weight ecosystem now is (Meta, 2025). For privacy-sensitive work, the practical upside is that a model living on your disk has no usage caps, no per-token billing, and no exposure of your prompts — the same open models (Llama, Gemma, Qwen, Mistral) that power local DIY setups also run inside supported products like AirgapAI.

Which GGUF Quantization Should You Download?

Download the Q4_K_M build of the largest model your memory can hold. It averages about 4.8 bits per weight, cuts a 16-bit model to roughly a quarter of its size, and costs a quality drop most people never notice. Step up to Q5_K_M or Q6_K only when you have memory to spare, and drop below 4-bit only to fit a class of model that otherwise will not run.

File sizes are the published GGUF builds for Llama 3.x, Qwen 2.5 and Mistral families; every runtime on this page reads the same files. Sizing rule: file size in GB is roughly parameters in billions × bits per weight ÷ 8.

What the GGUF suffixes mean

• Q8_0 — about 8.5 bits per weight — effectively indistinguishable from the full-precision model, and roughly twice the size of a 4-bit build. Use it when memory is abundant and the answer has to be exact.
• Q6_K — very close to Q8_0 at about three quarters of the size. A good stop when a 4-bit build feels careless but Q8_0 will not fit.
• Q5_K_M — a modest step up from 4-bit for roughly 15% more memory. Worth taking when the model is small relative to your RAM.
• Q4_K_M — about 4.8 bits per weight and the most-downloaded build of nearly every open-weight model. This is the default answer for almost everyone.
• IQ3_M / Q3_K_M — the fit-a-bigger-model option. Reasoning and instruction-following degrade measurably; a Q4_K_M build one size class down is usually the better trade.
• Q2_K — experiments only. Output quality falls off a cliff, and a smaller model at Q4_K_M will beat it on the same memory.

3 Step 3: Install the Tool and Download the Model

Installation is a single download for each tool, and pulling a model is one command or one click. Everything below runs offline after the model file finishes downloading. Here is the fastest path for each tool.

Copy-Paste Commands for Ollama, LM Studio and llama.cpp

Every runtime on this page follows the same three moves: install the runtime, pull a quantized model, then start either a chat session or a local OpenAI-compatible API. The blocks below are the commands for each one, including the endpoint it exposes, so an existing script can point at your own machine by changing a single base URL.

# 1. install (macOS and Linux; Windows has an installer at ollama.com)
curl -fsSL https://ollama.com/install.sh | sh

# 2. pull a 4-bit model and start chatting (first run downloads it)
ollama run llama3.1:8b

# 3. manage what is on disk
ollama list
ollama rm llama3.1:8b

# 4. run the local API in the background
ollama serve

# 5. call the model from a script
curl http://localhost:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"Summarize this in three bullets"}'

# 1. install the desktop app from lmstudio.ai, then add its command line
npx lmstudio install-cli

# 2. list the models you downloaded in the app
lms ls

# 3. load one into memory
lms load llama-3.1-8b-instruct

# 4. start the OpenAI-compatible server
lms server start

# 5. check what is running
lms ps

# 1. build (CMake detects Metal on macOS and CUDA on Linux/Windows)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

# 2. chat with a GGUF file (-ngl 99 offloads every layer to the GPU,
#    -c sets the context window, -cnv opens conversation mode)
./build/bin/llama-cli -m models/Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 4096 -ngl 99 -cnv

# 3. or serve the same model over an OpenAI-compatible API
./build/bin/llama-server -m models/Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 4096 -ngl 99

4 Step 4: Run and Chat (CLI or GUI)

Once the model is loaded, you chat with it exactly like a cloud assistant — in LM Studio's chat window, in Ollama's terminal prompt, or through a local API on your own machine. The first response may take a moment as the model loads into memory; after that, replies stream token by token.

• GUI chat: in LM Studio, type into the chat box and adjust temperature, context length, and system prompt from the sidebar — no code needed. For what temperature and the other sampling settings actually change, see the prompt engineering guide.
• CLI chat: with Ollama, ollama run llama3.1 opens an interactive prompt; type your message and press Enter to get a streamed reply.
• Local API: both Ollama and LM Studio expose an OpenAI-compatible endpoint (typically on localhost), so existing apps and scripts can point at your local model by changing one base URL — no key, no cloud.
• Editor integration: developer tools can connect to that local endpoint for in-editor coding help. For a packaged, supported local coding assistant, see AirgapAI Code.

Expect roughly 5–15 tokens per second for a 7B model on a recent laptop CPU, and 40–100+ tokens per second on a dedicated GPU. If responses feel slow, that is almost always a sign the model is too large for your memory — drop to a smaller model or a more aggressive quantization.

Watch: A Model Answering With the Network Unplugged

This 68-second clip shows a model answering on a stock laptop with the network cable pulled out — no API key, no round trip, no connection at all. The assistant on screen is AirgapAI, the packaged option covered further down, but the loop is identical to what Ollama, LM Studio or llama.cpp give you once the weights are on disk: prompt in, tokens out, nothing leaves the device.

Recorded on a stock laptop with the network cable unplugged. See the full AirgapAI demo.

5 Step 5: Add Your Own Documents (RAG Basics)

To make a local LLM answer from your own files, you use retrieval-augmented generation (RAG): your documents are split into chunks, converted to embeddings, stored in a local vector index, and the most relevant pieces are fed to the model with each question. This keeps everything offline while letting the model cite your PDFs, notes, and internal docs.

• Easiest path: LM Studio and front-ends like Open WebUI or AnythingLLM let you drag in documents and chat over them with a local model — no coding.
• Developer path: pair Ollama with a local vector database and one of the open embedding models to build a custom RAG pipeline you fully control.
• The accuracy lever: RAG quality lives or dies on how cleanly your source text is prepared. Messy, duplicated, or poorly chunked documents cause hallucinations.

This is where data optimization matters most. Iternal's Blockify restructures raw documents into compact, deduplicated IdeaBlocks before they reach the vector database — an approach that delivers roughly 78X more accurate retrieval using about 3X fewer tokens, and works with any vector store. For local RAG that needs to be trustworthy, cleaning the data first is the highest-leverage step you can take.

Chat With Your Own Documents Without Sending Them Anywhere

A document-grounded setup on your own machine is five parts: a folder of documents, a cleaning pass, a local embedding model, a local vector index, and the chat model you already installed. Everything stays on disk. The part that decides whether the answers are trustworthy is the cleaning pass, not the model you picked.

1. Collect the documents in one folder Start narrow: one project, one product line, one policy set. A tight corpus of 50 clean documents outperforms a dump of 5,000 mixed ones, because retrieval has fewer near-identical passages to confuse.
2. Clean and structure the text before it is chunked Strip navigation furniture, run OCR over scanned PDFs, and collapse the six near-duplicate versions of the same policy into one current version. This is the step most local setups skip, and it is the step that decides whether the answers are usable.
3. Pick an embedding model that runs locally Compact open embedding models such as nomic-embed-text or the BGE small family run on the same machine as the chat model and add very little memory. Whatever you choose, embed and query with the same model.
4. Store the vectors in a local index Chroma, LanceDB and Qdrant all run on a laptop, and front-ends like Open WebUI or AnythingLLM ship with a built-in store so there is nothing to configure. The index lives in a folder you control.
5. Query with citations turned on Ask the front-end to show which passages it retrieved. If an answer looks wrong, the retrieved chunks tell you immediately whether the problem is the model or the source text — and it is usually the source text.

What actually breaks local document chat

• Scanned PDFs with no OCR layer — they embed as empty pages, so the model answers from nothing.
• Near-duplicate documents — five revisions of one policy split the retrieval score five ways and the model quotes the stale one.
• Chunks that cut through tables — half a table has no meaning, and the model will confidently invent the other half.
• Boilerplate headers and footers — repeated on every page, they dominate similarity search and crowd out the real content.

Troubleshooting & Performance Tips

Most local-LLM problems trace back to one cause: the model is too big for your available memory. The fixes below resolve the overwhelming majority of slow, crashing, or out-of-memory sessions.

• Replies are very slow: the model is spilling out of RAM/VRAM. Use a smaller model or a lower-bit quantization (try 4-bit, or drop from 13B to 7B).
• Out-of-memory / crash on load: reduce the context length, close other apps, or pick a smaller quant. On GPUs, lower the number of offloaded layers.
• GPU not being used: confirm GPU offload is enabled in LM Studio's settings or set the GPU-layers flag in Ollama/llama.cpp; verify your drivers (CUDA/Metal) are current.
• Weak answers: raise the quantization (8-bit over 4-bit if memory allows), choose a larger or more recent model, or improve your system prompt and RAG document quality.
• Short, cut-off responses: increase the maximum output tokens and the context window in your tool's settings.

From DIY to Production: The Turnkey Team Option

A DIY local LLM is ideal for one person, but it breaks down for teams the moment you need support, central updates, audit logs, packaging for non-technical staff, or compliance. That is the line where organizations move from Ollama-on-a-laptop to a supported, air-gapped product. AirgapAI is that turnkey option: the same 100% offline privacy you get from DIY, delivered as a one-click install with real support behind it.

AirgapAI runs the same open models you would choose yourself — Llama, Gemma, Qwen, Mistral — and is optimized to run on Intel NPU laptops via OpenVINO, so it gets full local AI without a discrete GPU. Crucially, it keeps every prompt and document on-device — a fully offline AI deployment — which is why it suits regulated, defense, and government users who cannot send data to a cloud API. For the organization-wide server path (many concurrent users, GPU clusters, vLLM), pair this with How to Deploy an LLM On-Premise and the Private LLM guide. Comparing options? See the best local AI tools for enterprise.

Why Running an LLM Locally Is Worth It

Running an LLM locally gives you three things a cloud chatbot cannot: total privacy, zero marginal cost, and offline reliability. Your prompts never touch a third-party server, you pay nothing per token, and the model works with no connectivity at all. For individuals that means freedom from usage caps and data exposure; for organizations it means proprietary IP and regulated data stay inside the perimeter.

The data underscores the stakes. IBM's 2024 study put the global average cost of a data breach at USD 4.88 million, a 10% year-over-year increase (IBM Cost of a Data Breach, 2024). Sending sensitive prompts to an external model is one more exposure surface; keeping inference local removes it. That is the entire premise behind on-device AI — and why a growing share of developers and regulated enterprises now run models themselves rather than calling a cloud API.