What Is a Small Language Model (SLM)?
A small language model is a language model of roughly 100 million to 10 billion parameters, small enough to run on a laptop, phone, or edge device instead of a GPU server. Named families include Microsoft Phi, Google Gemma, Meta Llama at 1B–8B, Qwen small models, and Mistral small models.
It is the same transformer technology as a large model, just deliberately scaled down so it fits in the memory of a laptop, phone, or edge device and responds with low latency. The practical marker of an SLM is where it can run: because a 4-bit quantized model needs about half to two-thirds of its parameter count in gigabytes for its weights, plus headroom for the context window, a 1B–8B model fits comfortably on a modern laptop or an NPU with no GPU server at all. For how parameter counts map to memory and hardware, see the LLM parameter-size guide.
What Is a Large Language Model (LLM)?
A large language model is a general-purpose model of tens to hundreds of billions of parameters, trained to maximize reasoning and world knowledge across domains. It needs a GPU server or a cloud API to serve it, and the frontier systems behind ChatGPT, Claude, and Gemini all sit in this class.
Size buys capability because more parameters store more patterns and more world knowledge, which is what lets a large model handle open-ended prompts, follow long and ambiguous instructions, and reason across domains it was never explicitly tuned for. The trade-off is real: that capability comes with more memory, higher latency, and either a GPU server or a metered cloud API to serve it. The LLM parameter-size guide breaks down what each size class buys and what it costs to run.
For plain-English expansions of the terms used throughout this comparison — parameters, tokens, context window, retrieval — the AI glossary keeps them in one place. For how the categories themselves nest, the generative AI vs. LLM breakdown shows where a large language model sits inside the wider generative AI family.
SLM vs LLM: The Head-to-Head Comparison
The clearest way to choose is dimension by dimension — an SLM wins on size, cost, latency, offline operation, and privacy, while an LLM wins on raw breadth and open-ended reasoning. The table below is the side-by-side most teams need; for the parameter counts in its first row, see LLM parameters explained.
| Dimension | Small Language Model (SLM) | Large Language Model (LLM) |
|---|---|---|
| Parameters | ~100M – 10B | ~10B – 1T+ |
| Runs on | Laptop, phone, NPU, edge device | GPU server or cloud API |
| Works offline? | Yes — often fully on-device | Rarely — usually cloud-hosted |
| Latency | Low (local, small) | Higher (network + compute) |
| Cost model | One-time hardware, near-zero per query | Per-token or GPU-hour |
| Best at | Focused, high-volume, low-latency tasks | Open-ended reasoning and breadth |
| Privacy | Easiest to keep fully private | Depends on where it runs |
| Example families | Phi, Gemma, Llama small, Qwen small, Mistral small | GPT, Claude, Gemini, Llama 70B+ |
Named Small Language Models: Sizes, Memory and Where They Run
The small models teams actually deploy come from six open families — Gemma, Granite, Llama, Phi, Mistral, and Qwen — each shipping variants between roughly 0.6B and 8B parameters with downloadable weights. The variant you pick is mostly a memory decision: a 4-bit quantized model needs roughly half to two-thirds of its parameter count in gigabytes for weights, plus headroom for context, so the size class determines whether it runs on a phone, a laptop, an AI PC, or a single entry-level GPU.
| Model family | Small variants | Machine memory (RAM/VRAM, quantized) | What it runs on |
|---|---|---|---|
| Google Gemma | Gemma 3 — 1B, 4B | 2–4 GB (1B) · 6–8 GB (4B) | Phone or thin laptop at 1B, AI PC at 4B |
| IBM Granite | Granite 3 — 2B, 8B | 4–8 GB (2B) · 8–16 GB (8B) | Laptop or edge server; the 8B fits a single small GPU |
| Meta Llama | Llama 3.2 — 1B, 3B · Llama 3.1 — 8B | 2–4 GB (1B) · 6–8 GB (3B) · 8–16 GB (8B) | Laptop, NPU, or edge device, fully offline |
| Microsoft Phi | Phi-4-mini — 3.8B · Phi-3.5-mini — 3.8B | 6–8 GB | NPU laptop or AI PC |
| Mistral | Ministral — 3B, 8B · Mistral 7B | 6–8 GB (3B) · 8–16 GB (7–8B) | Laptop, NPU, or a single entry-level GPU |
| Qwen | Qwen3 — 0.6B, 1.7B, 4B, 8B | 2–4 GB (0.6–1.7B) · 6–16 GB (4–8B) | Phone at 0.6B, laptop and edge box from 1.7B up |
Memory figures follow the size tiers published in the LLM parameter-size guide, which maps each parameter class to the CPU, GPU, and RAM it needs; the hardware sizing guide turns that into a machine specification. For a ranked, model-by-model pick rather than the family view, see the LLM selection guide. AirgapAI runs Llama, Gemma and Qwen small models from this table offline on Intel NPU laptops, so it doubles as the shortlist for an on-device deployment.
When to Choose an SLM
Choose an SLM when latency, cost at volume, or data privacy matter more than raw breadth. On a narrow, well-defined task with clean data, a well-chosen small model is not a compromise — it is often the right tool, delivering the same answer faster and cheaper than a frontier model.
- On-device and offline. The workload must run on a laptop, phone, or air-gapped device with no internet — the defining SLM use case, and how a private LLM keeps data inside your boundary.
- High volume, low latency. When you run the same task thousands of times a day, a small local model's near-zero per-query cost and fast response beat metered cloud tokens.
- Narrow, well-scoped tasks. Classification, extraction, routing, tagging, and first-draft drafting are exactly the jobs a focused small model handles reliably.
- Privacy-sensitive data. If prompts or documents cannot leave your environment, a small model running locally is frequently the only viable option.
When to Choose an LLM
Choose an LLM when the task needs deep reasoning, broad world knowledge, or flexible multi-domain output that a small model cannot cover. The extra size and cost are worth it precisely when the work is open-ended and hard to bound in advance.
- Open-ended reasoning. Ambiguous prompts, long multi-step chains of thought, and tasks that require weighing trade-offs benefit from a large model's headroom.
- Breadth of knowledge. Questions that span many domains, languages, or specialties lean on the wide coverage only a large model carries.
- Complex agentic work. When an agent has to plan, use many tools, and adapt across a long horizon, a stronger model reduces the failure rate at each step.
- Low-volume, high-stakes output. When each answer matters more than per-query cost, the broadest available capability is the right default.
The Hybrid Pattern: SLM + LLM Routing
The dominant 2026 architecture is not SLM or LLM — it is both, with a router that sends routine work to a small local model and escalates only the hard minority to a large one. Most requests in a real workload are simple and repetitive; a well-chosen SLM handles that majority on-device at near-zero marginal cost, and the router promotes only the genuinely difficult 10–15% of cases to an LLM.
This keeps cost, latency, and data exposure low for the common case while preserving frontier capability for the cases that need it. The decision that makes it work is picking the right specific model for each tier — which is a model-selection question, not a category one. For the ranked, model-by-model shortlist, see the LLM selection guide, which owns the “which exact model” decision.
SLMs and Private, Offline AI
Small models are the mechanism behind on-device and air-gapped AI — the reason private, offline assistants are possible at all. A model has to be small enough to fit and run on local hardware before it can run without a cloud, so the shift to private AI and the rise of capable SLMs are the same story told from two angles. This is the part most SLM-vs-LLM comparisons leave out.
AirgapAI puts it into practice: it runs open small models such as Llama, Gemma, and Qwen fully offline on Intel NPU laptops via OpenVINO, so a regulated team gets a private assistant with no cloud dependency and no GPU server. The catch with any small model is accuracy on your data — a compact model has less room to memorize, so it depends on retrieval. Blockify restructures your content into clean, deduplicated IdeaBlocks so a small model answers reliably from your knowledge base rather than guessing. Together they let an SLM punch well above its size on the tasks that matter to you.
To go deeper on the deployment side: local LLM covers running a model on a single machine, how to run an LLM locally is the step-by-step, and the hardware sizing guide maps model sizes to the CPU, GPU, and NPU you actually need. For the on-device split between which models run on an AI PC NPU and which still want a GPU, the Intel edge page has the breakdown; and when the model you pick is your own, bring your own model covers running it inside AirgapAI.