Pick the model first and you will buy the wrong machine. The order that works runs the other way: the device decides which model can load, the model decides speed and answer quality, and only then does anyone know how many machines the job takes. Buyers put it plainly, early, in almost every deployment conversation: do the machines need a special AI chip or a dedicated GPU?
Reference Architecture & Sizing
What Hardware Do You Need for a Local LLM,
and How Do You Size the Deployment?
The memory ladder, the processor generations that qualify, what an NPU does and does not buy you, the throughput bands to plan against, and the arithmetic that turns users and documents into a server count.
Built from real buyer questions in our sales meetings
Direct Answer
Size the device before you size the model. Three things bind a local assistant: accelerator or system memory, the processor generation, and whether an NPU or a discrete GPU is present. Iternal recommends a 2024-or-newer device for AirgapAI, ideally carrying an NPU or a GPU. 32GB has been called the sweet spot, and 32GB with an i7 called good for the software though the graphics setup still matters — memory alone does not settle a machine. A customer described 16GB as not great but enough to run a 3B model. Keep the hedge; it is the claim.
The limit: a qualifying device buys you a smaller model, not parity. Iternal describes the product as hamstrung by whatever device the customer already has and the model that device can install, and puts a reasonably powered local machine about a year behind what a data center can do. Every throughput figure here is a planning band drawn from sales conversations, each with its own hedging and attribution. None is an independently measured result.
Verify it on one machine before you commit a fleet. AirgapAI prompts for a benchmark at setup and whenever a new model is added; the run scans the machine, confirms it is taking advantage of a GPU where one exists, and reports the hardware, average tokens per second and prefill time. On the server side, one benchmark on one lab machine reveals the server count quickly.
Specifying the deployment and choosing the route are different decisions. The specification each route implies is above: what a qualifying laptop looks like, what a shared server carries, and the arithmetic between them. For more information on choosing between those routes, visit the placement page; for how these figures are measured, visit the speed page.
AI Infrastructure, Explained On-Premises First
AI infrastructure is the compute, memory, storage, network and serving software a model needs to answer a question: an accelerator, the memory holding the weights, the runtime that loads them, the data set it retrieves from, and the management layer around all of it. On-premises, every one of those layers sits inside your own boundary.
Most published descriptions of artificial intelligence infrastructure start at a rented GPU cluster and work outward. Start instead where the data has to stay, and the stack reads differently: the same seven layers, with an accelerator you own at the bottom and no step where a request leaves the building. The specification work below — memory rungs, processor tiers, server counts — is just those layers with numbers in them.
The AI infrastructure stack, layer by layer
| Layer | What sits in it | What sets its size |
|---|---|---|
| Silicon | CPU, integrated GPU and NPU on the device; a discrete GPU on a workstation; Intel Xeon alongside NVIDIA accelerators, or Gaudi, on the server side | Processor generation and tier, and whether an accelerator is present at all |
| Memory | The resident model weights plus the working context, in system RAM or in VRAM | Parameter count and precision — the binding constraint on almost every fleet |
| Storage | The application, the model files and the local data set | Roughly 3.5GB per user once a model is in place; about 2.3GB for the application plus a 3B model |
| Network | Localhost by default on the device; a relay on the device when it reaches a server-hosted model | Whether inference runs on the machine or on a shared server inside your boundary |
| Runtime and model serving | The engine that loads the weights and executes them — OpenVINO-converted models running across CPU, GPU or NPU, or a server-side serving stack | Which silicon the models were compiled for; each NPU needs its own compiler |
| Data and retrieval | The data set the assistant answers from, its metadata and the generated vectors | Pages of new intellectual property created per period, and the processing window the use case allows |
| Management and measurement | Imaging and packaging, the benchmark run, model updates, support | Fleet size and how many standard builds the estate carries |
What changes when the data cannot leave
Four things, and each one moves a number in the specification later on this page.
- The boundary replaces the API key. There is no per-token meter and no request leaving the machine, so the budget conversation stops being consumption and starts being capital: silicon, memory and disk, bought once.
- Capacity is bought, not rented. Nothing autoscales at 9am. Peak concurrency has to be resident in hardware you already own, which is why the server count below is a division rather than a forecast.
- The model travels to the data. Retrieval runs against a data set that lives beside the model, on the device or on a server inside the same boundary, so ingestion volume becomes an infrastructure input rather than a content decision.
- Refresh is a distribution problem. A new model version is a file that has to reach every machine, so a fleet plan needs a path for model updates the same way it needs one for the application.
Infrastructure is AI-ready when each of those seven layers has an owner, a number and a measurement behind it: a device class per user population, a model and precision per class, storage per user, an ingestion volume, a throughput target, a concurrency figure per server, and one benchmark the whole set rests on. Anything short of that is a shopping list. Generative AI infrastructure is the same stack with a larger memory rung, so an AI ready infrastructure plan for a chat assistant differs from one for classification only in the parameter count and precision it has to hold. When part of that stack has to sit outside the boundary, the design becomes a hybrid one; for more information visit the hybrid AI infrastructure page.
The four AI infrastructure solutions buyers compare
Nearly every on-premises design lands in one of four categories, and they differ in where inference runs rather than in what the software does.
Solution categories
- An AI PC fleet. Inference on the endpoint, one model resident per machine, nothing shared. Sized by the memory ladder and the processor tier; the ceiling is whatever the smallest standard build can load.
- A shared server inside your boundary. One model serving many people from a Xeon-plus-NVIDIA machine — a DL380a on the private-cloud build, a PowerEdge XE-tier box for heavier models. Sized by concurrency arithmetic, not by seats.
- A mixed estate. Qualifying machines run the model locally while the rest reach a server-hosted model through a relay, which keeps the older builds in service and drops their local install to about a gigabyte and a half.
- A hybrid estate. The model and the retrieval set stay on-premises for the data that cannot leave, while workloads with no boundary constraint run on rented capacity; the on-premises half is sized exactly like the shared server above. For more information visit the hybrid AI architecture page.
Which category an estate belongs in is a placement decision rather than a specification; for more information visit the placement page. For the server-side arithmetic in accelerator terms — VRAM per model, cards per node, and what three years of it costs — visit the GPU sizing guide for LLMs. To score where your own compute, data and process maturity sits before the design work starts, take the AI Deployment Readiness Assessment.
Good, Better, Best: T-Shirt Sizing for Local AI Hardware
Buyers ask for this artifact by name. T-shirt sizing and rough order of magnitude are what people say when they want a starting configuration rather than a quote. Iternal publishes good, better and best recommendations for that moment: small, medium and large on the client side, and a server spreadsheet from extra small through large on ProLiant machines.
| Device class | Accelerator / memory | Largest model it runs | Throughput to plan for |
|---|---|---|---|
| The machine you already own Pre-2024 laptop |
No NPU; integrated graphics; commonly 16GB | 1B class; 2023-era hardware is already marginal | It works, and it is slow |
| Good Entry AI PC |
Intel Core Ultra 5, NPU plus integrated Arc graphics; 16GB | 3B class, described as not great but enough | Tokens arrive faster than a person reads |
| Better Mainstream AI PC |
Intel Core Ultra 7 (Meteor Lake, Arrow Lake, Lunar Lake, Panther Lake); 32GB | The bundled general-purpose models; a 12B may disappoint | Same band, more headroom |
| Best, single user Tower or mobile workstation |
Discrete GPU carrying more than 16GB of VRAM | 26B, 27B and 31B class | Benchmark the exact model; no planning band applies |
| Small desktop AI box | NVIDIA GB10 Blackwell superchip, rated at one petaflop | Larger open-weight models; the biggest want a cluster | Roughly 20 tokens per second (hedged — see below) |
| Shared server | Intel Xeon plus NVIDIA; the private-cloud build is a DL380a, heavier models want a PowerEdge XE-tier box | Bounded by server memory and model size, not by seats | Around 70 tokens per second (hedged — see below) |
The throughput column is a planning band for choosing hardware, not a measurement. None of it should be read as a benchmark result.
Read device vintage as a ladder, not a line. The dominant recommendation is a 2024-or-newer notebook or desktop on Intel Meteor Lake or better. Below it, hardware from 2023 or newer is pretty much certain to work while anything older is iffy — a floor, not a recommendation. Above it, any Dell PC from 2025 or newer should give a good end-user experience. So 2023 is where it starts working, 2024 is what to buy, and 2025 or later is where the experience stops being a compromise.
No single OEM is required. AirgapAI is optimized for Intel silicon through OpenVINO, which is why Intel integrated NPU and GPU machines perform best, and other PC manufacturers on the marketplace work fine. Iternal has proposed Blockify and AirgapAI together as a Xeon-based solution, with inference on both Xeon and Gaudi.
The Two Throughput Anchors, and Their Small Print
A planning band is a number you buy hardware against; a benchmark is one you measure. Two anchors are solid enough to plan against, and each carries small print.
Roughly 20 tokens per second on a small desktop AI box. Iternal describes a GB10-class unit running one of the big models at around 20 tokens per second. A separate account puts average reported throughput for a single GB10 near the same figure, relayed as an outside report rather than a first-party measurement. Both are hedged and they differ in who is speaking, so plan against roughly 20 without reading the pair as corroboration.
Around 70 tokens per second, in two different scopes. One Iternal account gives about 70 tokens per second as the chat throughput of a server-hosted model, hedged. Another states 70 flatly but attaches it to a 26B model rather than to a class of machine. Splicing those into one confident server-class figure would invent a claim neither makes. Use around 70 as a server-side planning band, and treat the 26B number as what it is: a model-specific generation speed.
Local throughput across the wider record runs from roughly ten tokens per second to several hundred depending on silicon, model and runtime, and what counts as usable output is its own question. Both belong to the page that owns measurement. For more information visit the speed page.
Pin it down: questions for your evaluation
-
Which device, model and quantization produced the tokens-per-second figure you are quoting us?Whether a planning band applies to the hardware you own, or to a machine nobody in your fleet has.
-
What accuracy is retained at the quantization you would ship for our device class?The real cost of squeezing a larger model onto a smaller machine.
-
What is the maximum concurrent user count per server for our model at our target response time?The divisor in the server-count arithmetic below. Get it in writing before you buy the second server.
-
Will you run the benchmark on one of our own machines and hand us the raw report?Every figure above, on your hardware. It outranks any table, including this one.
How Much RAM a Local Model Needs: The Ladder
Memory is the rung system. Each step up in parameter count demands a step up in memory, and the guidance arrives one rung at a time rather than as a curve. Read the rungs with their hedging intact, because the hedging is the useful part.
- 1B class. Runs on nearly anything, with 2023-vintage hardware already marginal.
- 3B class on 16GB. A customer, not Iternal, put it this way: 16GB is not great but is enough to run a 3B model. Not great but enough is the whole claim.
- 32GB is the sweet spot. A second account gives 32GB with an i7 as good for the software, though the graphics setup still matters — carry that clause along: a memory number alone does not qualify a machine.
- 12B class on 32GB. May not give a good experience.
- 26B, 27B and 31B class on 16GB of VRAM. Not runnable, except at very small quantizations.
Two consequences follow. Most users never push a workload past a 3B model, which makes an entry AI PC a real answer rather than a consolation. And defaulting a whole fleet down to a 1B model to make it fit leaves people with a noticeably weaker assistant — fit is solved by the machine or by a server, not by shrinking everyone. Very small quantization squeezes a large model onto a small device and trades accuracy for the fit; what it costs on your device class is a question to put in writing. Panther Lake systems are themselves capped near 32GB, and RAM is expensive.
Disk footprint: what the install actually takes
Storage is the easy part. AirgapAI plus a 3B model comes to 2.3GB; pointed at a server model with nothing local, about one and a half gigabytes. Average consumption lands near three and a half gigabytes per user once a model is in place, and top-tier Intel devices carry roughly ten gigabytes of on-disk capacity for models. The installer arrives as about a 4GB zip. What constrains a heavy user is the device hard drive, which bounds how much data can be worked with locally.
Do You Need an NPU, and Which Chips Qualify?
The hardware half is simple. An AI PC means the device carries an NPU, a GPU, or both. Intel sorts qualification by tier rather than generation, and the classifications that matter are Core Ultra 7 and Core Ultra 5, plus i7 and i5. Meteor Lake, Lunar Lake and Panther Lake all include an NPU, and every PC Dell ships now carries one, which makes the current catalog technically all AI PCs. Anything in the Copilot+ classification works, and higher-tier chipsets perform better.
An NPU is not a gate, and the TOPS rating is not a threshold. The stated figures put Panther Lake and Lunar Lake NPUs at 50 TOPS, with lower-end parts at 12 to 16. AirgapAI runs hybrid across CPU and GPU, so the rating does not gate it. The integrated GPU is always faster than the NPU though it draws more battery; the NPU loads slower and caps model size while being far more power-efficient once loaded. On Meteor Lake the workload lands on the integrated GPU instead: slower, and it works. The NPU earns its place on voice and transcription, where it does a phenomenal job, and on battery life.
The software half is where an unused NPU usually comes from. Buyers describe not knowing which applications in their stack touch the NPU at all, and the cause is rarely a missing part. Each NPU needs its own compiler, so models must be recompiled per processor before that silicon can be used. Iternal converted its model set for OpenVINO optimization precisely so the same models run on NPU, GPU or CPU, though coverage is not universal: vision language models have no NPU support yet. AirgapAI hides NPU-specific models when the machine has no NPU, which doubles as a fast diagnostic, and Iternal published an open-source npu-detect package because CPU and GPU statistics can be read from the system while NPU information cannot.
Hot start: the model loads once and stays
Hot start is the default. The model loads into memory when the application launches and stays resident until it closes, which avoids the NPU cold start and is why answers come back quickly rather than after a reload. The cost is paid at the front, and the NPU loads slower than the GPU. For sizing, that makes resident memory rather than peak memory the number to plan against: the model holds its footprint for the whole session.
Sizing Past One Machine: Four Inputs and One Formula
Above a single laptop the exercise stops being about silicon and becomes arithmetic. Four inputs drive it, and Iternal maintains a calculator you feed them into to get a server count back.
- Concurrent users. Real people at the same moment, not a share of the virtual CPUs. Given benchmark data for a cohort of 64 users, the parallel-threads input is 64.
- Model size. Parameter count and precision, because a larger model produces a lower tokens-per-second figure. Paired with it: how many tokens per second the chosen chipset can produce.
- Documents per ingestion run. Iternal sizes on pages of new intellectual property created each day, and server-side guidance follows pages ingested per month rather than headcount. The Blockify sizing model assumes 2,400 input characters per page.
- The acceptable processing window. The use case sets the service level in the sizing model. Overnight and interactive are different machines.
servers = required fleet concurrency ÷ max concurrent users per server, rounded up
Concurrent thread capacity is assumed to scale linearly with fleet size, which is what makes the division legitimate. Start with one server and daisy chain more later.
The divisor is the number worth arguing about. Concurrent users on a shared deployment are limited only by the power of the server and the size of the language model — the architecture holds no seat ceiling. One benchmark configuration carried a maximum of 56 concurrent users per server; another was run at 64. Degradation arrives as queueing rather than failure: the wait for a first token depends on free server slots and the requests sitting ahead of yours.
Benchmark one machine, then divide. A benchmark on a single Xeon lab machine shows quickly how many servers a fleet would need. Stand up one representative server, load the model you will actually run at the precision you will actually run it, drive it at your target concurrency, and read tokens per second and the wait for a first token. Benchmarks do not extrapolate across configurations, so measure the one you intend to buy.
Where the published guidance stops. Iternal states this openly: published sizing guidance is general, and specific configurations go back to the engineering team. The server spreadsheet covers a single model without stating its precision, and the sizing tool addresses data center configurations rather than PCs. Treat the calculator as a first pass and the benchmark as the answer.
When the Only Design Artifact Is a Slide
A deck is not a design. Buyers describe the failure in almost those words: bids arrive as a PowerPoint with nothing behind it, and customers cannot say how much infrastructure they need because they do not know how much data they will process. Projects then fail because the implementation was never properly designed. Iternal answers it by defining an architecture that names the components a customer must have in place, then laying the software on top. Every line below is a decision a design document has to record.
Reference architecture at a glance
- Device classes and counts. Which row above each user population sits in.
- Model per class, with precision. Parameter count and quantization named, not implied.
- Where inference runs. On the device, on a shared server inside your boundary, or a device that can reach one.
- Where the data set is built and lives. Ingestion volume per period, and the store holding metadata and generated vectors.
- Throughput target and processing window. The band to plan against and the service level the use case demands.
- Concurrency per server and the resulting count. Divisor, dividend, rounding.
- Storage per user and per server. Application, model, data set.
- The benchmark the numbers rest on. Which machine, model, precision and concurrency.
Fill those eight lines and the rough order of magnitude falls out. Leave one blank and it is still a slide.
Qualifying the Machines You Already Own
The test takes an afternoon and needs no spreadsheet. Install AirgapAI on one machine from each standard build in your estate and read what the application tells you. The model screen runs a hardware check and only surfaces the models that machine can actually run, and NPU-specific models stay hidden where there is no NPU. AirgapAI then prompts for a benchmark at setup and each time a new model is added: the run scans the machine, confirms it is taking advantage of a GPU when one exists, and reports the hardware, average tokens per second and prefill time. Whichever models the app offers on a build are the models that build supports.
When a build does not qualify, the model can move instead of the machine. AirgapAI supports a client-server architecture in which the language models, and optionally the data sets, sit on a server inside the secure location while laptops act as a terminal interface. With no model on the device the local install drops to about one and a half gigabytes, and the assistant gains capabilities it lacks standalone. AirgapAI connects only to localhost by default, so reaching a server-hosted model takes a relay on the device. Where a machine can carry only a model smaller than the catalog Llama model, accuracy tends to fall short, and the client-server route is the answer.
Which route to take across a whole estate is a separate decision from the specification each route implies. For more information visit the deployment pillar.
Answered elsewhere
- How throughput is measured and what counts as usable output — see the speed page.
- Choosing between the device, a server inside your boundary, and a managed environment — see the placement page.
- Which operating systems, virtual desktops and container platforms it installs on — see the environments page.
- Packaging and imaging the software for a managed estate at scale — see the rollout page.
- Which language model to run once the hardware qualifies — see the model selection section.
- What the hardware costs against a per-user subscription — see the cost comparison page.
FAQ
FAQ: Hardware, Memory and Sizing
The answer comes as rungs, not a curve. A 1B-class model runs on nearly anything. A customer described 16GB as not great but enough to run a 3B model. 32GB has been called the sweet spot, although a 12B model may not give a good experience there. The 26B, 27B and 31B class will not run on 16GB of VRAM except at very small quantizations, which is where workstation hardware comes in.
Not strictly. AirgapAI runs on CPU, integrated GPU or NPU, and it runs fine on a CPU-only home laptop — simply slower, which is what starts the upgrade conversation. Iternal recommends an AI PC carrying an NPU and a GPU, because the software is optimized for Intel silicon through OpenVINO. The NPU TOPS rating is not a threshold: AirgapAI runs hybrid across CPU and GPU.
2024 or newer is the recommendation, on Intel Meteor Lake or better. Read it as a ladder. Hardware from 2023 or newer is pretty much certain to work and anything older is iffy, which sets the floor. Any Dell PC from 2025 or newer should give a good end-user experience. So 2023 is where it starts working, 2024 is what to buy, and 2025 or later stops being a compromise.
Other PC manufacturers on the marketplace work fine — there is no hardware-brand tie. AirgapAI is optimized for Intel silicon through OpenVINO, so it runs on CPU, GPU and NPU, and Intel integrated NPU and GPU machines perform best. On the server side Iternal has proposed Blockify and AirgapAI together as a Xeon-based solution, with inference on both Xeon and Gaudi.
As many as the power of the server and the size of the language model allow — the architecture sets no seat ceiling. One benchmark configuration carried a maximum of 56 concurrent users per server, and another was run at 64. Degradation arrives as queueing rather than failure: the wait for a first token depends on how many server slots are free and how many requests sit ahead of yours.
Divide the required fleet concurrency by the maximum concurrent users per server and round up; thread capacity is assumed to scale linearly with fleet size. The four inputs are concurrent users, model size and precision, pages ingested per period, and the processing window the use case demands. A benchmark on one Xeon lab machine reveals the count quickly.
AI infrastructure for an on-premises deployment is the compute, memory, storage, network and serving software a model needs to answer a question, plus the data set it retrieves from and the management layer around it. On-premises the same seven layers apply, with one difference: an accelerator you own sits at the bottom and no step in the path sends a request outside your boundary.
Infrastructure is AI-ready when every layer has a number behind it rather than a piece of equipment: a device class per user population, a model and precision per class, storage per user, ingestion volume per period, a throughput target, concurrency per server, and the one benchmark the rest of the set rests on. Fill those in and the rough order of magnitude falls out; leave one blank and the design is still a slide.
Measure One Machine, Then Multiply
Sizing arguments end the moment somebody measures something. Install on one machine from each standard build, read which models the application offers, run the benchmark, and write the numbers down. Do the same on one server at your target concurrency. Those two measurements plus the four inputs above give you a server count you can defend in a budget meeting.