How to Deploy an LLM On-Premise in 2026

On-Premise LLM Deployment in 2026: What This Guide Covers

Deploying a large language model on-premise means running the full inference stack — model weights, serving engine, and API — on hardware you own and control, inside your own network boundary, with no dependency on a third-party API. For enterprises in regulated industries, that control is increasingly non-negotiable: by 2025, roughly 71% of AI infrastructure ran outside the public cloud, a shift driven heavily by financial-services data-residency requirements and the arrival of enforceable AI regulation.

The good news for platform teams is that the open-weight model ecosystem has matured to the point where self-hosted models rival frontier hosted APIs on most enterprise tasks, and the serving software — vLLM, NVIDIA NIM, SGLang, TensorRT-LLM — is production-hardened. The hard part is no longer "can we run it" but "how do we size it correctly and operate it reliably." This guide walks the full decision path in order: when on-prem makes sense, how to choose a model, how to compute exact VRAM requirements, how to select GPUs, how to pick and configure a serving stack, how to scale across GPUs and nodes, how to quantize, how to plan capacity from real demand, how to deploy in an air-gapped enclave, what it actually costs versus cloud, and how to run it in production.

A note on numbers: hardware specs and formulas in this guide are stable, but model versions and software defaults drift monthly. Where we name models, we use families and tiers rather than chasing point releases; where benchmarks are version-specific, we say so. For deeper dives, see our companion Hardware Sizing Guide, LLM Selection Guide, and Best AI Tools for Air-Gapped Environments.

Step 1: Choose the Right Open-Weight Model

License first: what you can legally productize

For enterprise on-prem, the license gates everything — pick the model your legal team can clear before you benchmark quality. Three tiers matter:

MoE vs dense: the distinction that decides serveability

Mixture-of-Experts (MoE) models — gpt-oss, Qwen 3.5 / 3.6, DeepSeek V3.2 / V4, Kimi K2 Thinking, GLM-5, MiniMax M2.7, Gemma 4 26B-A4B, Llama 4 — activate only a fraction of their parameters per token, which lowers per-token compute and raises throughput. But the critical sizing insight is that VRAM must hold all total parameters (every expert is resident in GPU memory), while only the active parameters drive compute. A 671B MoE with 37B active still needs roughly 700 GB at FP8, and the 2026 frontier open-weights (DeepSeek V4, Kimi K2, GLM-5) are 700B–1.6T total — they require multi-GPU and usually multi-node serving (see Step 6: Scaling). Dense models (Qwen3.6-27B, Phi-4, Gemma 3, Mistral Small) are simpler to serve, have more predictable latency, and are easier to fully fit and quantize on a single GPU — which is why they are often the better choice for constrained single-node on-prem. MoE models are also especially CPU-friendly because so few parameters activate per token — see Option B: Run Inference on Intel Xeon CPUs.

Table 1 — Open-Weight Model Landscape for Enterprise On-Prem (May 2026)

Total/active parameter counts marked with "~" are approximate where providers have not published exact figures for the newest frontier releases — verify specs and license terms against the model card before sizing. Benchmark scores and rankings for these models are tracked live on our LLM Benchmark Repository.

Table 1b — Full Variant Lineups: Qwen 3.5, Qwen 3.6 & Gemma 4 (May 2026)

These three families ship a complete size ladder from sub-1B edge models to 397B-parameter MoE flagships, all under permissive licenses — making them the most common starting point for on-prem standardization. The small Gemma 4 and Qwen variants are also the best fit for Intel Xeon CPU inference (Option B); the Gemma 4 26B-A4B MoE is the exact model benchmarked there.

Qwen "Plus" / "Max" tiers (e.g. Qwen3.5-Plus, Qwen 3.7 Max) are hosted, closed-weight Alibaba Cloud endpoints and are not deployable on-prem — only the numbered open-weight variants above ship downloadable weights. Gemma 4 ships under the permissive Apache 2.0 license — a notable change from the use-restricted custom Gemma license used through Gemma 3. Gemma 4 "E" sizes (E2B / E4B) use effective-parameter counts (per-layer embeddings / MatFormer), so on-disk size differs from the effective figure.

Model selection by use case

Table 2 — Use-Case Model Selection (on-prem)

For RAG specifically, model choice is only half the equation — retrieval quality dominates grounded accuracy. Pair a dense long-context model with a disciplined ingestion pipeline; see Blockify Data Ingestion for how to structure source data before it reaches the model. For a fuller decision tree across every family, see the LLM Selection Guide.

Step 3: Select Your GPUs (H100 / H200 / A100 / L40S / Blackwell / RTX)

The two axes that decide inference: capacity and bandwidth

Two GPU properties govern LLM serving. VRAM capacity gates which model and context length fit at all. Memory bandwidth governs decode/token-generation latency, because decode is memory-bandwidth-bound: every new token streams all model weights from HBM once per forward pass. This is why the H200 — which has compute identical to the H100 but 43% more bandwidth (4.8 TB/s vs 3.35 TB/s) — generates tokens roughly 43% faster in the small-batch (memory-bound) regime, despite no compute uplift.

Data-center and workstation GPU comparison

Table 5 — NVIDIA Data-Center / Pro GPU Specs for LLM Inference (2025–2026)

Organize procurement by generation, because precision support tracks it: Ampere (A100) tops out at INT8/FP16 with no FP8; Hopper (H100/H200) adds FP8; Blackwell (B200/GB200, RTX PRO 6000) adds native FP4/NVFP4, which roughly doubles throughput and halves memory versus FP8 and is the 2025–2026 cost-per-token frontier. Note that L40S, L4, and the RTX cards have no NVLink — they scale only over PCIe, which makes them better suited to pipeline parallelism than tensor parallelism (see Step 6).

Table 6 — Approximate Model-Size Fit by VRAM (weights-only, +20–40% for KV/runtime)

Consumer GPUs (RTX 4090 / 5090): where they fit and where they stop

For 7B–13B single-GPU inference, consumer cards are genuinely competitive and cost a fraction of data-center GPUs — an RTX 4090 can match or beat an A100 on small models. The ceilings are firm: a single 24GB 4090 tops out around 32B at Q4; a 32GB 5090 fits 32B at Q8 and 70B only at aggressive Q2/Q3 with tiny context; comfortable 70B-Q4 needs dual GPUs (48GB combined). Neither consumer card has NVLink, so multi-GPU communication runs over PCIe, achieving roughly 85–90% of NVLink-linked throughput with about a 30% loss versus a monolithic 80GB card. For any model needing 40–80GB+ of VRAM there is no consumer alternative — data-center cards are required.

Table 7 — Consumer vs Data-Center Spec Comparison

Step 4: Pick a Serving Stack — vLLM vs NVIDIA NIM

The serving engine is what turns model weights into a production API. The two leading choices for on-prem are vLLM (open-source, maximum flexibility) and NVIDIA NIM (enterprise-packaged, vendor-supported). They share an OpenAI-compatible API surface, so application code rarely changes when you switch.

vLLM: the open-source default

vLLM is a high-throughput, memory-efficient inference engine originally from UC Berkeley (2023), built around two innovations:

• PagedAttention applies OS-style virtual-memory paging to the KV cache. Each sequence's KV cache is addressed through a logical block table mapping to non-contiguous physical blocks (default block size 16 tokens), eliminating the contiguous-allocation fragmentation that wasted 60–80% of KV memory in naive serving — reducing waste to under 4%.
• Continuous (in-flight) batching schedules at the per-token level: when a request finishes it immediately frees its KV blocks and the next queued request is admitted on the following step, keeping the GPU near 100% utilized. vLLM cites up to ~4x more tokens/sec versus naive Hugging Face generation.

You launch it with vllm serve <model>, which listens on 0.0.0.0:8000 and exposes /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, plus /health and a Prometheus /metrics endpoint. Connect with the standard OpenAI Python client by setting base_url='http://localhost:8000/v1'; require auth with --api-key or the VLLM_API_KEY environment variable.

Table 8 — Most-used vllm serve engine arguments

The current V1 engine (default since vLLM v0.8.0) is a core rewrite delivering up to 1.7x higher throughput than V0, with FlashAttention 3, piecewise CUDA graphs, and near-zero-overhead prefix caching (under 1% throughput drop even at a 0% cache-hit rate, so it is on by default).

NVIDIA NIM: the enterprise-packaged option

NVIDIA NIM (NVIDIA Inference Microservices) packages a model, an optimized inference engine, and an OpenAI-compatible API server into a single prebuilt Docker container that runs on NVIDIA GPUs anywhere. It auto-selects among TensorRT-LLM, vLLM, and SGLang backends and applies performance-tuned settings; the NIM LLM 2.0 line moved to a "one container, one backend" philosophy built on vLLM for predictable behavior. The default serving port is 8000, with native OpenAI endpoints plus a /metrics observability endpoint.

Prerequisites for the latest NIM LLM (2026): NVIDIA driver 580+ with CUDA 13.0+ (older NIMs accept CUDA 12.1+), Docker ≥ 19.03, and the NVIDIA Container Toolkit; the CUDA Toolkit does not need to be on the host, only the driver. A typical single-node run:

Table 9 — NIM offering tiers

Table 10 — NIM licensing / access tiers

The AI Enterprise list price (~$4,500/GPU/year) is a starting figure subject to volume and term discounts — confirm with NVIDIA sales.

vLLM vs NIM vs the rest of the field

The honest tradeoff: vLLM gives maximum flexibility, zero license cost, and the fastest access to new open models, at the price of you owning integration, hardening, and support. NIM gives a turnkey container with vendor SLAs, proactive security patching, and validated performance profiles, at the price of NVIDIA AI Enterprise licensing and tighter version coupling. Raw throughput between the top GPU engines is narrow — within roughly 15% — and flips by workload.

Table 11 — On-Prem LLM Serving Engine Comparison (2026)

Table 12 — Single H100 SXM5 80GB Benchmark, Llama-3.3-70B-Instruct FP8 (~512 in / ~256 out)

The decisive operational figure is the cold start: TensorRT-LLM's ~28-minute first-time engine compile (subsequent reloads ~90s) makes it painful for rapid model iteration, whereas vLLM and SGLang start in about a minute. A common, sound pattern is to develop and prototype on Ollama or llama.cpp, then serve production on vLLM or NIM. For the broader tool landscape, see Best Local AI Tools for Enterprise.

Step 6: Scale Across GPUs — Tensor, Pipeline, and Expert Parallelism

When a model exceeds one GPU, you shard it. There are three primary strategies, and matching them to your interconnect is what separates a fast cluster from a slow one.

Tensor parallelism (TP) shards each layer's weight matrices across GPUs (the Megatron column-parallel to row-parallel pattern), producing exactly two all-reduce collectives per transformer layer in the forward pass. Llama-3-70B's 80 layers means 160 all-reduce synchronization points per forward pass — so TP is bandwidth-bound and effectively requires NVLink/NVSwitch. On 4x L40 without NVLink, communication can exceed 50% of prefill cost.

Pipeline parallelism (PP) splits the model by layers across stages and only passes activations at stage boundaries, so it tolerates slower inter-node links (InfiniBand or even Ethernet) far better than TP. Expert parallelism (EP) shards MoE experts across GPUs, using all-to-all dispatch/combine; it pairs with data-parallel attention for large MoE models like DeepSeek-V3/R1.

The vLLM decision rule is clean: TP inside a node, PP across nodes, with tensor_parallel_size = GPUs per node and pipeline_parallel_size = number of nodes. For 2 nodes x 8 GPUs: --tensor-parallel-size 8 --pipeline-parallel-size 2. The critical exception: if GPUs lack NVLink (e.g. L40S) or the GPU count does not evenly divide the model, use pipeline parallelism instead of tensor parallelism.

Table 15 — vLLM Parallelism Strategy Selection

Table 16 — NVLink / NVSwitch bandwidth by GPU generation

Step 8: Manage the KV Cache and Long Context

The KV cache caches Keys and Values from prior tokens to avoid O(n²) recomputation each decode step, and at long context it is the primary memory bottleneck, frequently exceeding weight memory. Per-token cost = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element, scaled by tokens × batch.

Table 19 — KV Cache per-token cost and VRAM (Llama 3.1 70B, single sequence)

Every additional 1,000 tokens of context adds ~310 MB for a 70B-class model at BF16, and FP8 KV-cache quantization halves the footprint. Two techniques tame this:

• GQA/MQA shrink the cache by the ratio of query heads to KV heads. Llama 3.1 70B's 8 KV heads (versus 64 query heads) give an 8x reduction — which can mean 2 GPUs instead of 4 at 128K context.
• Automatic prefix caching (vLLM --enable-prefix-caching, on by default) hashes complete 16-token KV blocks (SHA-256) and reuses them across requests sharing a prefix — system prompts, tool definitions, few-shot examples — with LRU eviction and a cache_salt for multi-tenant isolation.

Table 20 — PagedAttention vs prior serving systems (vLLM paper)

Capacity Planning: A Worked Sizing Example, End to End

This is where the math becomes a purchase order. The flow: define demand to compute memory to compute per-token timing to convert to GPU count to apply SLO-driven utilization ceilings and headroom.

Worked GPU-count example

Suppose peak demand is 1,000 requests/sec, average service time 40 ms, target GPU utilization 70%. Per-H100 service rate = 0.70 / 0.040 = 17.5 RPS. GPU count = ceil(1000 / 17.5) = 60 H100 instances. But the SLO sets the utilization ceiling, because P99 TTFT degrades nonlinearly with concurrency:

Table 22 — P99 TTFT degradation vs concurrency (70B FP8 on H100 SXM5, 512-token prompts)

Table 23 — SLO target to max GPU utilization ceiling, and resulting fleet (1,000 RPS, 40ms service, H100 spot)

Total Cost of Ownership: On-Prem vs Cloud

The GPU sticker is only about 35% of five-year TCO — power, cooling, networking, redundancy, and staff make up the rest.

Table 26 — On-Prem GPU Server CAPEX (full system, Lenovo Press 2026, priced Jan 15 2026)

An 8x H100 server pulls ~10 kW at full load (~$10,500/yr electricity at $0.12/kWh), with cooling adding ~30%. Staff is typically the single largest line item, exceeding hardware depreciation over three years:

Table 27 — 3-Year TCO of One 8x H100 SXM5 Server (Spheron cost model, 2026)

Table 28 — Break-Even Time, On-Prem 8x H100 vs Azure (Lenovo 2026)

Table 29 — Per-Token Cost: On-Prem vs Cloud/API (Lenovo 2026)

Printable On-Prem LLM Requirements Checklist

Frequently Asked Questions

Related Resources