Selection Guide for Private and On-Premise RAG

Embedding Models Best Embedding Models Open-Source Embedding Models Text Embedding Model Vector Embeddings

Embedding Models Compared: How to Choose One for Private RAG

The embedding model decides what your AI can find. This guide covers what one does, the seven criteria that separate them, and how the open-source options compare on dimension, context length, licence and local inference.

By John Byron Hanby IV

CEO & Founder, Iternal Technologies • Published September 2026 • 14 min read

See Local Embedding in AirgapAI

What is an embedding model?

An embedding model is a neural model that converts text, images or audio into a fixed-length vector of numbers — an embedding — so that items with similar meaning sit close together. In retrieval and RAG it decides what gets found; you choose one by dimension, context length, language coverage, licence and where it can run.

Everything downstream inherits that choice: the retriever ranks only what the embedding model brought close together, and the language model answers only from what the retriever returned.

What the published numbers say

Datasets in the original MTEB embedding benchmark, spanning 8 task types and 112 languages (Muennighoff et al., 2022)

14x

Smaller embeddings at comparable accuracy using Matryoshka representation learning (Kusupati et al., NeurIPS 2022)

$12.9M

Average annual cost of poor data quality per organization (Gartner) — the input an embedding model cannot fix

2.29X

More accurate vector search from distilled IdeaBlocks vs. naive chunking of the same documents (Blockify Performance Analysis)

Benchmark scope per the MTEB paper (Muennighoff et al., 2022); truncation economics per the Matryoshka Representation Learning paper; data-quality cost per Gartner; vector-search gain from Iternal’s Blockify benchmark data.

On this page

What an embedding model does
Embeddings vs. the model that makes them
How to choose: seven criteria
Open-source options compared
Where the embedding model runs
Evaluating on your own data
Why the corpus outranks the model
Embedding in AirgapAI and Blockify
Frequently asked questions

What an Embedding Model Does

A text embedding model places meaning in space. Feed it a sentence and it returns a vector — a fixed-length list of numbers, commonly 384 to 4,096 of them — whose position encodes what the sentence is about. Two passages that say the same thing in different words land near each other; two that share vocabulary but not meaning do not. Semantic search, deduplication, clustering and RAG all rest on that property.

The training objective produces it. Models are trained contrastively — shown pairs that should be close, such as a question and the passage answering it, and pairs that should be far apart — so distance approximates relatedness. The embedding model is therefore not interchangeable with the language model it feeds: one generates, the other only measures.

Split the source content

Documents are divided into passages small enough for one vector to represent faithfully. Too coarse and a vector averages three unrelated ideas; too fine and the passage loses its context.

Encode each passage

The model tokenizes the passage, runs one forward pass and pools the result into a fixed-length vector. The same model and version must encode every passage in the corpus.

Index the vectors

Vectors are written to an index built for nearest-neighbour search, alongside the original text and metadata so a hit resolves to a citable source.

Encode the query, then rank

At query time the question is encoded by the same model and compared by cosine similarity. The closest passages become the context the language model may answer from.

Queries and passages are not the same input

Retrieval is asymmetric: a nine-word question must be matched against a 200-word passage. Most current models handle that with instruction prefixes — E5 expects query: and passage:, Nomic expects search_query: and search_document:, and the Qwen3 family takes a task instruction. Omitting the prefixes the model card specifies is a common cause of retrieval that tests well in a notebook and disappoints in production (Source: sentence-transformers documentation).

The stakes are those of enterprise search at AI scale: McKinsey puts adoption at 78% of organizations using AI in at least one business function, and Forrester estimates 60% to 73% of enterprise data goes unused for analytics. The embedding model decides which slice an assistant can reach (Sources: McKinsey, The State of AI; Forrester).

Embeddings vs. the Model That Makes Them

The two terms are used interchangeably in product marketing and should not be. Vector embeddings are the artefacts: lists of numbers, stored and searched, cheap to produce one at a time. The embedding model is the function that produced them, and it is the decision with consequences — every vector in the index is meaningful only relative to the model and version that created it.

The embedding model vs. the vector embeddings it produces
Facet	The embedding model	A vector embedding
What it is	A trained network you download, licence or call	The numeric output for one passage or query
How many you have	One per corpus, pinned to a specific version	One per passage — millions in a mature index
Where it lives	On a CPU, NPU or GPU, in the inference runtime	In an index, beside the source text and metadata
What changing it costs	A full re-embed of the corpus and an index rebuild	Nothing on its own — a single passage can be re-encoded
What it is compared against	Other models, on benchmarks and your evaluation set	Other embeddings from the same model, by cosine similarity

That last row is the one teams learn late. Vectors from two models do not share a coordinate system, so there is no partial migration and no A/B test on a live index: switching means reprocessing every document, rebuilding the index and re-validating retrieval before cut-over. Treat the choice like a database engine, not like a prompt.

Where the vectors are stored

Storage is a separate component. Once passages are encoded, the vectors need somewhere that holds millions of them and returns the nearest neighbours in milliseconds — typically an HNSW or IVF index, standalone or as an extension to a database you already run. That layer and its trade-offs are covered in the companion guide on what a vector database is.

How to Choose an Embedding Model: Seven Criteria

Benchmarks narrow the field; they do not make the decision. The Massive Text Embedding Benchmark is the right first filter — the original release spanned 8 task types, 58 datasets and 112 languages (Source: MTEB, Muennighoff et al. (2022)) — but a benchmark average describes somebody else’s corpus. These seven criteria decide whether a model survives contact with yours.

Seven selection criteria, what to check, and how each one fails in production
#	Criterion	What to check	Failure mode if you skip it
1	Retrieval quality on your material	Start from public MTEB scores for the languages you serve, then re-rank on your own labelled queries.	A leaderboard winner trained on web text loses to a mid-table model on your contracts or clinical notes.
2	Dimension	Read the native dimension and whether the model supports Matryoshka truncation to a shorter vector.	A 3,072-dimension default multiplies index size and query cost for accuracy you could have had at 768.
3	Maximum context length	Compare the limit against your real passage length — 512 tokens is roughly 380 English words.	Passages are silently truncated at encode time, so half of every long clause is invisible.
4	Language and domain coverage	Confirm the training languages and look for domain-adapted variants for code, legal or biomedical text.	Cross-lingual queries return the English document every time; code search matches comments, not behaviour.
5	Licence and provenance	Read the licence on the model card: Apache 2.0 and MIT are permissive; some strong models are non-commercial.	A model shipped in a product turns out to be non-commercial, and the corpus is re-embedded under deadline.
6	Inference footprint	Size the model against the hardware that will run it, then measure passages per second at batch size.	Initial ingestion takes weeks, or per-query encoding adds latency the assistant cannot absorb.
7	Stability and exit cost	Pin a version, record it with the index, and confirm you can run those weights for the life of the corpus.	A hosted endpoint is deprecated, and re-embedding becomes an unplanned migration.

Dimension is a cost decision, not a quality ranking

More dimensions do not mean better retrieval. Dimension sets how much memory each vector occupies and how much arithmetic every query performs; quality comes from training data and objective. Matryoshka representation learning trains a model so the leading portion of its vector is itself usable — the paper reports up to 14x smaller embeddings at comparable accuracy (Source: Kusupati et al., NeurIPS 2022). The Nomic, Arctic, Jina, Qwen3 and EmbeddingGemma families let you cut a 1,024-dimension vector to 256 and measure the saving against the recall cost.

Context length has to match your passages, not your documents

An 8,192-token window is not an instruction to embed 8,192-token passages. One vector is one point in space, so a passage covering four topics produces an average that matches none of them. Size passages to a single idea, then confirm the model’s limit exceeds the longest. Long-context capability earns its keep on policies and contracts, where a self-contained clause runs to a thousand words. Current scores sit on the public MTEB leaderboard.

Open-Source Embedding Models Compared

Open-weight models dominate private deployments for one reason: you can hold the weights. The eight families below cover the range most enterprise teams shortlist. Read the licence column as carefully as the quality column — it is the field that most often invalidates an otherwise finished decision.

Parameters, dimension, context length, licence and language coverage as published on each model’s Hugging Face model card
Model	Params	Dimensions	Max context	Licence	Languages	Where it fits
Qwen3-Embedding-0.6B	0.6B	1,024 (Matryoshka from 32)	32,768 tokens	Apache 2.0	100+	Long passages and instruction-aware retrieval; 4B and 8B siblings trade footprint for quality
BAAI BGE-M3	~570M	1,024	8,192 tokens	MIT	100+	Hybrid retrieval — dense, sparse and multi-vector representations from one pass
Snowflake Arctic Embed L v2.0	~570M	1,024 (truncatable to 256)	8,192 tokens	Apache 2.0	Multilingual	Multilingual retrieval where index size matters and truncation must stay safe
Nomic Embed Text v1.5	137M	768 (Matryoshka from 64)	8,192 tokens	Apache 2.0	English-first	Long-context English retrieval on modest hardware; open training data
multilingual-E5-large	560M	1,024	512 tokens	MIT	~100	A dependable multilingual default for short passages; needs query:/passage: prefixes
Jina Embeddings v3	570M	1,024 (Matryoshka from 32)	8,192 tokens	CC BY-NC 4.0 — commercial use needs a separate licence	89	Task-specific adapters for retrieval and classification — check the licence first
EmbeddingGemma-300M	308M	768 (Matryoshka to 512/256/128)	2,048 tokens	Gemma Terms of Use	100+	On-device and laptop deployment where memory is the binding constraint
all-MiniLM-L6-v2	22.7M	384	256 tokens	Apache 2.0	English	Prototypes and CPU-only pipelines that need throughput above nuance

Rank order is perishable as new checkpoints land; the structural columns are durable. For current scores by task and language, consult the MTEB leaderboard directly.

Best for multilingual corpora

BGE-M3, Arctic Embed 2.0, multilingual-E5 and the Qwen3 family train across 89 to 100+ languages, so a query in one language retrieves a document written in another.

Best for on-device work

EmbeddingGemma at 308M parameters and all-MiniLM at 22.7M are sized for laptops and NPUs, where nothing leaves the machine.

Best for long structured documents

Qwen3-Embedding at 32k tokens and the 8k-context families take self-contained clauses in policies and contracts without truncation.

Best for unrestricted commercial use

Apache 2.0 and MIT models — Qwen3-Embedding, BGE-M3, Arctic Embed, Nomic, E5, MiniLM — ship inside a product without a separate agreement.

Managed embedding services, and when they fit

Hosted endpoints remain the fastest path when the data is already in the cloud. OpenAI’s text-embedding-3 models support dimension reduction through the API, Amazon Titan Text Embeddings V2 on AWS Bedrock offers selectable 1,024, 512 and 256-dimension outputs inside an existing AWS boundary, and Cohere Embed is well established for multilingual search. NVIDIA takes the middle path with NeMo Retriever embedding microservices, packaged for self-hosting on your own GPUs.

The dividing line is not quality; it is where the text goes at encode time. A managed embedding API sends every passage and every query outside your boundary — workable under an appropriate agreement, disqualifying inside a closed network.

Where the Embedding Model Runs: Local, On-Premise and Air-Gapped

Embedding models are small. The largest one most teams deploy is a fraction of the language model it feeds, and the useful range starts at 22 million parameters, so placement is open in a way it rarely is for generation: a model of a few hundred megabytes runs on the laptop holding the documents, on a server in your rack, or inside a facility with no route to the internet.

One rule admits no exception. In a private or air-gapped design, the embedding model must run inside the same boundary as the data: a pipeline that calls a remote embedding API sends every passage and every user question outside the perimeter at the exact moment the architecture claims nothing leaves. It is the most common way a closed deployment is broken by accident. The pre-staging procedure and verification steps are set out in the guide on how to deploy an LLM on-premise, and sizing is covered in how to run an LLM locally.

Laptop CPU or NPU

20M – 350M parameters

Field and personal deployments where the corpus runs to thousands of passages and nothing may leave the device. Intel Core Ultra NPUs and Apple silicon handle this class.

Workstation or single server GPU

350M – 1B parameters

The common enterprise case. One mid-range GPU embeds millions of passages overnight and encodes queries in single-digit milliseconds.

On-premise server fleet

1B – 8B parameters

Large corpora, continuous ingestion, or a bigger model justified by measured gains. Xeon-class CPU inference works for batch ingestion when GPUs serve generation.

Air-gapped enclave

Any size, pre-staged

Weights, tokenizer, runtime and index are transferred across the gap and pinned. No first-run download, no telemetry, no registry call.

The hardware picture in 2026

Client silicon gained dedicated inference hardware, so an AI PC with an Intel Core Ultra NPU can encode a document set without touching the GPU; Intel’s OpenVINO toolkit exposes that path for common sentence-transformer architectures. Dell’s workstation and PowerEdge lines put the same capability in the rack, and NVIDIA’s NeMo Retriever packages embedding and reranking as self-hosted microservices for GPU estates already running generation.

Two operational notes follow. Ingestion is throughput-bound and query encoding is latency-bound, so measure both. And embedding competes with generation for the same accelerator: pinning the embedding model to CPU or NPU while the language model holds the GPU often decides whether the assistant stalls during ingestion.

Evaluating Embedding Models on Your Own Data

A shortlist of three models can be settled in two weeks by one engineer, and it pays for itself the first time it rules out a model that looked obvious. The sequence measures retrieval and cost only, and keeps answer quality — which depends on the language model and the prompt — out of the comparison.

Build a labelled query set

Collect 50 to 100 real questions from the people who will use the system, and record the passage that answers each one. The artefact outlives every model you test.

Fix everything except the model

Freeze the splitting rules, index parameters and retrieval depth. If chunking changes between runs, you are measuring the chunker, not the encoder.

Measure retrieval, not answers

Score recall@10 and mean reciprocal rank against the labelled set. If the right passage was never retrieved, no prompt engineering recovers it.

Sweep the dimension

For a Matryoshka-trained model, re-run at full dimension and at each truncation. Where recall starts to fall is where the index saving stops being free.

Time both paths

Record passages per second at production batch size, and single-query latency on the hardware you will deploy. One sets the migration window; the other is felt by every user.

Cost the exit before you enter

Estimate what a full re-embed costs in compute and elapsed time. That is the price of changing your mind, and it belongs in the decision record.

Record the winner, the version string and the scores alongside the index. When someone proposes an upgrade eighteen months from now, the labelled query set makes that a measurable question rather than an argument.

Why the Corpus Outranks the Model

Teams arrive here because retrieval is disappointing and a better embedding model looks like the fix. Usually it is not. Moving from a mid-table model to a leaderboard leader buys a few points of recall; cleaning the material being embedded routinely buys more, because the failures below are properties of the corpus that no encoder can see past.

Duplicate passages crowd the results

The same policy paragraph living in nine documents produces nine near-identical vectors. A top-3 retrieval returns three copies of one idea and drops the two that would have completed the answer.

Contradictions retrieve equally well

Superseded guidance sits as close to the query as current guidance, because similarity measures aboutness, not correctness. No model can rank truth it was never told about.

Boilerplate dilutes the signal

Headers, footers, legal preambles and navigation text pull a passage vector toward the average of the document set, which is precisely where nothing is distinctive.

The underlying problem is well documented: IDC puts average enterprise data duplication between 8:1 and 22:1, and Gartner estimates the cost of poor data quality at $12.9 million per organization per year. Both describe the input side of a retrieval system, unaffected by which model encodes it.

Iternal built Blockify for this step: it restructures raw enterprise documents into deduplicated, self-contained IdeaBlocks before anything is embedded. On the published benchmark, IdeaBlocks return 2.29X more accurate vector search than naive 1,000-character chunking of the same material — a 56.26% precision gain — while averaging roughly 98 tokens per block against about 303 for a naive chunk, a 3.09X token reduction per query. The method and the caveats on the aggregate figures are published in the Blockify benchmark data.

Because it runs ahead of the encode step this is not a competing choice to the comparison above: it works with any embedding model, open-source or managed, and compounds with a good one. The same principle drives the economics in the guide on reducing AI token costs — how the data is structured matters more than which encoder processes it.

Embedding in AirgapAI and Blockify

Iternal builds both halves of this decision into products, because both have to hold for a private deployment to work: the encode step happens where the data already is, and the material encoded is worth retrieving.

The encode step stays on the device

AirgapAI runs entirely on your own Windows or macOS machine — no cloud services, no internet connection, no data transmission — so indexed passages never reach an external endpoint.

You choose the embedding model

Creating a dataset is a four-step flow: create a corpus, name it, pick an embedding model, and choose the files to process. This guide is written for that step.

Blockify prepares what gets embedded

Blockify distils source documents into deduplicated IdeaBlocks first, so vectors represent distinct ideas instead of nine copies of one paragraph.

Model-agnostic by design

Blockify output is compatible with different embedding models — OpenAI, AWS Bedrock, Mistral or open-source alternatives — including a local embedding option for offline-only environments.

Together they make local retrieval practical on ordinary hardware: a distilled corpus is smaller, so the index fits on a laptop, and a locally selected embedding model keeps every passage on the machine. For more information visit the AirgapAI and Blockify pages.

Choosing Well, Then Moving On

A good embedding-model decision takes two weeks and holds for years: filter on licence, context length, language coverage and the hardware you can run it on; score the survivors against your own labelled queries; record the version with the index; price the re-embed before you commit.

Then turn to the input. The corpus is where the remaining accuracy lives: deduplicated, current, self-contained passages do more for retrieval than further tuning of the encoder.

AirgapAI

Run retrieval where the documents already are.

AirgapAI is a 100% local AI assistant: models, workflows and your documents stay on your laptop, with a dataset flow that lets you create a corpus and choose the embedding model that processes it.

Explore AirgapAI

Blockify

Fix the corpus, and every model gets better.

Blockify distils enterprise documents into deduplicated IdeaBlocks before they are embedded — 2.29X more accurate vector search and 3.09X fewer tokens per query on the published benchmark.

See How Blockify Works

Frequently Asked Questions

What is an embedding model?

An embedding model is a neural network that converts text, images or audio into a fixed-length list of numbers — an embedding — positioned so that items with similar meaning sit close together. Retrieval systems compare those positions to answer a query by meaning rather than by keyword match, which is what makes semantic search and RAG possible.

What is the difference between an embedding and an embedding model?

The model is the thing you choose and run; the embedding is what it produces. Vector embeddings are the numeric outputs stored and searched at query time, and they are only comparable to other embeddings from the same model and version. Swap the model and every stored vector becomes meaningless until the corpus is re-embedded.

What are the best open-source embedding models?

The strongest general-purpose open-weight families are Qwen3-Embedding, BAAI BGE-M3, Snowflake Arctic Embed 2.0, Nomic Embed Text v1.5, multilingual-E5 and, for on-device work, EmbeddingGemma. Which is best is workload-specific: check the MTEB leaderboard for quality, then filter on licence, context length and the hardware you can run it on.

Do more dimensions make an embedding model better?

No. Dimension sets storage and search cost, not quality: a 1,024-dimension model can beat a 3,072-dimension one on the same corpus. Several current models are trained with Matryoshka representation learning, so a long vector can be truncated to a shorter one with a modest quality cost, which is the practical way to trade index size against recall.

Can an embedding model run locally or fully offline?

Yes, and for private deployments it must. Embedding models are far smaller than the language models they support — most run between 20 million and 8 billion parameters — so a CPU, an NPU-equipped laptop or a single server GPU is enough. In an air-gapped design it has to be pre-staged inside the enclave, because a remote embedding API call silently breaks the gap.

What happens if I change embedding models later?

You re-embed everything. Vectors from two models are not comparable, so a change means reprocessing the corpus, rebuilding the index and re-validating retrieval against your evaluation set. Budget the compute and the cut-over window before you pick, and prefer a permissive licence and a pinned version.

Does a better embedding model fix bad retrieval accuracy?

Rarely. Retrieval quality is capped by the corpus: duplicated, contradictory and boilerplate-heavy content produces near-identical vectors that crowd the results whichever model embedded them. Cleaning the corpus first moves accuracy further than a model upgrade — Iternal measured 2.29X more accurate vector search from Blockify-distilled content against naive chunking of the same documents.

About the Author

John Byron Hanby IV

CEO & Founder, Iternal Technologies

John Byron Hanby IV is the founder and CEO of Iternal Technologies, a leading AI platform and consulting firm. He is the author of The AI Strategy Blueprint and The AI Partner Blueprint, the definitive playbooks for enterprise AI transformation and channel go-to-market. He advises Fortune 500 executives, federal agencies, and the world's largest systems integrators on AI strategy, governance, and deployment.

G Grokipedia LinkedIn X Leadership Team