A retriever does not read your document. It measures distance, returns the fragments nearest your question, and trusts one of them holds the answer. Cut those fragments at an arbitrary character count and the trust is misplaced — the model, having nothing better, answers from what it was handed. Technical buyers work this out fast, and their question is rarely is the AI accurate. It is what does it retrieve, and what did the cut destroy?
Retrieval Architecture
How Does AI Retrieval Actually Work,
and Why Does Chunking Break Answers?
What a retriever hands the model, what a fixed character cut does to a table row and a numbered procedure, how large a local corpus gets, and where a plain split is still right.
Built from real buyer questions in our sales meetings
Direct Answer
Retrieval returns a fragment, and the fragment is only as good as the cut. Classic RAG slices documents at a fixed character count — commonly a thousand to two thousand characters, plus overlap — embeds each piece as a vector, and returns the pieces nearest your question. A table gets severed mid-row, so a price binds to the wrong product name; a procedure loses its order, so step four arrives without steps one to three. Iternal cuts differently: Blockify converts source content into IdeaBlocks, each holding a human-readable name, a critical question, a trusted answer and extracted entities, so the retrieved unit is already a self-contained answer.
The limit: restructuring does not make retrieval failure-proof. When the blocks a query pulls back do not match what was asked, Iternal has seen a local model repeat itself in a loop — corrected by moving generation flags off their defaults, but real. The step also rewrites: this version of Blockify may change some of the semantic wording while preserving the underlying message, so what returns is a restatement, not a quotation, and a reader who needs the original words clicks through to the source. AirgapAI also does not label a loaded data set as restructured or plainly split.
The device-local store is modest by Iternal's own account. On a laptop the vector store is a flat JSONL file queried by a WebAssembly component, chosen for the size of the machine, and Iternal says plainly that it holds because nobody runs anything heavy against it. Volume work belongs on a server. Get the ceilings in writing — the questions are below.
Mechanics and accuracy are two separate questions. Chunking, embedding, the retrieved unit, corpus scale and where the corpus lives are the mechanics. Why an answer goes wrong, and how a person proves one right, is the other one. For more information visit the accuracy and source-proof page.
The RAG Pipeline, Stage by Stage
A RAG pipeline runs in two halves. Build time ingests source documents, parses them, cuts them into units, distills those units, embeds each one as a vector and writes it to an index. Query time embeds the question, retrieves the nearest units, reranks them, generates an answer and cites the units it used.
The first six stages run once per document. They happen before anyone asks anything, which is why the decisions taken here are the expensive ones to change: the cut and the stored unit are fixed at ingestion, and the only way to revise them is to build the index again.
| Stage | What happens | What it decides |
|---|---|---|
| 01 Ingest | Source files are collected from a folder, a share or a connector and queued for processing. | Which formats the pipeline accepts at all. For more information visit the file types and formats page. |
| 02 Parse | Text, tables and layout are pulled out of each file into plain characters. | Whether a scan or a spreadsheet arrives as structure or as a wall of characters with the rows already gone. |
| 03 Chunk | The parsed text is cut into pieces. The Blockify API documentation calls a thousand to two thousand characters the sweet spot and recommends roughly 10 to 20 percent overlap. | Whether a table row or a numbered step is severed at the cut. For more information visit the naive-chunking failure page. |
| 04 Distill | Blockify sits here. Each piece becomes a block carrying a name, a critical question, a trusted answer and extracted entities; a second pass clusters near-duplicate blocks and strips the redundancy. | What the stored unit holds. Iternal puts a finished data set at roughly two and a half percent of the size of its source documents. |
| 05 Embed | Every unit is turned into a vector by an embedding model, chosen from the embedding models your stack already supports. | That the same model has to run at query time. Change it later and the index is rebuilt, not patched. |
| 06 Index | The vectors are written to a store. On a laptop that store is a flat JSONL file queried by a WebAssembly component; on a server it is a vector database. | How much the corpus can hold and who can query it. The ceilings are below. |
The last four run once per question, in the second or two between a person pressing enter and reading a reply.
| Stage | What happens | What it decides |
|---|---|---|
| 07 Retrieve | The question is embedded and the nearest units come back — commonly the top five, ten or twenty. AirgapAI combines semantic search with keywords and regular expressions. | Whether the answer is in the material the model is about to read. |
| 08 Rerank | A second scoring pass reads each candidate against the question and reorders the list. | Whether the unit that answers arrives first or ninth. The pass is described below. |
| 09 Generate | The model writes the answer from what it was handed, plus the instructions in the prompt. | How much of the bill you pay. For more information visit the token and inference cost page. |
| 10 Cite | The answer points back at the units behind it; each block keeps lineage to the piece and the file it came from. | Whether a reader can check the claim. For more information visit the accuracy and source-proof page. |
Two stages carry most of the outcome. Where the text is cut, and what the stored unit holds, decide what the model can possibly read; everything around them is plumbing you can keep. Iternal designed Blockify to slot into stage four for exactly that reason — it takes input from any chunking approach, sits between the cut and the vector database, and asks for no change to the parser or the reranking model already in the pipeline. It slots into the stacks teams already run, compared side by side on the enterprise RAG platforms and frameworks page. For more information visit the retrieval versus fine-tuning comparison.
Inside the Unit a Retriever Returns
Every retrieval architecture answers one design question: what is the smallest thing worth storing? A fixed split answers with arithmetic. Iternal answers with meaning — a block holds one self-contained concept and never mixes two ideas, which is where Iternal says the gain comes from. Four parts travel together, as standard XML:
| Part | What it holds | What retrieval gains |
|---|---|---|
| Name | A human-readable label for what the block is about. | A person can scan the data set without opening the source. |
| Critical question | The question someone interested in this information would ask. Blockify writes it when the source never asked it. | The stored text resembles the query it should match, which is what makes it findable. |
| Trusted answer | The response, drawn from the source — two or three sentences, roughly a paragraph. | The model gets a finished answer, not five or ten sentences to mine. |
| Entity extraction | Entities, keywords, tags and metadata wrapped around the pair. | Filtering and permissioning by subject — a laptop, a desktop or a server. |
The shape in practice. A mission statement buried on page nine of a deck becomes a block whose critical question asks what the mission is and whose trusted answer is the statement itself. Ask for the mission and the retriever returns that block whole. Ask a fixed-size index and it returns whichever thousand-character window scored highest.
One Page, Two Ways: A Price Table and a Numbered Procedure
Two document shapes break a fixed cut, and buyers described both in almost the same words. A PDF of promotional material broke retrieval because the model did not understand rows in a table and returned the price of the item above the one asked about. Technical manuals lost step order until the procedure became, as one buyer put it, unordered chaos. Neither failure is a model problem. Both were decided at ingestion.
| Source content | What a fixed character cut stores | What a restructured unit stores |
|---|---|---|
| A price table | A window beginning mid-table. Row boundaries vanish, so the model pairs a product name with the neighboring row's number. | One block per row: the item as entity, its price as the trusted answer. A separate Iternal pipeline turns a scan or a CSV into a structured table. |
| A numbered procedure | Steps split across pieces with no record of sequence. Retrieval may return step four alone. | A technical-manual pipeline structures blocks for order and links them so the flow of steps survives. |
| A short prose memo | A clean, coherent window. Nothing structural to destroy. | Marginal gain for real cost. |
Where the plain split is still the right answer, and it genuinely is. Read the third row before the first two. Basic chunking is good enough when one person controls ten or fifteen files with no version sprawl, when the documents are short, uniform and prose-only, and when nothing depends on rows or ordered steps. It is also right when a pipeline already exists around an embedding store and the team wants a fast path: AirgapAI ships basic chunking as exactly that alternative, and Iternal is candid that Blockify suits specific use cases rather than most, and runs slowly enough on a laptop that a server is recommended.
Restructure when the structure carries the meaning. Tables, ordered procedures, large collections carrying duplicated and superseded passages — there an arbitrary cut costs you answers. The extra step does not force a rewrite: Blockify takes input from any chunking approach, sits between the chunk and the vector database, and needs no change to the re-ranker or parser you already run.
Reranking: Putting the Best Unit at the Top
Reranking is a second scoring pass over the units a first retrieval returned. The first pass is fast and approximate; the reranker reads each candidate against the question and reorders them, so the unit that actually answers arrives at position one instead of position nine, and the model reads fewer wrong pieces.
- It reorders; it cannot repair. A reranker scores what the index gave it. If the cut severed a table row from its header, every candidate carries the damage and reordering them changes only which damaged piece is read first. The fix is upstream, at stage three and stage four.
- Blockify sits before the reranker, not in place of it. Restructuring runs between the cut and the vector database, so an existing reranking model keeps its job and scores better-formed candidates.
- Fewer candidates make the pass cheaper. Reranking cost scales with how many pieces are scored and how long each one is. Iternal frames the retrieval shape as around ten blocks in place of twenty chunks, and states the input-token reduction against traditional chunking at about three times — Iternal figures, from Iternal's own comparisons.
- On the device the pass is simpler. AirgapAI's on-device retrieval combines semantic search, keywords and regular expressions. Where a dedicated reranking model is a requirement, it belongs in the server-side pipeline, which Blockify feeds without displacing anything already there.
The Numbers Behind the Split
Iternal publishes the working figures rather than hiding them behind the patent, and every one is approximate by design: content varies, so the pipeline is tuned. How the cutting methods differ from one another is set out in chunking strategies compared.
- About 2,000 characters go in. Iternal states the standard input size as approximately 2,000 characters, roughly one page. It is configurable to a thousand or four thousand, with 2,000 reported as best; the Blockify API documentation calls a thousand to two thousand the sweet spot and recommends roughly 10 to 20 percent overlap.
- Around 5 to 10 blocks come out. The stated yield from about 2,000 characters of input, hedged as a range because the density of the source decides it. A page of specification produces more; a page of narrative, fewer.
- Then the set gets smaller, not bigger. Distillation clusters semantically similar blocks and a second pass strips the redundancy. Iternal puts a finished data set at roughly two and a half percent of the size of its source documents.
The counter-intuitive consequence. Blocks are smaller than chunks, so the system issues more queries per question — yet chunks are larger in aggregate, which is where the saving lands. Iternal frames it as around ten blocks in place of twenty chunks.
How Big the Corpus Gets, and Where It Lives
Two questions arrive together, and buyers repeatedly asked both: how much can one data set hold, and can what sits on a hundred laptops be pulled into one place. Both answers turn on where the work runs.
On the device, stay selective. Keep as many data sets on the machine as you like and toggle them on and off per chat; a query is scoped to a specified data set, which is why chunk-built and block-built embeddings never mix in one result. Iternal is direct about the rest: on-device data set creation is not adequate for the volumes large firms hold, a large batch added to an existing on-device corpus can bottleneck, and ingestion there runs at roughly a page a minute, against Word files and PDFs only.
On a server, the numbers change shape. Ingestion runs up front, before content reaches the device, and the pipeline handles thousands or tens of thousands of documents in one run. Iternal describes a gigabyte-scale data set as holding hundreds of thousands of blocks rather than millions, and notes that a small accelerator box caps how much gets processed.
Sharing across users is the gap. The current AirgapAI implementation cannot centralize the corpus across users: every install is single tenanted, and no user sees another user's material. Where one shared index is the requirement, Blockify is decoupled from the assistant and can power a central vector database behind a different front end. For more information visit the enterprise AI search page. Pin the ceilings down before you commit:
Pin it down: questions for your evaluation
-
How many data sets can a single query span in the release we will receive?Whether a question crossing two subject areas takes one query or two, which changes how you carve up the corpus.
-
What document count and total size have you run on the exact device class we are buying?A real on-device ceiling tied to your hardware rather than to a demo machine.
-
What is the per-message input ceiling on our hardware, and how is it set?How much material one question carries. AirgapAI benchmarks the machine at first run and derives the window from hardware and model, so any single published figure would mislead you.
-
Does the interface expose the original source text behind a block?Whether a reader who needs the exact wording reaches it from the answer, or opens the source separately.
Past the Pilot: What Actually Limits Volume
The sharpest objection here is not about quality. It is that the approach quietly assumes a pilot: a hundred locally processed vector databases for a hundred staff on one repository does not sound like a good idea, one instance will not serve a firm of three hundred, and restructuring multiplies indexed data points so queries must cost more. The reassuring answer would be that it all scales. It does not.
- The index does not stay multiplied. The expansion buyers worry about is measured before distillation, and the set shrinks substantially afterwards. Block queries are also more precise and run on the CPU rather than a GPU, keeping them viable on an ordinary machine.
- The device store is deliberately simple. Iternal grants that the flat file arrangement works because nothing serious runs against it on the endpoint. Treat it as a per-user working set, not a firm-wide index.
- Volume is a two-stage process. Ingest and convert on capable hardware first, then load the processed data and query it. Iternal does not offer unlimited large-scale ingestion, because the processing cost sits with Iternal: volume is scoped, and pricing tiers by document count.
A laptop holds a curated working set; a server holds the corpus. Choose between them before the pilot, not after it.
Fewer Input Tokens for the Same Question
The retrieval unit sets the bill. Feed the model twenty windows of a couple of thousand characters each and it pays to read past the irrelevant parts; feed it ten finished answers and it does not. Iternal states the reduction in input tokens against traditional chunking at about three times, and describes block-carried context as roughly three times smaller — Iternal figures, from Iternal's own comparisons, which Iternal will re-run on your data during an evaluation.
What that does to a monthly bill belongs elsewhere. For more information visit the token and inference cost page.
Adjacent questions are answered on their own pages, including AI grounding and traceable answers.
Answered elsewhere
- Why an answer comes back wrong, and how a person proves one right — see the accuracy and source-proof page.
- Which files the pipeline accepts, and what happens to scans and spreadsheets — see the file types and formats page.
- Crawling file shares, object storage and databases, and refreshing after a source changes — see the connectors and re-indexing page.
- Governing material spread across many separate stores — see the governance page.
- How the ingestion engine, the local assistant and the finished data set relate — see the components page.
- What one query costs to run — see the token and inference cost page.
FAQ
FAQ: Retrieval Over Your Own Documents
Because the cut is arithmetic, not structural. A fixed-size split ends a piece at a character count, so a table row is severed from its header — buyers described a PDF that returned the price of the item above the one asked about — and a numbered procedure lands across pieces with no record of sequence. Iternal handles both at ingestion, with one pipeline for structured tables and another that preserves step order in a technical manual.
Retrieval-augmented generation runs in two halves. Build time parses your documents, cuts them into units, embeds each unit as a vector and writes it to an index. Query time embeds the question, returns the nearest units, reorders them and hands only those to the model, which writes the answer from what it was given rather than from memory. In this pipeline a distillation step sits between the cut and the index: Blockify converts each unit into a name, a critical question, a trusted answer and extracted entities, so the piece the model reads is already a finished answer. The ten stages are walked out above.
The ordered set of stages between a source file and a cited answer: ingest, parse, chunk, distill, embed and index, then retrieve, rerank, generate and cite. The first six run once per document, ahead of any question; the last four run once per question. Two of them carry most of the outcome — where the text is cut, and what the stored unit holds — and the rest is plumbing you can keep, since restructuring slots between the cut and the vector database without displacing the parser or the reranking model.
The stored pieces nearest your question in vector space — not the document. Standard practice returns the top five, ten or twenty, each a window of a couple of thousand characters, leaving the model to hunt for one sentence inside ten thousand. AirgapAI retrieval combines semantic search, keywords and regular expressions; graph retrieval is not part of it today.
More than a demo, less than a firm. Iternal calls on-device data set creation inadequate for the volumes large firms hold, and a large batch added to an existing on-device corpus can bottleneck. On a server, one pipeline run handles thousands or tens of thousands of documents, and Iternal describes a gigabyte-scale data set as holding hundreds of thousands of blocks rather than millions.
Not inside the current AirgapAI implementation. Every install is single tenanted, the corpus is not centralized across users, and no user sees another user's material. Where one shared index is the requirement, Iternal points at Blockify: it is decoupled from the assistant and can power a central vector database behind a different front end.
On a server, yes; on the endpoint, inside limits Iternal states openly. The device vector store is a flat JSONL file chosen for the size of the machine, and Iternal grants it works because nothing heavy runs against it there. Volume is two-stage: ingest and convert on capable hardware, then load the processed data and query it.
Not always word for word. Iternal states that this version of Blockify may change some semantic wording while preserving the underlying message, so a retriever returns a restatement rather than a quotation. Each block retains lineage back to the chunk and file it came from; whether the interface puts the original text beside the block is worth confirming in your build.
Build the Same Corpus Twice
One experiment settles the architecture question. Take the ugliest document you own — the one with a price table and a numbered procedure in it — build a data set both ways, and ask each the same five questions. What comes back tells you which architecture your content needs, and no datasheet outranks that result.