Every AI project reaches a moment when the pile of documents stops being background and becomes the project. It arrives late, usually after the model has been picked and the date has been promised. Buyers described the situation to us in almost identical words: everybody needs to get their data house in order, and nobody knows how to do it. The instruction is universal. The method is missing.
Getting Data Ready for AI
How Do You Get Messy Enterprise Data
Ready for AI?
Five steps in a fixed order, an owner and an exit test for each one, and the page that settles the question every step raises.
Built from real buyer questions in our sales meetings
Direct Answer
Readiness is a sequence, not a purchase. Retrieval cannot repair what ingestion never fixed, so the work runs in a fixed order: inventory what you actually hold, de-duplicate it, restructure it into units a machine can address, govern it, and only then retrieve against it. Skip a step and the failure surfaces two steps later wearing a different costume. Each stage below has a fixed position in that order, a named owner and an exit test that says when it is finished.
The limit: nothing in the sequence removes the human decision at the point where two documents say different things. Iternal is direct about it. Blockify cannot choose on its own between source documents that say different things, or between a current file and a superseded one, and needs a person to pick; spotting the divergence in the first place still ends at a human reviewer rather than at an agent. Cleaning the corpus makes the divergence visible. It does not settle it, and it will not tell you which of two documents is the live one. Budget for that adjudication, because it survives every step below.
Three things to settle in writing before the first batch runs. Which part of the cleaning work Iternal performs and which part stays with your team. Who signs off on the survivor when two versions of one document differ. How small the first corpus is allowed to be. The targeted questions below put each of them on paper.
Six pages carry the detail behind the sequence. The prepared unit and the cost of a fixed-size cut belong to the retrieval architecture page; formats and source condition to the file-format page; keeping the set current to the connectors and refresh page; material spread across many stores to the governance page; trusting and proving an answer to the accuracy page; and the running order of the software to the components page.
Everyone Says Get the House in Order. Nobody Says How.
Two complaints arrive together and get treated as one problem. They are two problems, and only the second one is fatal.
The first is the state of the material. Buyers repeatedly described their own corpus as messy, duplicated, unclassified and written for a human eye rather than for a machine. Their diagnoses run sharper than the label. Chunking a large proposal library loads the vector database with enormous amounts of redundant, duplicative content. Paragraphs come back noisy because people compress several ideas onto one page. Commentary either side of the useful sentence burns tokens that bought nothing. One buyer reduced the category to four words: rubbish in, rubbish out.
The second is the absence of a method, and it is the one that kills quarters. Buyers told us that data preparation is the key obstacle with every customer; that projects stall between pilot and production because the material was never structured for AI; that teams who skip the preparation discover it mid-project and stall there. Published guidance names the obstacle and then stops.
A problem everybody names and nobody sequences stays a problem. What follows is the sequence.
The Five Steps, the Owner and the Exit Test
Order matters more than tooling here, because each step consumes the output of the one before it. Give every stage a name on it and a test that says when it is finished. An unowned step is not a step. It is a hope with a slide about it.
| Step | Who owns it | Done when |
|---|---|---|
| 01 · Inventory Decide what goes in and what stays out. |
The business owner of the content, not the platform team. | Every folder in scope is named in writing, and so is every folder left out. |
| 02 · De-duplicate Collapse the copies competing to be retrieved. |
The pipeline, with a named reviewer over the survivors. | A passage that existed in dozens of copies is now a countable set of variants somebody has read. |
| 03 · Restructure Turn pages into units a retriever can address. |
The pipeline. | A sample of retrieved units each stand on their own, without the page around them. |
| 04 · Govern Attach source, owner, sensitivity and version. |
A named data owner per corpus. | Any unit traces to a document and to a person you can call when it turns out wrong. |
| 05 · Retrieve Point the assistant at the finished set. |
Whoever operates the assistant. | A reader walks from an answer back to the passage behind it, on your documents rather than on a demonstration set. |
The order is not a preference. De-duplicate before deciding what belongs and you clean material you should have excluded. Govern before restructuring and you tag units that get rebuilt. Retrieve before any of it — the default most organizations reach by accident — and day one delivers a confident wrong answer.
What AI-Ready Data Means, and How to Tell When You Have It
AI-ready data is material that has been made findable, deduplicated, current, attributed and chunk-ready for one named use case. It is not a grade a data set carries permanently: a corpus ready to answer service questions is not automatically ready to answer contract questions. The five properties below are the test.
Most teams meet the phrase before they meet the work. Gartner predicts that through 2026 organizations will abandon 60% of AI projects unsupported by AI-ready data, and found that 63% either lack the data management practices AI needs or cannot say whether they have them (Gartner, Lack of AI-Ready Data Puts AI Projects at Risk, 2025). The finding travels well. The definition travels badly. An executive repeats the number, a data team is asked to make the data ready, and nobody can say what state to stop at.
Readiness is qualified by the question you intend to ask. There is no universal grade here — a set is ready for a defined use case, on the evidence that use case needs, or it is ready for nothing. That is exactly why “get the data house in order” fails as an instruction and the five steps above do not: a house has no exit test, and one question does.
| Property | What it means once it is true | How you check it |
|---|---|---|
| 01 · Findable | Every source in scope is named, reachable by the pipeline and in a condition a parser can read — and every store deliberately left out is named beside it. | The in-scope list names systems and their owners rather than folders on somebody’s laptop. Reach is settled on the connectors and refresh page, condition on the file-format page. |
| 02 · Deduplicated | The near-identical copies of one passage have collapsed into a countable set of variants somebody has actually read, instead of competing with each other for the same query. | Ask the question that used to return four versions of one paragraph and count what comes back. What that does to the error rate is the accuracy page. |
| 03 · Current | The set has a refresh cadence and a named owner, so a superseded file stops answering on the day it is replaced rather than on the day somebody notices. | Change one source document and time how long until the assistant answers from the new one. The refresh mechanics are the connectors and refresh page. |
| 04 · Attributed | Every unit carries source, owner, sensitivity and version, so entitlement and provenance travel with the content rather than with the folder it used to sit in. | Pick any unit at random and name both the document it came from and the person you would call when it turns out wrong. Tagging across many stores is the governance page. |
| 05 · Chunk-ready | Units stand on their own: a table row keeps its header, a procedure keeps its step order, and a retrieved fragment reads without the page around it. | Read ten retrieved units cold and see how many answer anything. What a prepared unit holds is the retrieval architecture page. |
Readiness checklist: six answers before you call a corpus ready
- Is every system in scope named in writing, with the stores you excluded named beside them?
- Where copies collapsed, has a person read the survivors?
- Does one named owner hold the refresh, on a cadence you could put in a contract?
- Does every unit carry a source, an owner, a sensitivity and a version?
- Does a unit sampled at random answer its question without the page around it?
- When two current-looking units say different things, is there a named person who picks?
Two of the five properties are produced by software; three are decisions. Blockify performs steps two and three of the sequence: it takes messy source documents — the duplicated proposal libraries, the handbooks written for a human eye — and returns units built to be retrieved, with the copies collapsed into variants and the structure the source lost put back. Findable, current and attributed stay with people: what goes in, who refreshes it, who owns it. The pipeline can enforce those three once you have decided them, and it can enforce none of them for you. For more information on which software runs which step, visit the components page.
One Handbook, Two Outcomes
Argument persuades nobody at this altitude. A before-and-after on a single document does, so take one Iternal has actually run.
The before. A clinical reference handbook covering the thousand most common diseases and ailments, split at a fixed size, embedded and queried through a vector database — the pipeline most teams build first.
The after. The same book, run through Blockify, loaded the same way and asked the same questions. Iternal reports a 611% improvement in response quality and accuracy from that comparison, and ships the prepared handbook as one of the sample data sets inside AirgapAI. Same source, same questions, same model. Preparation was the only variable that moved.
Read the result for its direction rather than its size. The multiplier tracks how much redundancy and structure the source carried to begin with, and a reference book carries a great deal of both; your own material may carry more or less. The figure comes from Iternal testing rather than from an independent benchmark. The narrow claim it supports is the one that matters here: the result moved because of what happened to the document before the model ever saw it. For more information on what a prepared unit actually contains, visit the retrieval architecture page.
Start With the Smallest Corpus That Answers Something Real
One objection lands harder than the rest, and buyers put it in these words: it still comes back to rubbish in, rubbish out, and our material is nowhere near clean enough. Right about the premise. Wrong about what follows from it.
Dirty material argues for a scoped first corpus, not for a year of preparation. Iternal recommends the opposite of a program: start small, take an AI win at low cost, prove it works, then scale. A program to clean everything runs for quarters and shows nothing at the end of the first one. A scoped batch answering one question a department actually asks produces something a skeptic can look at.
Choose the batch deliberately, and never aim ingestion at the whole share. Iternal advises against pointing the pipeline at a raw, sprawling file server, and buyers supplied the reason: indexing somebody’s local drive drags in material nobody wants an assistant citing back. Cleaning the source sets is where the main work lies, and the accuracy gains depend on what you chose to feed it. Two scales exist for that cleaning, one light enough to run on a device and one built for volume. For more information visit the components page.
Scoping the first batch also scopes the controls it has to carry: a smaller corpus makes tagging, ownership and permissions tractable before the volume arrives. For more information visit the AI data governance page.
Four written answers turn the sequence from a diagram into a plan somebody can be held to:
Pin it down: questions for your evaluation
-
Which part of the preparation does Iternal perform, and which part stays with our team?The line between preparing material and processing it, before it becomes a schedule surprise.
-
Who signs off on the survivor when two versions of one document say different things?That the human step the sequence cannot remove has an owner and an audit trail.
-
What is the smallest corpus you would run this sequence against, and what does a first pass involve?Whether a first proof fits inside an approval you already hold.
-
Which steps can run against tooling we already own, and which need Iternal software?Where the sequence touches your stack, and what a license has to cover.
Where Each Step Gets Answered
Every stage raises one specialist question, and each has its own page. Work the sequence in order and the reading order writes itself. Steps three and five are the two halves of retrieval, and the retrieval architecture page walks the RAG pipeline stage by stage.
| Step | The question it raises | Where it is answered |
|---|---|---|
| 01 · Inventory | Which systems can the pipeline reach, and who keeps the set current? | the connectors and refresh page |
| 01 · Inventory | Which formats are accepted, and in what condition? | the file-format page |
| 02 · De-duplicate | What does removing the copies do to the error rate? | the accuracy page |
| 03 · Restructure | What does the prepared unit hold, and why does a fixed-size cut sever a table row? | the retrieval architecture page |
| 04 · Govern | How do we tag, permission and version scattered material? | the governance page |
| 05 · Retrieve | Can a reader prove an answer back to the passage it came from? | the source-proof page |
| All five | Which software runs the sequence, and in what order? | the components page |
Answered elsewhere
- Whether a local model can be tuned on your own documents instead — see the model provenance page.
- Which machine the pipeline and the assistant install on — see the placement page.
- What the software keeps or sends once material is loaded — see the data-retention page.
- What the ingestion work and the licenses cost — see the cost page.
- How to pick the first question this corpus should serve — see the use-case selection page.
FAQ
FAQ: Preparing Enterprise Material for AI
Run five steps in order: inventory what you hold, de-duplicate it, restructure it into units a machine can address, govern it with source, owner, sensitivity and version tags, then retrieve against it. Give every step a named owner and an exit test. Iternal states the pipeline still cannot pick between two source documents that say different things without a person choosing, so budget for that decision.
Material that is findable, deduplicated, current, attributed and chunk-ready for one named use case. Readiness is not a permanent grade a data set carries: a corpus ready to answer service questions is not automatically ready to answer contract questions. Gartner predicts that through 2026 organizations will abandon 60% of AI projects unsupported by AI-ready data (Gartner, Lack of AI-Ready Data Puts AI Projects at Risk, 2025). The five properties above each carry a check you can run.
Three things, and only the last is software. Deciding what belongs in the corpus and what stays out. Getting the material into a condition a parser can read. Then the cleaning and restructuring pass itself. Buyers told us preparation is the key obstacle with every customer, and that teams who skip it discover it mid-project. For more information visit the file-format page.
Inventory, de-duplicate, restructure, govern, retrieve. The order is load-bearing. De-duplicating before you have decided what belongs cleans material you should have excluded; governing before restructuring tags units that will be rebuilt; and retrieving before any of it is the arrangement that produces a confident wrong answer on day one.
Yes, provided you scope it. Iternal recommends starting small, taking a win at low cost, proving it works and then scaling, and advises against pointing ingestion at a raw, sprawling file server. Pick the batch that answers one question a department actually asks. A program to clean everything runs for quarters and shows nothing at the end of the first one.
No. Retrieval returns what ingestion left behind, so duplicated and superseded passages simply reach the model carrying a high similarity score. Iternal reports a 611% improvement in response quality and accuracy from preparing one clinical reference handbook before querying it, with the source, the questions and the model all held constant.
No, and Iternal is direct about the boundary. Blockify cannot choose on its own between source documents that say different things, or between a current file and a superseded one, and needs a person to pick; spotting the divergence in the first place still ends at a human reviewer rather than at an agent. Preparation makes the divergence visible and finite. A named reviewer settles it.
Start With the Inventory
Name the batch, name its owner, and the next four steps have somewhere to run. Every question the sequence raises — the formats, the copies, the units, the tags, the proof — has a page above that settles it.