A metered bill is a forecast nobody in your building controls. Executives told us the same thing in a dozen phrasings: a finance chief cannot protect an annual budget when the charge lands per token, and cloud tokens get burned with no view of the value coming back. The question underneath is narrow — which levers actually move the number?
Token & Inference Cost
How Do You Reduce
LLM Token and Inference Costs?
Three levers ranked by size, the arithmetic to run them on your own volume, and the limit on the biggest.
Built from real buyer questions in our sales meetings
Direct Answer
Three levers, in order of size. Move the high-volume, low-complexity work onto hardware you already own, where an extra call costs nothing extra because the processing happens on the device. Shrink the context you send by retrieving structured blocks instead of raw chunks: Iternal states Blockify-optimized data cuts input tokens by roughly three times on retrieval and agentic search, the same measurement Iternal also states as a 75% reduction in input tokens. Then cap consumption per user and per endpoint.
The limit lands on the biggest lever. Moving work onto hardware you own removes the per-call charge and leaves a real cost behind: a device, its power draw and a smaller model. Running locally also means no outside internet connection and no access to the largest models with million-token context windows, so a share of the work does not move. Local models sit closer to 2024-era hosted quality than to a top-tier data-center model.
The third lever is thinner than it sounds. Metered tools do not always expose a switch that caps spend by the day or by the month. Where it is missing, the ceiling has to be enforced elsewhere in your estate — rate limiting per session, identity signals beyond a raw IP address, daily resets.
Settle three numbers before you model any of it: the price per million on your own contract, the input-to-output split on your workload, and the share of queries a smaller model answers well enough. Iternal states AirgapAI carries no token cost because everything is processed on the local AI PC, so no unexpected bill arrives at quarter end. For more information visit the page on what a private assistant costs.
What a Token Is, and What One Actually Costs
Buyers ask it plainly, before price comes up: do you work on tokens, and how do tokens work? A token is a fragment of text — roughly four characters, about three-quarters of a word — so a page of business writing runs around 600 to 800 tokens.
The count is bigger than the sentence you typed. Prompt tokens cover the system prompt, any persona prompt, your question and every context item retrieved to answer it; tool definitions alone can add a couple of thousand tokens before a single result comes back.
Price is quoted per million, and the range is wide. Higher-tier hosted models run about $5 to $10 per million, a small hosted model near a dollar, cached input an order of magnitude less. On hardware you own the per-call charge is zero. Tokens are the meter. Context is what spins it.
Lever Two: Send Less Context for the Same Answer
Most retrieval systems answer by shoveling raw chunks of source documents into the prompt and hoping the right sentence sits in the pile. Iternal built Blockify to send structured blocks instead, and states the result as roughly three times fewer input tokens on retrieval and agentic search than traditional chunking — the same measurement Iternal also states as a 75% reduction in input tokens and as about a three-times infrastructure saving at the same load. One number, three vocabularies: a quarter of the volume is three-quarters fewer tokens.
Scope matters more than the multiple. The figure applies to input tokens on the data Blockify has processed, so it moves your input side and leaves output alone. It compounds with volume: at hundreds of thousands of queries a day, a third of the input load separates a budget from an incident.
Optimizing the data is itself a token cost, paid once. Iternal states a standard Blockify chunk consumes about 500 input tokens and produces about 1,250 output tokens: you pay once per document and collect the reduction on every query after that.
Pin it down: questions for your evaluation
-
Which part of our bill does the three-times input-token reduction apply to, and what does it leave untouched?Whether to model the saving against the whole token bill or only the retrieved-context share of input tokens.
-
Can you run the token-savings report on our own documents before we commit?The figure measured on your corpus. Iternal states the report shows accuracy block by block and in aggregate, and calculates token savings on the customer data.
Lever Three: Put a Ceiling on What Reaches the Meter
Consumption pricing reports after the fact, and the timing is the whole objection. Buyers described costs they cannot anticipate before the bill lands, and services that announce you are out of tokens with the only remedy being to buy more. Exposure carries the tail risk: a public-facing AI page attracts spam queries, and one heavy user drives the whole cost once a tool is opened outward. One executive put the trade cleanly — the public cannot be limited, and usage cannot be left unbounded either.
The controls that answer it are ordinary engineering. Iternal builds rate limiting and session-ID controls into the current application, tracking messages per session in the database and identifying users by IP address, browser and a few other signals. Daily resets cap usage without denying access: come back tomorrow. One weakness to design around — rate limiting on IP address alone exhausts quickly when your internal users all arrive from a single VPN address.
The ceiling you cannot always set inside the tool. Metered products do not universally offer a per-day or per-month spend cap, and where that control is absent the limit lives in the layer you build around the model. Local execution removes the meter instead: Iternal sells AirgapAI as a one-time perpetual license per device with no token cost, and buyers repeatedly said they valued that predictability.
Lever One: Why the Frontier Bill Stops at a Handful of People
The ceiling arrives long before the whole workforce does. Buyers told us their cloud language-model bill keeps growing and they cannot afford to roll it out to everybody; that a thousand employees on frontier models is unaffordable; and that frontier token costs climb high enough to make hiring a senior developer the cheaper option. A metered plan breaks on population before it breaks on price.
Match the model to the job, because no single model fits every use case. Frontier models earn their price on coding and product building; smaller models handle data processing and the everyday questions that make up most of the volume, and pre-processing with a small model costs far less than running a tier-one model over dirty data. AirgapAI starts with small language models on the device and bursts to large models in a data center when the work warrants it, putting the population on hardware you own and reserving the meter for hard problems.
The constraint is real for us too. Iternal watches its own model budget, gives the allowance to developers first until the limit is hit, and runs into throttling on its own frontier usage. Shrink the share of work reaching the meter and headcount stops setting your bill.
Run the Arithmetic on Your Own Monthly Volume
Every figure below is replaceable; the shape survives. Take your query count from your usage report, then use the deliberately high 20,000-tokens-per-query assumption Iternal applies when it models customer workloads, built with roughly a five-times cushion so the estimate errs against itself:
| Line | Where the number comes from | Worked example |
|---|---|---|
| Queries per month | Your platform usage report | 10,000 |
| Tokens per query | A deliberately high ceiling | 20,000 |
| Tokens per month | Queries × tokens per query | 200 million |
| Price per million | Your contract; higher tiers about $5 to $10 | $5 |
| Bill before | (Tokens ÷ 1M) × price | $1,000 per month |
| Share moved on-device | Your call: the low-complexity work | 60% |
| Tokens still metered | Remaining share × tokens per query | 80 million |
| Input share of what remains | Ask for your input and output split | Half, or 40 million |
| Input tokens after the context lever | Roughly three times fewer | About 13 million |
| Bill after both levers | (53M ÷ 1M) × price | About $265 per month |
Read the shape, not the total. The input-to-output split is the one number that has to come from your own workload, and it decides how much of the second lever you collect. The lesson outlives the figures: the lever that moves the most money takes work off the meter entirely, and the second multiplies what is left.
Answered elsewhere
- What a private assistant costs to own — see the cost breakdown page.
- How the license is structured — see the licensing page.
- Weighing a device purchase against a monthly seat charge — see the device and subscription comparison.
- How retrieval is structured, and what a block is — see the retrieval architecture page.
- Day-to-day work with coding agents and the limits they hit — see the coding agents page.
- Where model prices are heading — see the model economics page.
FAQ
FAQ: Token and Inference Cost
A token is a fragment of text, roughly four characters or about three-quarters of a word, so a page of business writing runs around 600 to 800 tokens. Hosted models bill per million: higher tiers about $5 to $10, a small hosted model near a dollar, cached input far less. The count covers the system prompt, any persona prompt, your question and every retrieved context item. On a local AI PC there is no token cost at all.
Three controls: rate limiting tracked per session in the database, identification combining IP address with browser and a few other signals, and a daily reset telling a heavy user to come back tomorrow rather than locking them out. IP address alone exhausts fast when internal users arrive from one VPN address. Metered products do not universally expose a per-day or per-month cap, so the hard ceiling often lives in the layer you build.
Because consumption pricing reports after the fact. Buyers described costs they cannot anticipate before the bill lands, usage that blows through a budget inside a month, and a service announcing you are out of tokens with the only remedy being to buy more. Two structural fixes exist: move predictable, high-volume work onto hardware you own, and put hard limits around whatever still reaches a metered model.
More than most organizations will fund. Buyers told us a growing cloud bill already rules out rolling it out to everybody, that a thousand employees on frontier models is unaffordable, and that frontier token costs climb high enough to make hiring a senior developer cheaper. Match the model to the job: frontier models for coding and product building, smaller models for data processing and everyday questions.
Iternal states roughly three times fewer input tokens on retrieval and agentic search when data has been Blockify-optimized rather than traditionally chunked, and states the same measurement as a 75% reduction in input tokens and as about a three-times infrastructure saving. It applies to the input side of the workloads Blockify processed, and optimizing the data is itself a token cost paid once per document.
Model It Before You Negotiate It
Take one month of real query volume, run it through the table above, and you will know within an hour which lever is worth a project. Then measure the second one on your own documents, not on a published figure.