Most of your AI spend re-buys the same context. Route every request right. Pay for repeats once.
Ultracache is the routing and caching layer of Iternal’s AI gateway — one governed connection point that sends each request where your policy prefers, keeps conversations where their reuse is already warm, and switches on each provider’s published repeat-context discount. Your applications change nothing.
Ships off by default — identical to today until you choose otherwise.
Provider-published pricing and documentation, quoted as provider facts — not Iternal-measured results. Savings depend on your provider pricing and how much of your workload is repeated context. Confirm in a pilot.
Your AI bill is mostly re-runs.
Your AI spend grows faster than your AI usage — and the fastest-growing line item is work you already paid for.
You pay over and over for the same context
Every turn re-sends the same instructions, reference material, and history — re-billed at full price. The published discount never reaches your invoice, because capturing it takes per-provider discipline nobody has the bandwidth to build.
Every team quietly picked a provider — and locked you in
Hard-coded choices mean switching is a rewrite. That is negotiating leverage you have already given away.
No one can see where the money goes
No central policy, no record of why a request went where it did, no apples-to-apples view across providers.
One governed layer, and the economics invert.
Put one governed layer between your applications and the providers, and repeated context stops being a bill you pay twice.
Every dollar routed intelligently
Each request goes where your policy prefers; each provider’s published discount is switched on for you.
No provider holds your future
Switching or mixing providers becomes a policy change, not an engineering project.
Spend you can see and govern
One cross-provider view of usage and reuse efficiency; every decision on the record.
An advantage that compounds
The more AI you run, the larger the pool of spend this layer works on.
Four things happen to every request, in order.
One connection point in front of the providers — and the same four movements on every request, every time.
Arrive
One endpoint, your own model names; applications change nothing.
Select
Policy ranks the valid routes; the same inputs always produce the same choice.
Hold the track
Ongoing conversations stay where their reuse is already warm — durably.
Settle
Repeated context is processed once and reused; the decision and the savings go on the record.
Routing is invisible on the wire — the answer always comes back under the model name you asked for.
Same conversation, two routing postures
IllustrativeProvider caches expire on their own schedule; holding the track keeps consecutive turns landing where the context is still warm.
Why discipline beats discounts
The published discount only exists where reuse is warm. Without a layer that holds the track, ordinary load-spreading silently lands every turn cold — and the discount evaporates.
Spend less. Govern more.
Spending less must never mean governing less — this is a native layer of a gateway you already govern, not another middleman.
A native control point, not a middleman
It inherits the platform’s identity, authorization, tenant isolation, usage metering, and audit trail — one governance model across many providers, rather than a bolt-on that becomes its own dependency.
Savings without a new leak surface
The stamp handed to providers is sealed and organization-scoped. Raw request content and hints are never stored, never logged, and never forwarded.
Off by default, zero regression
The capability ships disabled. With routing off, behavior is identical to the prior release — a guarantee enforced by the test suite.
No new way to fail
It only re-ranks routes your platform already validated; anything that failed before fails identically now. Usage is metered exactly as it is today.
Every decision explainable
Each request records its route and the reason, drawn from a small fixed set. Record-keeping is advisory — it can never fail a request.
The first release passed a multi-stage adversarial engineering review — every finding remediated before release.
The register, plainly marked.
Everything the argument above rests on is shipping in the current release — 9 capabilities you can exercise on your own workload.
| Capability, in plain terms | Status |
|---|---|
| One endpoint, your own model names — applications never learn or care which provider served them; providers stay portable | Ships |
| Central, governed routing policy — which models serve which traffic is owned by your platform, not scattered across teams’ code | Ships |
| Predictable, explainable route selection — the same inputs always produce the same choice; only fully validated routes are ever eligible | Ships |
| Each provider’s published repeat-context discount switched on automatically — no per-provider engineering, no application changes | Ships |
| Conversations held on the route where their reuse is already warm — so the discount is realized, not lost to load-spreading | Ships |
| A sealed, organization-scoped identity for reuse — raw request content and hints never stored, logged, or forwarded | Ships |
| One cross-provider view of usage and reuse efficiency, plus per-response proof of savings | Ships |
| A record of which route served each request and why — and record-keeping can never fail a request | Ships |
| Off by default, opt in one model at a time — a zero-regression guarantee when disabled | Ships |
What prompt caching is, and where it saves tokens.
Prompt caching is a provider feature that processes a repeated block of input once and reuses it on later requests at a published discount instead of full input price. It covers the parts of a request that never change: standing instructions, retrieved documents, tool definitions, and the conversation so far.
Where the repeated tokens actually are
Four places account for most of the input on a mature AI workload. All four are written once and sent thousands of times.
Standing instructions
The system prompt, the tone and policy rules, the refusal boundaries. Identical on every call — and re-billed on every call.
Retrieved context
The documents your retrieval step attaches. The same policy manual rides along on every question anyone asks about it.
Agent scaffolding
Tool definitions, function schemas, worked examples. An agent with twenty tools carries all twenty into every step it takes.
The conversation so far
Turn twelve resends turns one through eleven, so a long session costs far more than its length suggests.
Cache-hit economics: one twelve-turn session
IllustrativeTake an internal agent whose every request carries 20,000 tokens of standing context — system prompt, tool definitions and two retrieved documents — plus 500 tokens of genuinely new input, over twelve turns. The context is identical every time.
| Session shape | What the session bills | Change |
|---|---|---|
| Every turn cold — no reuse | 246,000 input tokens at full price | Baseline |
| Warm from turn 2, at Anthropic’s published up-to-90% cheaper reused input | ≈ 48,000 full-price-equivalent tokens | About 80% less |
| Warm from turn 2, at OpenAI’s published approximately 50% cheaper reused input | ≈ 136,000 full-price-equivalent tokens | About 45% less |
Arithmetic on the stated session shape at each provider’s published reused-input rate — Anthropic publishes up to 90% cheaper reused input, OpenAI approximately 50%. Iternal publishes no measured savings. Your number depends on your provider’s current pricing and how much of your traffic is repeat.
How Ultracache routes and caches across models
Reuse is provider-side and route-bound: each provider warms its own copy of your prefix, and it stays warm only for as long as that provider publishes. Ordinary load-spreading scatters consecutive turns across providers, so every turn arrives somewhere cold and the discount never appears on the invoice. Ultracache selects a route from policy — the same inputs always produce the same choice — and then holds an ongoing conversation on the route where its reuse is already warm, switching on each provider’s repeat-context discount without any application change.
This is prefix reuse, not answer reuse. The provider is always called and always writes a fresh response; what stops being re-billed is the processing of material you already sent. And it is not a learned router model guessing which model to use: a router model that varies its choice from one turn to the next also varies where the context is warm, which is exactly how the saving gets thrown away. Deterministic policy first, cache economics second.
Where prompt caching does not help
The reused block has to be a stable prefix. Put a timestamp, a user name or a request id at the top of the prompt and every request looks new, so nothing is ever warm — move the variable parts to the end. Short one-off prompts with no shared opening have nothing to reuse, and a cache-hit rate near zero on genuinely novel traffic is the correct result, not a misconfiguration.
For the token math behind these figures, see the token usage guide. To price your own workload against current provider rates before you change anything, use the LLM API pricing calculator.
What an AI gateway (LLM gateway) is, and where Ultracache sits in one.
An AI gateway is the single control point every AI request passes through. It routes each call to the right model, reuses repeated context instead of re-billing it, enforces rate limits and budgets, holds the provider keys so applications never do, and logs every call for audit. An LLM gateway is the same pattern named for language models.
The five jobs a gateway is asked to do
- Route Send each request to a model that satisfies the policy for that route, and fail over when a provider degrades.
- Reuse Keep repeated context warm so the same standing instructions and documents are not re-billed at full price on every turn.
- Meter Count tokens and spend rather than requests, and enforce per-team budgets and rate limits before the invoice arrives.
- Hold the keys Keep the provider credentials in one governed place so applications and agents never carry a long-lived key of their own.
- Record Log where every request went, which model answered and under whose authority, in a form an auditor can read.
Three ways estates end up with one
Every team that runs more than one model eventually builds this control point, deliberately or by accident. Three patterns dominate, and each is a reasonable answer to a different starting position.
API gateway plus AI plugins
Teams that already run an API gateway extend it. Kong and the Envoy AI Gateway project add model-aware plugins so AI traffic inherits the policy plane the rest of the estate already uses. Strongest when the platform team owns the API gateway and wants one place to reason about traffic.
A dedicated AI gateway
Cloudflare AI Gateway, Vercel AI Gateway, Portkey and TrueFoundry ship the model-specific parts as a product: provider fallbacks, caching, token accounting and per-model limits, available in an afternoon. Strongest when the priority is model-aware control quickly, without building it.
Native to the platform you already run
IBM, Microsoft and the other enterprise platform providers publish gateway capability alongside their model services, so identity, networking and audit come from the platform of record. Strongest for regulated estates that must keep the control point inside an environment they already accredited.
AI gateway against API gateway
An API gateway routes HTTP by path and method, authenticates the caller, and rate-limits by request count; by design it does not care what is inside the payload. An AI gateway does that same policing for model traffic and adds the work that only makes sense there: choosing which model answers and failing over when one degrades, keeping repeated context warm so it is not paid for twice, accounting in tokens and spend rather than request counts, and recording where each request went and why. Most estates run both — the API gateway in front of applications, the AI gateway in front of the models.
Where Ultracache sits
Ultracache is the routing and caching layer of that control point, not a middleman in front of it. It inherits the identity, authorization, isolation, metering and audit trail of the gateway you already govern, and it changes nothing about how your applications call models: one connection point, your own model names, the answer returned under the name that was asked for. Credential custody is its own discipline — credential mediation issues agents short-lived, scoped credentials so no long-lived provider key ever sits in an agent. Teams that need the whole control point inside their own network run it against models they host themselves; the practicalities are in the guide to deploying an LLM on-premise, and the trade-offs between candidate models are in the LLM selection guide.
One siding. One workload. Your own numbers.
No platform bet and no big-bang rollout — a contained pilot with a clear decision point at the end.
Turn it on for one model about a week
Everything else runs unchanged. The step is reversible, non-disruptive, and fully in your platform team’s control.
Prove it on one real workload two to four weeks
Pick one repeat-heavy workload and define success up front: measurably fewer full-price repeats against the provider’s own pricing, a faster first response on long context, and a complete record of where each request went and why.
Decide with evidence
Your own numbers against real provider pricing. Expand model by model, or stop — nothing locked in, applications unchanged throughout.
Ultracache is one layer of Iternal’s agentic platform — it pairs naturally with Ultramemory and the rest of the product line, each adoptable on its own.
Straight answers, no hedging.
Is Ultracache storing our AI’s answers?
No. Ultracache reuses repeated context, not answers. The provider is always called and always returns a fresh response written for that exact request. What gets reused is the processing of the material you send over and over — the standing instructions, the reference documents, the conversation so far. That repeated context is processed once and reused while it stays warm, instead of being re-billed at full price on every turn. Nothing about the answer itself is stored or replayed.
Where do the savings figures come from?
Every figure on this page is published by the provider, not measured by us. Anthropic publishes up to 90% cheaper reused input; OpenAI publishes approximately 50%; provider documentation cites up to roughly 85% faster first responses on long, repeated context. Iternal publishes no measured savings or speed benchmark. What you realize depends on your provider’s current pricing and how much of your workload is repeated context — and usage is metered exactly as it is today. A pilot on your own traffic produces your number.
Do our applications have to change?
No. Your applications call one connection point using your own model names, exactly as they do today, and the answer always comes back under the name they asked for. They never learn which provider served them, and no provider names, model mappings, or caching behavior get written into your code. That is what makes providers portable: switching one out, or mixing several, becomes a configuration change owned by your platform team rather than an engineering project across every application.
What if we turn it on and something breaks?
It ships off by default, and with it off, behavior is identical to the prior release — a guarantee enforced by the test suite. Three gates must agree before anything activates: a global off switch, a per-model opt-in, and a per-route opt-in. When it is on, it only re-ranks routes your platform already validated, so it cannot introduce a new way to fail; anything that failed before fails identically now. One operation returns you to the baseline.
Doesn’t one shared layer create a new place for data to leak?
The identity handed to providers is a sealed, organization-scoped stamp — it identifies your traffic without revealing anything inside it. Raw request content, and any hints your applications send, are never stored, never logged, and never forwarded. On the standard path, organizations sharing the platform cannot collide with or probe one another. And this is not a third-party middleman: it is a native layer of a gateway you already govern, inheriting its identity, authorization, isolation, metering, and audit trail.
What is the difference between an AI gateway and an API gateway?
An API gateway routes HTTP by path and method, authenticates the caller and rate-limits by request count; by design it does not care what is inside the payload. An AI gateway does that same policing for model traffic and adds the work that only makes sense there: choosing which model answers and failing over when one degrades, keeping repeated context warm so it is not paid for twice, accounting in tokens and spend rather than request counts, and recording where each request went and why. Most estates run both — the API gateway in front of applications, the AI gateway in front of the models. Ultracache is the routing and caching layer of that AI gateway, inheriting the identity, authorization and audit trail you already govern rather than adding a middleman.
What does it take to try?
Turn it on for one model — about a week. Everything else runs unchanged, and the step is reversible and fully in your platform team’s control. Then prove it on one real, repeat-heavy workload over two to four weeks, with success defined up front: measurably fewer full-price repeats against the provider’s own pricing, a faster first response on long context, and a complete record of where each request went and why. Then decide with your own evidence. Nothing is locked in.
Stop re-buying the same context. Start compounding the advantage.
Every turn, you pay full price for context you already sent — and every hard-coded provider narrows your options further.
One governed layer turns that into a compounding cost advantage: no application changes, no new lock-in, no new failure mode.
Claims trace to the Ultracache engineering record as of July 2026. All cost and speed figures are provider-published pricing or documentation, attributed to the named provider — not Iternal-measured results. Figures labeled illustrative are concepts, not measurements.