Response cache

The response cache answers a repeated chat completion from storage: no provider call, no hold, no charge. It is an exact-match cache, off by default, and turned on per inference key — a choice made for the key, not for the organisation. It is not the providers’ prompt caching, which prices repeated prompt prefixes and shows up as cached_tokens (Models & providers lists those prices).

Turning it on

{"cache_enabled": true, "cache_ttl_seconds": 3600}

cache_enabled and cache_ttl_seconds (60 to 86,400; default 3,600) on POST /v1/keys or PATCH /v1/keys/:hash, or the Cache button on the Keys page. Inference keys only. Turning it off stops lookups at once; what is stored expires on its TTL, or goes now with DELETE /v1/cache.

What a hit is

A non-streaming chat completion identical to an earlier one from the same organisation: the same model as asked for and the same body, apart from user, metadata, stream, stream_options, session_id, routing_stability and prompt_cache_key, which change nothing the model answers. A field we are unsure about makes two requests differ, which costs a miss, never a wrong answer. The lookup runs after every gate and routing filter, so a hit is refused for whatever a miss would be refused for, and is served only while routing still reaches the model that answered.

• x-cache: HIT — answered from the cache: usage.cost 0, usage.cached: true, the original’s token counts, under a new generation id that Logs and the broadcast record mark as a hit.
• x-cache: MISS — answered by the provider and stored, unless the answer ended in error or is over 1 MB.
• x-cache: BYPASS — the request sent x-cache: bypass: answered by the provider, nothing stored.
• No header — the request was not cacheable.

• The same answer, whatever the temperature. Turning the cache on is choosing that.
• Never cached: streamed requests, server tools, MCP tools and connected tools (declared, or a key’s connected_tools default), any plugins (web search, file parsing, moderation, response healing, the auto-router), n above 1, a run_id, and any message part that is not text (images, files, audio).
• Limits. One organisation only; answers over 1 MB are not stored, nor beyond 10,000 live entries per organisation.

The organisation’s view

GET reports available, entries and bytes. DELETE empties the cache for every key (Owners and Admins, or Settings): the next request on each key costs what a miss costs.