seldon
v0
Log inSign up
Docs menu

Gateway

Response caching

Opt in per request to exact or semantic response caching, control its lifetime, and read cache accounting in usage.

Response caching is off unless a request opts in with a cache object. Sending the object is the consent; workspace settings cannot turn caching on for requests that don't include it.

Turn it on

Python
response = client.chat.completions.create(
    model="openai/gpt-5.4-nano",
    messages=[{"role": "user", "content": "What is our refund window?"}],
    extra_body={"cache": {"mode": "exact", "ttl_s": 3600}},
)
FieldValuesDefault
modeexact, semantic or offexact when the object is present
ttl_sPositive integer, in seconds3,600. Values above 604,800 (7 days) are clamped.
similarity_thresholdNumber from 0 to 1, for semantic0.97
no_storeBooleanfalse. When true, the request neither reads nor writes the cache.
  • Exact matches requests whose response-affecting fields are identical: model, messages, parameters and the eligible provider set.
  • Semantic also matches paraphrased requests above the similarity threshold. Every semantic write also stores an exact entry.

What gets cached

  • Only complete, non-streaming responses whose choices finished with stop.
  • Not streaming requests, and not responses that contain tool calls.
  • Entries are scoped to your workspace and to the credentials that served them.

Reading a cache hit

A hit returns the stored response with usage.computed_cost set to "0" and a seldon_cache object describing the hit:

JSON
{
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 9,
    "total_tokens": 23,
    "computed_cost": "0",
    "seldon_cache": { "hit": true, "tier": "exact", "avoided_cost": "0.0000052" }
  }
}

Semantic hits add similarity. Cache hits are free, so they are served even when the workspace has no available credit.

On a miss, usage.seldon_cache reports provider prompt-cache accounting instead: cached_read_tokens, cache_write_tokens and cache_hit_ratio.

Provider prompt caching

Separately from response caching, providers discount repeated prompt prefixes. Seldon keeps a prefix on the same provider where it can (provider.sticky, on by default) so those discounts apply. See Models and routing.

Privacy

Cached entries store the response body. Use cache.no_store: true, or omit the cache object, for requests whose responses must not be stored. See Data handling.