Retries and idempotency

What Seldon does for you#

Within one request, Seldon tries eligible providers in order and falls back when one fails, is rate limited or has an open circuit breaker. The whole request, including queueing and fallback, has a 55-second budget. If no response is ready by then, it returns 504 request_deadline_exceeded.

Use provider.allow_fallbacks: false to stop fallback beyond the providers you list. See Models and routing.

Which errors to retry#

Use exponential backoff with jitter and a retry limit.

Idempotency-Key#

Send an Idempotency-Key header to make retries of non-streaming requests safe. A retry with the same key and the same request returns the stored response instead of calling a provider again.

import uuid

key = str(uuid.uuid4())
response = client.chat.completions.create(
    model="openai/gpt-5.4-nano",
    messages=[{"role": "user", "content": "Draft a reply."}],
    extra_headers={"Idempotency-Key": key},
)

• Keys are 1 to 255 characters and scoped to your API key. Blank or longer keys are ignored, and the request runs normally.
• Completed responses are kept for 120 seconds. Responses over 64 KiB are not stored for replay.
• If the original is still running, the retry waits for it within the request budget.
• Reusing a key with a different body or correlation headers returns 409 idempotency_conflict.
• Streaming requests ignore the header.

Function calls also accept Idempotency-Key, but require a UUID. See Call from your app.

Mid-stream failures#

A stream that has already started cannot fall back. If the provider fails, Seldon sends an error event followed by data: [DONE]. Retry the request if your application can tolerate a repeated partial answer. See Streaming.