seldon
v0
Log inSign up
Docs menu

Gateway

Models and routing

Choose models by ID, let Seldon pick with auto, add fallbacks and control which providers serve a request.

A model in Seldon can be served by more than one provider. You choose the model; Seldon chooses among the providers that offer it, tries them in order and falls back when one fails.

Model IDs

Use the provider-prefixed ID from Models, such as openai/gpt-5.4-nano. The part after the first / also resolves, so gpt-5.4-nano works when it is unambiguous. List the models available to your workspace with GET /v1/models.

Let Seldon choose: auto

"model": "auto" considers every routable model, applies your provider filters and policies, and orders candidates by provider.sort. Use it when any capable model is acceptable.

Fallback models

models lists fallbacks. Seldon tries model first, then each entry in order:

Python
response = client.chat.completions.create(
    model="openai/gpt-5.4-nano",
    messages=[{"role": "user", "content": "Summarize this ticket."}],
    extra_body={"models": ["openai/gpt-4.1-mini"]},
)

Provider preferences

The provider object controls which providers may serve the request and in what order.

FieldTypeEffect
orderarrayTry these providers first, in this order. Others follow.
only, allowarrayAllowlist. If you send both, only providers in both lists qualify.
deny, ignorearrayExclude these providers.
sortstringprice, latency or throughput. Without it, Seldon uses catalog order.
allow_fallbacksbooleanDefault true. With false, only providers in order or the allowlist are used.
stickybooleanDefault true. Prefer the provider that last served the same prompt prefix, to reuse provider prompt caches.
max_priceobjectExclude offerings above input_per_1m or output_per_1m USD.
zdrbooleanRequire zero-data-retention providers.
data_collectionstring"deny" excludes providers that may train on data.
data_residencyarrayKeep providers whose regions overlap these values. Providers with unknown regions are excluded.
JSON
{
  "model": "openai/gpt-5.4-nano",
  "messages": [{ "role": "user", "content": "Hello" }],
  "provider": {
    "order": ["azure-openai", "openai"],
    "max_price": { "input_per_1m": "1.00", "output_per_1m": "4.00" }
  }
}

Provider slugs appear in each model's provider list in GET /v1/models.

How the controls combine

Filters run first: deny, allowlist, open circuit breakers, structured-output capability, max_price, then data policy. Ordering runs next: order, then sort. Sticky routing can move a provider to the front only if it survived every filter and costs no more than 5% above the cheapest candidate.

Price caps

max_price.total_estimated is accepted only together with input_per_1m or output_per_1m, because Seldon has no token estimate before dispatch to enforce a total on its own. When no offering fits the cap, the request returns 402 price_cap_exceeded.

Data policy

zdr, data_collection and data_residency rely on provider policy metadata. When that metadata is unknown, the provider is treated as non-compliant, so strict policies can return 422 no_policy_compliant_provider. A workspace may enforce stricter defaults; a request cannot loosen them (422 data_policy_loosening_forbidden). See Data handling.

Reasoning controls

Seldon accepts both OpenAI-style reasoning_effort and the reasoning object:

  • reasoning_effort or reasoning.effort: none, minimal, low, medium, high, xhigh or max.
  • reasoning.enabled, reasoning.exclude and include_reasoning: booleans.
  • reasoning.max_tokens: a positive integer.

Contradictory combinations, such as reasoning.enabled: false with an effort other than none, return 400 invalid_request. Models apply only the efforts they support.