Streaming, tools and structured outputs

Seldon passes OpenAI Chat Completions features through to providers that support them. When a model cannot honor a requested feature, the request fails early with 400 feature_unsupported instead of silently dropping it. Check a model's capabilities on Models.

Streaming#

Set stream: true to receive server-sent events:

stream = client.chat.completions.create(
    model="openai/gpt-5.4-nano",
    messages=[{"role": "user", "content": "Write a haiku about routers."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

• Provider chunks are relayed as they arrive, and the stream always ends with data: [DONE].
• Seldon asks OpenAI-compatible providers to include usage, so a late chunk usually carries token usage. Streamed responses do not include computed_cost; find the cost in Calls & usage.
• The X-Seldon-Trace-Id header is set on the streaming response.
• If the provider fails mid-stream, Seldon sends one error event and then [DONE]:

data: {"error": {"code": "upstream_unavailable", "message": "provider stream failed"}}

data: [DONE]

Streaming requests are not cached and do not support Idempotency-Key.

Tools#

Send tools and tool_choice as you would to OpenAI. Tool calls come back in choices[0].message.tool_calls. Responses that contain tool calls are never stored in the response cache.

Images and other inputs#

Send images as image_url content parts. Audio and video inputs are passed to models that declare support for them. Requests with an input the model does not support return 400 feature_unsupported.

JSON and structured outputs#

response_format: {"type": "json_object"} asks for JSON. json_schema goes further: Seldon validates the provider's output against your schema before returning it.

response = client.chat.completions.create(
    model="openai/gpt-5.4-nano",
    messages=[{"role": "user", "content": "Extract the city: I live in Lisbon."}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "city",
            "schema": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
                "additionalProperties": False,
            },
        },
    },
)

• Only providers that support json_schema are considered. If none qualify, the request returns 422 structured_output_unsupported.
• If the output fails validation, Seldon asks again and may fall back to another provider. When every attempt fails, it returns 422 schema_validation_failed with detail.attempts; invalid JSON is never returned as success. Token usage includes every attempt.
• A malformed schema returns 400 invalid_response_schema.
• json_schema cannot be combined with stream: true (400 schema_streaming_unsupported).

Other parameters#

Standard parameters such as temperature, max_tokens, top_p, stop, seed and user are passed through to the provider unchanged. Seldon reads provider, models, cache and preset to route the request; see Chat completions for the full field list.