seldon
v0
Log inSign up
Docs menu

Functions

Compare and activate

Read the comparison report, try alternatives on your own inputs and activate the pipeline that fits your application.

A finished run publishes up to three executable pipelines beside your original call, the Reference LLM. Activation is always your decision; the report informs it but never blocks it.

Read the report

Each pipeline card shows its pass rate, cost per measured execution, p95 latency and whether it is Available to Try and Activate. Select a card to see the full measurement table against the reference:

RowMeaning
Evaluator pass rateOutputs that passed the evaluation. Denominators include unresolved judgments.
Scored inputs, Unresolved judgmentsHow many inputs were judged, and how many could not be. An unresolved judgment never counts as a pass.
Critical evaluation failuresOutputs with a critical finding.
Cost / measured executionAverage cost of successful executions.
p50 latency, p95 latencyMedian and tail latency of measured executions.
Execution failures, Refused requestsCalls that errored or that a provider refused.
Fallback usesExecutions that fell back to the original workflow.
Successful primary executionsExecutions completed by the pipeline itself, without fallback.

The Paired change table summarizes the selected pipeline against the reference on the same inputs: pass-rate difference in percentage points, and relative cost, p50 and p95. Missing values read Not measured, never zero.

Check the evidence line

Under the cards, a line states the evidence behind the numbers:

  • Synthetic examples or Recorded examples.
  • Held-out evaluation on inputs search never saw, or Search observations; not held-out evaluation.
  • Evaluator reliable or unreliable, and the number of paired observations.

When evaluation is unreliable or incomplete, a warning explains what the numbers do and don't establish. Cost and latency exclude failed executions and offline evaluation; total compilation charges stay in the run summary.

Try before you activate

Enter an input under Try input JSON:

  • Try · charges usage runs the selected pipeline once. Use Retrieve execution to fetch the result again without rerunning it.
  • Compare on this input runs the pipeline and the reference on the same input, with a cap of up to twice the per-call reservation.

Try results are not production evidence. Evaluate your whole application before switching traffic.

Activate

Choose Activate revision. A pipeline is activatable when its program and bindings are valid, its models are pinned to supported versions, and it completed at least one schema-valid primary execution. Fallback-only success does not count.

Activation:

  • makes the revision callable through the function's alias;
  • does not edit your code or move traffic on its own;
  • clears the 30-day expiry that applies to alternatives you have not activated.

Runs created before comparison reports keep their original qualification rules: only a qualified revision from such a run can be activated.

Next step

Call a function from your app

Compare and activate | Seldon