Docs menu

Functions

Evaluation

How Seldon judges a function's outputs, and how to adjust the criteria, test examples and import your own.

Every function has one Evaluation card. During the preview, Seldon prepares an evaluation from your instructions and output schema. The default needs no action from you; Edit and Test are optional.

What the evaluation checks

New evaluations return pass or fail with a short explanation for each output. They combine:

  • Built-in checks for the output schema, collections, spans and quotations grounded in the source input, allowed values and any expected answers you supply.
  • Semantic judgment of correctness and completeness against the source input and instructions. The judge is the pinned reference model, called at temperature zero. Each mandatory criterion is scored from 1 to 5, and an output passes only when every one scores at least 4.

Because outputs are judged against the source and instructions, a candidate can pass on an input where the reference answer is wrong.

Before results are trusted, Seldon judges 20 frozen outputs twice and requires 95% pass/fail consistency. If the check fails, the report marks evaluation as unreliable. The measurement table still shows the observed pass rates and says they do not establish quality. The tradeoff map does not plot that pass rate, and the paired pass-rate difference reads Not measured. The pipeline can still be activated when it is otherwise eligible. Prompt refinement is skipped for that run.

Edit the criteria

Choose Edit and describe success in plain language under How should success be evaluated? (up to 8,192 characters), then Save evaluation. This changes evaluation for future work only; your task instructions stay the same and completed runs are not rescored.

Editing the evaluation of an active function creates a new function. The active one keeps serving unchanged.

Test examples

Choose Test to check the evaluation on specific outputs before you optimize:

  1. Enter Evaluation input JSON and the Output to evaluate JSON, or choose Generate a new output.
  2. Optionally add the expected behavior under Correct this example or add expected behavior. Corrections apply only to that example.
  3. Choose Run evaluation.

Tests run within the preview's compile cap and reserve the remaining cap while they run. While a test runs, Optimize is unavailable; you can Cancel evaluation test. Unused cap is released when the test ends.

Import examples

Import examples accepts a JSON array or JSONL file with 1 to 1,000 examples, up to 5 MiB. Each example has an input object and optional output, expected, assertions, value, weight, group and role fields:

JSONL
{"input": {"message": "Ada Chen is an engineer at Acme."}, "expected": {"jobs": [{"name": "Ada Chen", "company": "Acme", "position": "engineer"}]}}
{"input": {"message": "Ben Park formerly worked at Acme."}, "assertions": ["Marks the Acme role as former, not current."]}

assertions are plain-language checks, up to six per example. The import is atomic: either every example is saved or none is. Imported corrections stay out of synthetic-data prompts and fresh held-out sets.

Task metric

When at least one example has an expected answer, the Evaluation card shows Task metric. It is optional. None (judge only) keeps the rubric above.

A new run can save Exact match, F1, or Value-weighted recall. The menu also lists other metrics, including mean absolute error and macro F1. Saving one of those returns evaluation_fidelity_disabled until that capability is enabled. A saved task metric scores labeled outcomes. The judge remains available for runs without one.

A judge pass rate, a saved task metric and an application result answer different questions. The first two describe outputs on Seldon's evaluation examples; neither substitutes for scoring your full application on a separate held-out set. Keep that set out of the examples you import, do not use it to choose an alternative or retune the scorer after seeing its outcomes, and report failed and unresolved cases alongside the metric.

Next step

Optimize

Evaluation | Seldon