Evaluation

Every function has one Evaluation card. During the preview, Seldon prepares an evaluation from your instructions and output schema. The default needs no action from you; Edit and Test are optional.

What the evaluation checks#

New evaluations return pass or fail with a short explanation for each output. They combine:

• Built-in checks for the output schema, collections, spans and quotations grounded in the source input, allowed values and any expected answers you supply.
• Semantic judgment of correctness and completeness against the source input and instructions. The judge is the pinned reference model, called at temperature zero. Each mandatory criterion is scored from 1 to 5, and an output passes only when every one scores at least 4.

Because outputs are judged against the source and instructions, a candidate can pass on an input where the reference answer is wrong.

Before results are trusted, Seldon judges 20 frozen outputs twice and requires 95% pass/fail consistency. If the check fails, the comparison report marks evaluation as unreliable and omits quality ranking.

Edit the criteria#

Choose Edit and describe success in plain language under How should success be evaluated? (up to 8,192 characters), then Save evaluation. This changes evaluation for future work only; your task instructions stay the same and completed runs are not rescored.

Editing the evaluation of an active function creates a new function. The active one keeps serving unchanged.

Test examples#

Choose Test to check the evaluation on specific outputs before you optimize:

1. Enter Evaluation input JSON and the Output to evaluate JSON, or choose Generate a new output.
2. Optionally add the expected behavior under Correct this example or add expected behavior. Corrections apply only to that example.
3. Choose Run evaluation.

Tests run within the preview's compile cap. While a test runs, Optimize is unavailable; you can Cancel evaluation test.

Import examples#

Import examples accepts a JSON array or JSONL file with 1 to 400 examples, up to 5 MiB. Each example has an input object and optional output, expected and assertions fields:

{"input": {"message": "Ada Chen is an engineer at Acme."}, "expected": {"jobs": [{"name": "Ada Chen", "company": "Acme", "position": "engineer"}]}}
{"input": {"message": "Ben Park formerly worked at Acme."}, "assertions": ["Marks the Acme role as former, not current."]}

assertions are plain-language checks, up to six per example. The import is atomic: either every example is saved or none is. Imported corrections stay out of synthetic-data prompts and fresh held-out sets.

Next step#

Optimize