← Blog
November 19, 202611 min readSeldon

The state of decision models, Part 2: Your LLM call is doing more than you think

Semantic ComputeFunctionsDecision ModelsEvaluation
A small ivory cube casts an impossibly intricate architectural shadow across a charcoal surface, suggesting the hidden complexity behind a simple interface.

Audiala replaced an LLM router with a decision model and rolled it back 23 minutes later. The replacement could classify intent. The incumbent also resolved cities, conversational references, and tool-control flags. The team had evaluated one responsibility and replaced several.

The migration account was one of the clearest examples in Part 1, our survey of decision-model adoption. Other integrations exposed related problems: candidate construction constrained action selection, missing handoff instructions weakened downstream agents, and faster decisions sometimes produced more recovery work. Those reports describe particular implementations, not a universal penalty for specialization.

They do suggest a practical starting point: the unit of replacement is the behavior the application depends on. This article follows that idea into the implementation. We will recover a call's contract, construct alternatives, evaluate the complete replacement, and account for keeping it reliable.

A small output can hide a large responsibility

Consider an illustrative invoice-processing application with this interface:

resolve_vendor(invoice, catalog) → vendor_id

It appears to select a vendor. But which entity counts as the vendor: the issuing legal entity, a trading name, or the parent company? What happens when two catalog entries share a name? Does a tax identifier override the letterhead? Can the function return an entity absent from the catalog? Does a successful result merely populate a review screen, or does downstream code use it to choose a payment destination?

The original prompt may have absorbed several of those responsibilities. It might interpret the invoice, reconcile conflicting evidence, identify an entity, and explain uncertainty in a field the next step reads. A replacement that returns a valid identifier can still lose behavior the application needs.

Start with the consumers. Inspect field access, database lookups, tool arguments, intermediate messages, and recovery paths. Read the prompt alongside those dependencies. A field ignored by the immediate caller may still become evidence in a later model call. Conversely, a requested explanation nobody consumes may not belong in the required output.

Part 1's LAS case makes this concrete: planners supplied handoff instructions, and reviewers supplied repair guidance. A finite choice could replace one judgment while removing information needed downstream. If that information remains necessary, preserve it or assign its production to another step.

Illustrative replacement review

One return value, several responsibilities

Invoice + catalogresolve_vendorvendor_id

Required behavior

Proposed choice-only replacement

  • Interpret the invoiceStill unassigned
  • Identify the issuing entityStill unassigned
  • Reconcile conflicting evidenceStill unassigned
  • Choose a catalog candidateAssigned to choice model
  • Preserve evidence and uncertaintyStill unassigned
Returning the same field is insufficient when the application also depends on evidence, resolution, and uncertainty handling. Unassigned responsibilities still need an owner in the model or surrounding code.

Recover the contract before choosing a model

For our example, suppose the application owner confirms that the function must resolve the issuing legal entity within a supplied catalog snapshot, preserve supporting evidence, and refer unresolved ambiguity for review. A conceptual result type could be:

Resolution =
    Matched(vendor_id, catalog_version, evidence)
  | NeedsReview(reason, candidate_ids)
  | NoMatch(reason)

Each outcome needs semantics. Matched requires a member of the supplied catalog and evidence for choosing it. NeedsReview includes conflicting identifiers, ambiguous entities, and insufficient evidence. Here, NoMatch means the available evidence establishes that no catalog entry qualifies under the matching policy. An empty shortlist alone cannot establish that; retrieval may have missed the answer.

This changes the caller too. It must handle review and no-match outcomes explicitly instead of treating every syntactically valid identifier as success. Existing authorization checks still decide what a match permits the application to do.

Keep three standards separate: matching the output schema, agreeing with the incumbent, and meeting the application's requirements. Agreement can preserve an incumbent's mistakes. Historical traces help identify common inputs and dependencies, but cannot establish every required behavior. A previously accepted wrong vendor is evidence of a failure, not a new specification.

Record which requirements came from application policy or adjudicated examples and which remain inferred. Resolve consequential ambiguity before measuring candidates. Calling this work “refactoring” has a limit: learned replacements can change answers and error distributions. The goal is to meet declared requirements with evaluated tolerances; universal equivalence to the incumbent is usually not established.

Construct alternatives around the responsibilities

Begin with the cheapest experiment to implement: send the existing request to a cheaper general model. A useful result may require no decomposition. Compare it with alternatives that make the recovered responsibilities explicit:

ImplementationWhat changesMain assumption to test
Same prompt, cheaper modelModel choiceThe replacement handles the full contract
Extraction, retrieval, specialist matchingEvidence and candidate selection become separate stagesExtraction and retrieval preserve the information needed to match
Guarded exact lookup, then semantic matching for unresolved inputsSome requests avoid learned matchingThe identifier is authoritative, unique in scope, and acceptable under the matching policy

An exact lookup is justified only under those preconditions. Text that resembles a tax identifier is not automatically authoritative. A catalog alias can also be shared by several legal entities. The implementation needs an unresolved path when its assumptions fail.

The candidate set is part of the program

Part 1's Stagehand example showed why action selection depends on candidate construction. The same issue appears here: a specialist cannot select the correct vendor if retrieval excludes it. For cases with a correct catalog match, on a path that must choose from its retrieved candidates:

P(correct match)
  = P(correct vendor retrieved)
    × P(correct selection | correct vendor retrieved)

Illustratively, 95% retrieval coverage and 98% conditional selection accuracy yield 93.1% correctness on that population. These are hypothetical numbers, but the implication is operational: evaluate retrieval separately and together with selection. A strong classifier score measured only on lists containing the answer hides an upstream failure mode.

Illustrative arithmetic · not benchmark results

The selector cannot recover a missing candidate

1,000 invoices

Each has a correct vendor in the catalog. This path must choose from retrieved candidates.

950 · correct vendor retrieved

95% retrieval coverage

931 · correctly selected

98% of the 950 retrieved cases

19 · wrong or unresolved

2% of the 950 retrieved cases

50 · correct vendor omitted

The correct match is impossible on this selection path. Review or fallback may recover it.

Example: V17 exists in the full catalog, but the shortlist contains only V42 and V58.

931 / 1,000 = 93.1% correct on this path

A 98% selector achieves 93.1% correctness here because retrieval excludes the answer in 5% of cases. Omitted candidates may lead to review or fallback; they are not all wrong automatic matches.

Preserve candidate identities through the stages. If the model evaluates pairs, attach each judgment to a pair ID rather than assuming responses retain their order. Similarly, extracting a list of companies and a list of addresses does not establish which address belongs to which company. Those relationships need evidence and explicit representation.

Separate judgment from mechanical work

Some responsibilities can become ordinary computation once their inputs are explicit. A model can identify relevant line items while a decimal-arithmetic kernel computes their total, or quote supporting text while a grounding kernel locates it in the source. In vendor resolution, deterministic operations can construct candidate pairs, preserve their IDs, and assemble match decisions without guessing relationships from list order. The model interprets evidence; code performs the declared transformations. These operations also make intermediate results available for inspection and testing.

Determinism does not establish that the interpretation was correct. An accurately located quotation can carry the wrong label, an exact name match can identify the wrong legal entity, and a correct sum can use incorrectly extracted operands. Failure policy changes the result too: rejecting an ungrounded quotation may trigger expensive fallback, while dropping it may omit something important. For redaction, that omission could expose sensitive information. Evaluate those choices as part of the replacement, including completeness and downstream errors; deterministic operations remove particular failure modes, not the need for semantic evaluation.

Evaluate the complete replacement

Define the acceptance criteria before tuning the candidates. In invoice matching, wrong automatic matches, unnecessary reviews, and unresolved invoices have different consequences. Count them separately. A higher aggregate score does not decide whether extra incorrect matches are acceptable in exchange for fewer reviews.

Use three layers of checks:

LayerWhat it establishes
Contract checksOutputs have the required fields, identities refer to the correct catalog, and unresolved outcomes are handled
Paired function evaluationComplete implementations are compared on the same inputs for correctness, consequential errors, unresolved results, cost, and latency
Workflow executionThe changed output works through subsequent state transitions, recovery, and task completion

Build examples around the failure mechanisms, not only common successful invoices. Include a correct vendor omitted from retrieval, similar parent and subsidiary names, conflicting identifiers, a vendor absent from the catalog, and insufficient evidence. Reorder candidates and vary equivalent wording to test whether incidental presentation changes the answer. Include untrusted document text that attempts to direct the model; schema validation does not establish resistance to such instructions.

Use development data to construct candidates and tune prompts or thresholds. Freeze the chosen implementation before final held-out evaluation. Group related invoices or templates when necessary to prevent near duplicates crossing the split. If the intended claim is generalization to unseen vendors, the split must test that too. Repeated runs measure variability on the same examples; they do not create additional task coverage.

Historical answers can support disagreement analysis, but meaningful correctness claims need an appropriate authority: adjudicated matches, trusted identifiers, or verified outcomes. Report the source of the labels and uncertainty in consequential error estimates, especially when important failures are rare. Zero observed errors on a small set does not establish a zero error rate.

Replay does not cover every trajectory

For a bounded resolver, replay against captured invoices and catalog snapshots is useful. For an agent controller, replaying incumbent traces evaluates states the incumbent reached. A changed action can lead to different states, arguments, and recovery paths that never appear in those traces.

Part 1's complete-task comparisons illustrate why this matters: improved component speed or median episode time did not consistently preserve success and tail latency. Exercise the affected workflow in a controlled environment with external effects contained. For the invoice application, follow ambiguous results into review and failed matches into recovery; for an agent, execute the subsequent trajectory. Measure completed tasks and retain failed attempts in the accounting.

Illustrative workflow · not observed results

Replay stops where the new trajectory begins

Shared state: ambiguous invoice

Replay can compare the two implementations’ first decisions here.

Recorded incumbent path

Request clarification

Keep the invoice unresolved

Receive additional evidence

Captured in this illustrative trace

Resolve with new evidence

Recorded continuation is available

Replacement path

Accept vendor V42

A different first decision

Different downstream state

No clarification was requested. The recorded response cannot be assumed to occur.

Outcome requires execution

Measure completion, recovery, and errors in a controlled workflow.

Replay compares decisions at recorded states. A changed decision can lead to states the recording never contains. Dashed boxes mark continuations that require execution to evaluate.

Acceptance, fallback, and permission are separate decisions

A confidence threshold answers only part of the deployment question. Distinguish three decisions:

  • Applicability: are this implementation's assumptions satisfied, including input format and catalog scope?
  • Acceptance: does this particular result meet the policy for automatic use?
  • Authorization: may the application perform the resulting action?

A model-generated assertion that a customer granted permission is not a substitute for trusted authorization state. Part 1's uncertainty and trust cases show why high confidence cannot supply missing instructions or protect the evidence channel.

Evaluate an acceptance policy using both coverage, the fraction of requests it accepts, and error among accepted answers. Accepting very few cases can make the specialist look accurate while sending almost everything to an expensive fallback. Also report errors after fallback, unresolved outcomes, total cost, and latency. The policy's complete behavior determines its usefulness.

Fallback cannot repair an error the system accepts. It helps when a detected failure or rejected result sends the request elsewhere. A fallback model can still share the original misunderstanding. Where practical, let it inspect the original evidence; sending only the incomplete shortlist can preserve the cause of the error.

Illustrative failure path

An accepted error never reaches fallback

The model selects the wrong vendor: V42

Known to the evaluator; not detected by this runtime policy.

Runtime checks

  • Valid output schema PASS
  • V42 belongs to the catalog PASS
  • Acceptance threshold PASS

Accepted · this example

Wrong match reaches the application

Fallback never runs

Separate application checks

Authorization may block an action. If permitted, the wrong match may cause a downstream error.

Rejected · alternate path

Run fallback

Only after a detected failure or rejected answer

Fallback can also return a wrong or unresolved answer. It needs evaluation too.

A semantically wrong answer can pass structural and acceptance checks. If it is accepted, fallback never runs. Authorization is a separate application check and does not establish that the vendor identity is correct.

Specify boundary behavior: a score exactly at the threshold, missing fields, invalid identities, timeouts, and an exhausted fallback budget. Part 1's Fastbrowse threshold issue is a reminder that a small branching mistake can defeat an otherwise careful evaluation. These cases deserve direct tests, independent of average model accuracy.

Count the cost of keeping the result valid

Compare economics among implementations that satisfy the required outcome constraints. For a defined workload and period:

Net benefit over the period
  = execution savings
    − migration and evaluation cost
    − maintenance and revalidation cost

Execution savings must include both specialist attempts and fallback, retries, serving overhead, and any changed review or recovery work. Distinguish cost per attempt from total spending divided by successful tasks; the latter keeps failed attempts in the numerator. A high-volume stable function may recover its setup cost quickly. An occasional or rapidly changing task may not.

Treat a measured revision as more than a model name. Record its prompt, schemas, candidate builder, catalog or index version, acceptance policy, evaluation data, and evaluator. Changing retrieval can change candidate difficulty and score distributions while leaving the model untouched. The earlier threshold evidence may no longer support the new implementation. Check which assumptions changed and rerun the relevant evaluation before relying on the old result.

Instrument the revision and branch that actually execute, including fallback and subsequent recovery. Part 1's integration-path reports show why this matters: a host change can bypass the measured implementation without changing its model. Retain a usable previous revision and an explicit rollback path. New measurements can inform a deployment decision without automatically replacing the implementation currently serving requests.

From a measured replacement to a maintained function

These requirements motivate Seldon's view of Semantic Compute: an application should define a semantic function whose implementation can evolve under an explicit contract and measured tradeoffs. Today, Seldon Functions starts with a developer-declared request or supported short workflow, searches supported alternatives—including model calls combined with predefined deterministic operations—and hosts the revision the developer chooses to activate. Its measurements concern that bounded function; the surrounding application still needs its own validation.

In the next article, we'll follow one Function through that process: freezing its contract, constructing executable candidates, separating search from held-out evaluation, and serving a selected revision with bounded fallback. We'll examine what the evidence establishes, where it remains incomplete, and when keeping the original implementation is the right result. The practical test is whether this reduces the work required to reach and maintain an acceptable replacement.

Open beta

Route your traffic through Seldon and investigate recurring work that could benefit from a specialized program.

Sign up for open beta →
The state of decision models, Part 2: Your LLM call is doing more than you think | Seldon