Observational reconstruction, then the inverse problem

We capture traces so we can stop paying frontier prices for the same data workflow, run thousands of times a day, wearing a prompt. Reconstructing those traces is necessary and mostly a known engineering problem. Turning them into a compiled route is not: many latent programs can produce the same prompt/response pair, and a cluster is not a license to serve.

Seldon’s bet is simple to state. A large share of production LLM calls are not irreducibly open-ended reasoning. They are repeated behavioral contracts — classify this ticket, extract these fields, validate this schema — that a cheaper plan could serve if you could find them, prove a substitute, and fall back to the provider when you cannot. The developer experience is deliberately boring: change base_url, keep an OpenAI-compatible client, get routing, cost accounting, and traces. Over time the same gateway becomes an optimisation layer.

That bet fails in two different ways, and the industry has mostly solved only the first.

The first failure is blindness. If you cannot reconstruct what crossed the wire — streamed tool calls, schema re-asks, which attempt was served — you cannot attribute cost per task, you cannot build an eval set, and you cannot tell an incident review what the model actually did. LangSmith, Langfuse, Helicone, Phoenix, Portkey, and OpenAI Traces all implement this physics. Seldon’s router and Import Audit sit in the same family. This post treats that layer as a prerequisite, with enough mechanism that you can see where audits silently lie.

The second failure is false authority. Once the exchanges exist, it is tempting to hash a prompt template, embed a summary, or ask a model to label the “workflow,” then compile the cluster. We have written as if that were enough: contract-first clustering asked which traces are the same replaceable program; program synthesis treated the cluster as the specification. Both remain useful as discovery. Neither is identity, and neither is serving authority. A sealed measurement on production Live traffic, plus a sequence of feasibility spikes, made that precise.

This matters beyond Seldon. Any team that wants to cache, route, distill, or compile from traces — an agent gateway, an eval harness, a prompt-to-ETL research stack — eventually has to answer: same contract, or merely similar text? Get that wrong and you merge two users, split one job into twelve “opportunities,” or serve a compiled path to a request it was never qualified for. Fail-open to the provider is the right runtime when you cannot tell. It is not a theory of identity.

What we are trying to do

The loop we want an engineer to trust:

Success is not “we recovered the hidden agent graph.” Success is: this request shape has been seen often enough to matter; this Candidate reproduced the oracle on held-out examples under named gates; this workspace chose to serve it; anything that does not match falls through to the model the application already called.

The shipped product already walks that loop. Live Audit groups traces. An Optimization Case freezes one group as a Candidate, evaluates it, and — only after human-gated adjacent steps — binds @signature/{digest} in shadow, canary, or live. Import Audit does the same discovery on Langfuse, LangSmith, or Braintrust JSONL, in an isolated scope that must never write the live routing map.

What the loop still compresses, and what this post is about, is the key. Today grouping, Candidate membership, compile topology, and serving all hang off one LLM-derived digest. That is the implementation we run. It is also the load-bearing mistake the rest of the challenges explain.

Why the motivation is economic, not cosmetic

A chatbot turn is typically one inference. An agentic task is a loop: plan, propose tools, consume results, retry, sometimes hand off. Practitioner bands put simple agents around five to fifteen LLM calls per completed task; research and coding agents run higher (Vortenza’s 2026 agent cost breakdown).

Call count understates the bill. Each subsequent call re-sends the static prefix (system prompt, tool definitions) plus accumulating conversation and tool results. If N is model turns, S the static prefix, u new text per turn, and r average tool-result size:

total_input_tokens ≈ N·S + u·N(N+1)/2 + r·N(N-1)/2

Naive history resend grows roughly with N². Tian Pan’s five-step climb 888 → 3,400 → 8,900 → 14,200 → 18,900 is a concrete instance (The token economy of multi-turn tool use). The claim you can verify on your own traces is narrower: tokens per completed task rise faster than calls per task.

Two consequences follow.

First, capture bugs multiply. A streaming stub that blinds one chatbot reply blinds one row. The same bug on an agent blinds fifteen billed inferences and shatters cost-per-task. Reconstruction quality is FinOps infrastructure, not a pretty session UI.

Second, the compile target is usually an inner hop, not the whole agent. The outer loop may still need a frontier model for planning and tool choice. The hop that extracts invoice fields on every ticket is the thing you might replace — if you can recognise it across sessions, users, and paraphrases. Session keys keep two users apart. They do not tell you those users are doing the same paid work.

That is why this is a general compiler problem, not an observability feature request. Process mining reconstructs cases. We need to identify workload contracts under a replacement policy.

Challenge 1 — Reconstruct the observation without lying

Engineers overload “replay.” Observational reconstruction asks: given stored evidence, what crossed the instrumented boundary? Execution time-travel asks: can I fork the agent from a checkpoint? LangGraph checkpointers do the latter; nodes re-execute and can diverge (time-travel docs). A payload log is not a checkpointer. If your “replay” never calls a model again, you are reconstructing. That is all this section claims.

The industry data model is a session of turn-level traces, each a tree of typed spans.

The session identifier is chosen by the application, not derived from the API key or wall-clock proximity. A turn is a tree: flattening it into “prompt / completion” destroys tool causality. Without the session key you have a bag of traces. Langfuse’s rule of thumb is the right one: one chatbot turn becomes one trace; the conversation becomes one session (good-trace FAQ).

Every reconstruction is relative to a boundary.

A gateway sees POST /v1/chat/completions. It does not see in-process tools, private RAG, or MCP unless the result later appears as role="tool". An SDK can wrap local functions. “We log all LLM traffic” is not “we reconstructed the agent.” Resent messages arrays are a claim by the client, not proof of what you previously served — clients truncate, summarise, and drop tool results.

The failure modes that actually destroy audits are ordinary, and they have to be named as statuses rather than repaired in silence.

Streaming. Relaying SSE correctly while persisting only a usage stub is the classic lie: the product looks fine, every stream is an analysis error. The fix is incremental assembly before yield: merge id/model when first seen, append delta.content, concatenate delta.tool_calls by (choice_index, tool_index) — providers split one argument JSON across many chunks — record finish_reason and terminal usage, persist complete | stream_error | client_disconnected | partial | truncated. Never materialise an empty assistant message and call capture complete. Tool-call-only outputs (content: null, finish_reason: tool_calls) are valid evidence; requiring output_text drops the program’s next step.

Tools. Join role="tool" rows by tool_call_id. If the model proposed call_abc and no result appears, the row is result_not_observed. A later prose sentence that mentions a balance is not the tool return value.

Attempts versus the served response. Schema re-asks and provider fallbacks are extra upstream calls for one client request. Showing only the winner hides the cost. Keep client request and effective attempt request distinct when the gateway inserts a correction turn.

Two ledgers. Provider success and capture success diverge. Billing follows the provider. Audit completeness follows capture. Collapsing them produces the report in which a successful, billed call “did not happen” because a blob write timed out.

Identity heuristics are not identity. API key, time adjacency, and overlapping history hashes collide under concurrent users and shared service accounts. Missing session headers yield unlinked or inferred, never a silent merge.

Seldon’s Live router implements this contract: ordered messages after redaction, assembled streams, attempt records, optional conversation headers, source usage on the full population of successful provider calls even when a subset fails analysis. Import Audit is a second seat — vendor JSONL, coverage receipts for missing I/O, vendor-reported cost left unknown when absent — isolated from Live billing and routing.

That is the prerequisite. It is also where most LLMOps products stop. The compiler cannot.

Challenge 2 — The observation does not identify the program

Let W be the latent task graph that actually ran (or that a Candidate might run), X the input, O the captured prompt/response. The forward map is W + X → O. Grouping and compilation attempt O → W.

In general that inverse is not unique. Distinct graphs produce the same JSON:

• extract fields, then normalize dates, versus one fused typed-generation step
• validate against schema in-process, leaving no extra span on the wire
• two topological orders of the same operators
• a shorter pipeline that is observationally equivalent to a longer one

A classifier can be made repeatable (temperature=0, fixed seed, structured output). Repeatability is not identifiability. Seeded JSON constrains shape, not membership. A committed invoice-shaped probe on our signature model alternated C.key_value_field_extraction with the same step plus F.date_currency_unit_normalization (stability 0.667). When the two calls agreed, they agreed. They did not prove that pipeline executed.

Parsimony — “shortest ordered subgroup list that explains the output” — is a canonicalization heuristic. It is not a discovery theorem. Switching to an unordered 41-task set plus a frozen display order would remove permutation noise only after you have chosen a policy. It still would not recover latent W.

What you can do without identifying W: fingerprint declared and observed wire facts; store versioned interpretations labelled as interpretations; qualify a concrete Candidate against a contract by behavior. Live Audit today does the first two in compressed form and treats the third as “held-out members of this digest.” That compression is Challenge 3.

Challenge 3 — One hash is being asked to do three jobs

A compiler intake has to answer three questions that are not the same relation.

Non-transitivity is the part clustering papers skip. Candidate X may qualify for contracts A and B, Candidate Y for B and C, with nothing qualifying for both A and C. A “replaceability cluster” would mark A and C as the same program. They are not.

v1 compresses those relations into one digest:

workload_signature_digest
  = sha256("workload-signature-output-v1" ∥ inferred_pipeline ∥ canonical_output_schema)

The inferred pipeline is a 41-task ordered string from a model call. The schema is derived from this trace’s JSON (keys, types, nullability, bounded-enum flags) with values stripped. Grouping, Candidate freeze, DAG node sequence, and @signature/{digest} all use that hash.

The UI already tells the truth the key does not: interpretation_authority = model_inferred, execution_observed = false. Pattern rollups in Audit hash the ordered task ids only and add no serving authority. The load-bearing path still keys off the digest. Digest equality therefore claims, inside the group, a replaceability the evidence did not earn.

Challenge 4 — The serve lookup is not a function

Three different facts are in play, and they are assigned to different roles.

Work a support-router example. Same system prompt, JSON Schema enforced:

user:  order #A-4471 arrived cracked. I paid $59.90
out:   {"intent":"refund_request","order_id":"A-4471","refund_amount":59.90}

user:  order #B-8820 never showed up. refund $12.00
out:   {"intent":"missing_package","order_id":"B-8820","refund_amount":12.00}

After masking, both share a template. After inference they share A.intent_detection>C.key_value_field_extraction>C.schema_constrained_json and the same value-free schema. Different intents and amounts do not split the group. That is what we want for “same job, different payloads.”

A third trace on the same prompt that omits refund_amount has a different observed schema, a different digest, and a separate group. Optional-field presence is enough. Correct if identity is the observed wire shape; fragmentation if you wanted “the support-router contract.” null versus populated, integer versus number, empty versus populated arrays do the same split.

Now invert the mapping. Serve does not re-infer a pipeline. It hashes the incoming masked template, reads a template map (workspace, template_hash) → digest, and may execute the DAG bound to @signature/{digest}. Execute uses the last user message as INPUT.text. Select and execute read the request two different ways.

Because the template is the lookup key and the digest includes observed schema, the map is not a function. One template, two contracts — json_schema versus json_object sharing a masked hash is the production case — and the row is ambiguous rather than overwritten. Lookup is LLM-free. Ambiguous, missing, or stale fails open, unless exactly one live/canary occupancy already matches the request’s output contract. That occupancy fallback is a serving patch. It does not make the map a function, and it is not a qualification edge.

The operational corollary developers hit in proofs: header=None after retries is a skip (map status, occupancy, contract-guard), not lag. Retrying the same request cannot disambiguate two digests. Hash equality of a masked template is not proof compiled serve will hit.

A related persistence bug: json_object replies were historically not stored as structured output, so later contract inference saw content_unavailable. The wire call succeeded; the retained observation did not. Capture completeness and contract identity are different ledgers.

Challenge 5 — Coarse keys hide identity; fine keys explode the tail

Identity is a partition. Finer keys split traffic; coarser keys merge it. Merge safety is a labeled policy, not a property of SHA-256.

On a sealed, privacy-safe Live sample — 16,397 retained-readable observations, 31 workspaces, 2026-07-20 to 2026-08-24 — different deterministic arms induce materially different partitions. One workspace is 97.6% of the sample; treat the globals as a description of this window, not a TAM. No arm was selected. Perturbations showed byte-stability. They did not measure accidental merges or paraphrase recall.

Read the table as an engineer, not as a clustering leaderboard.

The v1 digest is a coarse classifier: 59 groups, 99.52% of mass in the top five. Almost every structured trace shares a digest with a huge bucket. Digest equality then carries almost no identity information — which is why using it as a compile-and-serve key overstates replaceability.

Template-plus-declared is too fragmented for a compiler head: about 12% of observations sit in groups of size ≥ 3 (Wilson 95% interval 11.90%–12.91%). Most groups are singletons. You cannot compile a singleton.

Occurrence-exact structural identity sits in between: most groups are singletons, but 84.6% of observations sit in groups of size ≥ 3 and 72.2% in the top five. That is the economic shape we want — compile the head, ignore the tail — at structural contract granularity, not at LLM-digest granularity.

The general lesson: “we clustered production traces” is meaningless until you say which relation the key implements, and whether the head is large enough to matter without merging jobs that must not share a Candidate.

What we have built so far

The current hosted path is operational. The difficulty is not missing plumbing.

Capture. OpenAI-compatible Live router with complete-exchange persistence, stream assembly, attempt records, redaction, and conversation headers. Compiled-served traces are excluded from later Audit grouping so the compiler cannot train on itself. Import Audit accepts Langfuse, LangSmith, and Braintrust JSONL into a workspace-owned isolated scope: recoverable LLM-call coverage, skipped reasons, no invented cost, no write to the online map.

Admission and grouping. Per trace, admit structured output (JSON object always; non-JSON only with structured-text evidence and model agreement). Prose is free-text: no model call, not grouped. Across the snapshot, one signature-model call per reuse key — masked template × response protocol × whether JSON Schema enforcement succeeded — not one call per trace. Each JSON body still derives its own schema, so two traces that shared a model call can still hash to two digests. Pipelines containing Other are counted, not offered as picks. Support below a threshold (default 3) is a signature but not opportunity-eligible.

Candidate and qualify. A Candidate freezes one opportunity-eligible digest: membership, pipeline, output schema. Qualify splits members, uses the structured response as oracle, and scores composition agreement, schema, and latency. Qualification is offline and cannot create a serving binding. Import cases stop after evaluation.

Compile. The frozen pipeline becomes a typed DAG, one node per inferred step. The digest is recomputed and fails closed on mismatch. Realizability is compile, bounded_repair, or unrealizable. The shipping MVP is DAG-only; each node may still be an LLM call. That proves lifecycle and routing. It does not by itself prove cheaper-than-provider ETL. Optimization Journey copy already refuses that savings claim for the MVP path.

Route. Workspace routing mode off | observe | serve. Per-binding rollout off → shadow → canary → live, adjacent, human-gated, instant provider rollback. Serve is LLM-free. Fail-open on streaming, schema miss, executor failure, untrusted import, and map miss/stale/ambiguous. observe records map telemetry; it does not yet emit original-versus-Candidate pairs.

That is a real product. It is also why the challenges hurt: every one of those stages currently shares the v1 digest as if it were identity, interpretation, and replaceability at once.

What the spikes changed — and what they refused to claim

We ran seven read-only feasibility spikes (identity, rubric, task-layer value, typed graph search, active probes, risk envelopes, qualification edges) against a pinned Live window rather than guessing a v2 schema.

Typed search is tractable as a proposal engine. On pinned operator contracts, typing pruned 98.36%–98.95% of untyped graphs; bounded search matched exhaustive enumeration. That supports generating Candidate DAGs. It does not prove the inferred pipeline was the graph that ran.

Active probes beat passive replay on fixtures (19/20 seeds), and stop at uniqueness, not correctness. A noisy teacher trips a predeclared stop. An unavailable oracle eliminates nothing. Oracle authority is the binding constraint, not probe cleverness.

The three relations are separable at machine level. Two independent blinded machine-annotator sessions labelled 366 pairs: κ 0.973 overall; per-relation 1.000 / 0.989 / 0.931 (contract / intent / replaceability). Replaceability is the least consistent, as predicted. This is rubric applicability, not human agreement. Machine labels cannot select an accounting policy. A human confirmation subset remains required. The sample was essentially tool-free and contained zero non-transitive triples — agreement on an easy draw is not merge-tolerance on a hard one.

Identity arms were measured, not crowned. Spike A’s receipt is partition evidence only. Choosing a fingerprint requires labeled merge tolerance. v1 is structurally barred from selection (comparator only): it is blind to tools and transport, so tool/no-tool callers can collide under one binding.

Organic qualification evidence does not exist yet. Candidate lineage in the census was 8 known fixtures versus 79 unknown-provenance rows. Unknown is never relabelled organic. Outcome-based tests of the 41-task layer, family risk envelopes, and edge-versus-proxy prediction therefore returned unmeasurable_insufficient_data with named floors — not failed experiments. A first Import census looked for blob payload refs and saw zero readable rows; payloads were inline. Reconstruction failed as a boundary theory.

The evidence factory we do not yet run in production: observe-mode dual-run on eligible families, emitting paired original-versus-Candidate events, with fixture-versus-organic stamped at Candidate creation. Until those pairs accrue, serving authority should not move.

The solution shape we are aiming at

Not a new slogan — a split of jobs the current digest conflates.

Say interpreted as / compatible with / proposed unless evidence supports declared / verified / observed. Keep producer declarations, deterministic observations, classifier labels, human judgments, and behavioral outcomes in separate lanes. Latest-value overwrite of a trace’s signature columns hides that an interpretation changed; assessments should be append-only.

If experts cannot consistently label replaceability inside head groups, the broad vision narrows to curated family templates with stronger producer-declared contracts (schemas and structured response modes first). Same compiler, smaller provable scope. That is a kill gate, not an embarrassment.

Until then the sentence we can defend is:

It is not that we recovered the hidden workflow. It is not that signature membership proves one Candidate may replace the provider for every member of the group.

Why this is worth other people’s time

If you ship agents, you already pay the N² transcript. If you only reconstruct sessions, you can debug a run and still not know which inner hop dominates spend. If you cluster by embedding or by an LLM workflow label, you will get a beautiful map whose buckets are not compile keys. If you compile from those buckets, you will discover in production that the lookup is ambiguous, that optional JSON fields split the group, or that a Candidate which passed on membership A is being asked to serve C.

The industry solved observational reconstruction well enough that the remaining work looks like product polish. It is not. The remaining work is identifiability, partition policy, and qualification as an edge — the same issues compilers and process-mining systems hit whenever the spec is examples rather than a declared graph. Capture completely. Count a contract you can defend. Qualify a Candidate against that contract. Fail open when you cannot. That is the whole job, and it is harder than logging the agent.