AI applications need better primitives than text generation
A production LLM call is often a latent program. It looks like generate(). Classification, extraction, linking, ranking, normalization, and arithmetic each have their own output and their own way to fail. The work that matters sits under the prompt. Name the operation. Recover a candidate program from the traffic. Then choose an implementation for each step.
TypeSafe's Jev corrects one of those shapes. A category, an ordered score, or a yes-or-no probability is a better interface for a machine than a paragraph. Decisions start the vocabulary. They do not finish it. Name the operation and the runtime can stop treating the original call as one thing.
One call, already a program
"Read this annual report and return every customer-supplier relationship as {customer, supplier, evidence}" sounds like general reasoning. It is a classical extraction stack. Named-entity recognition. Normalization. Candidate generation. Entity linking. Relation extraction. Schema validation. The frontier model belongs on the cases those operators were not built for.
A support ticket has the same shape and a different chain. Language, intent, and sentiment are classifications. Priority is an ordered score that follows those judgments. Account resolution is an entity link. The record is a schema check. The API records one inference. The chain records a dependence. Severity depends on what the ticket is about, so priority comes after intent and sentiment. Drop that dependence and you have a different program.
Keep the dashed bypass when you factor a call. The original email still lands on the triage record. Structured fields do not stand in for the text they were read from.
The frontier model earns its place at the start. A team can ship while the ontology, the schema, and the exceptions are still unclear. After enough traffic the runs are examples: inputs, outputs, corrections, and the cases that broke. Finite examples do not fix one program. Many programs agree on the traffic you have. They split on the next request. The examples support a search, inside a vocabulary small enough to mean something. That search is programming by example. The traces are the spec. The families below are the language the spec is allowed to use.
Decisions start the vocabulary
PII detection wants a probability over a boolean. Department routing wants a choice from a fixed set. Severity wants an ordered score. Make the model write, and you throw away structure the application already has. A probability becomes an action only with a threshold. Set the threshold by what a wrong decision costs.
You can push classification further. Retrieval is a choice among documents. Span extraction is a choice among substrings. Entity resolution is a same-entity decision over pairs. Those reductions hold only after someone has built a finite set of candidates. That step is the structure the reduction erases. Without the set, a softmax over a corpus will not run. Pairwise resolution is quadratic. Classifying every substring is quadratic. Sequence labeling is linear.
A primitive should keep what an optimizer can use. A searchable set of candidates. Span locations. Canonical identities. Exact arithmetic where an exact function exists.
A working taxonomy
This is how we are thinking about the solution — but it is by no means the only taxonomy or even the optimal one.
Also, maybe fine-tuned small LLMs.
Potentially, different teams and systems may have a proprietary taxonomy of tasks that fits them better.
disposition
The line under each family names one implementation that often fits. It is not a law. A small classifier for a stable set of labels. A lookup index for a bounded catalog. A formula engine for arithmetic and dates. Span labeling is often a sequence model or a fine-tune. The note in the margin of the figure says a small language model is one way to run the step, next to rules and indexes. The hatched cell is the rest. Those calls stay on a frontier model. No operator in the vocabulary can do them.
This is the cut we use for repeated workloads: seven families, forty-one subgroups in the v1 registry. It is a catalog of shapes and a sketch of how they compose. It becomes an instruction set when each operator has a meaning, a way to compose, and a cost. The categories will be cut again. Law will want clauses and citations. Finance will want security resolution and corporate actions. Healthcare and logistics will bring their own. We publish the taxonomy so a runtime can see the shape of the work and pick an implementation, and so a domain can cut the work to fit itself.
The operation and the implementation are different layers
ENTITY_MATCH(company_mention, company_master) can run as an exact lookup, a fuzzy rule, an embedding index, a gradient-boosted model, a cross-encoder, a decision model, a small language model, or a frontier model. Those are ways to run it. The operation means the match.
Databases made this split long ago. SQL states the join. The planner picks a hash join, a merge join, or an index lookup, from theorems of the relational model. A learned operator does not get those theorems. CLASSIFY(email, INTENT) is an estimator tied to a label set, a threshold, and a training distribution. Swap a small model for a frontier model and the interface stays. The function changes. You can still choose an implementation. Held-out traces tell you whether the new plan does the job. That is a measurement on a distribution. It is not a proof that two plans compute the same relation.
The application can still name the steps:
language = CLASSIFY(email, LANGUAGE) intent = CLASSIFY(email, INTENT) sentiment = SCORE(email, SENTIMENT) priority = RANK(email, PRIORITY) account = ENTITY_LINK(email.sender, CUSTOMER_MASTER) record = VALIDATE(TriageRecord)
Routing picks a model for a request it will not split. Compilation can change the request. It starts once a candidate program exists. Recovering that program from traffic is the synthesis step. Lowering each operator comes after.
What the plan changes
One frontier call can come down to the chains in the figures: deterministic preprocessing, a small classifier, an index, a matcher, a normalizer, a validator, and a frontier fallback behind a novelty gate. The gate is Chow's old rule. Answer when you are inside the region you know. Defer when you are not. You save money when most of the probability mass sits where the cheap operators already agree with the original call. Defer often, or trust a confidence that is off, and the cascade costs more. The slow tail gets worse. The diagram is a plan you can measure. It does not prove the savings.
Explicit types give the rest of the system a place for the evidence. Retrieval can return references. Extraction can keep spans. Linking can expose canonical ids. A score is not a calibrated probability yet. Show the calibration before a threshold is safe to automate. A deterministic node is exact for the policy it states: a calendar, a timezone, a tokenizer. A box on a diagram does not make it exact.
Generation got the work started. Teams built programs before they understood them. There is enough production traffic now to search for the program inside a vocabulary you declared and, where the measurements hold, run that program on cheaper machines. The layer that does this says what the application is doing, apart from which model did it first. A production LLM call becomes a program with a plan.
What matters sits under the prompt and above the model: a vocabulary of operations, a search over traces, and a runtime that picks an implementation for each step.
Open beta
Route your traffic through Seldon and investigate recurring work that could benefit from a specialized program.