How much of your LLM bill is just ETL?

The monthly invoice says "LLM usage." That is not a category of work. It is a category of billing. Inside the line item are many different computations: some semantic, some statistical, some mechanical, some accidental. The interesting question is not whether language models are useful. It is whether the bill is paying for language modeling at all.

Abstract

There is no public, reliable estimate of the fraction of production LLM calls that do not require an LLM. The available evidence is indirect. Model-routing papers show that many requests can be served by cheaper models without large quality loss. Practitioner reports show that classification, extraction, normalization, and validation often migrate from prompts into smaller models, rules, or conventional data pipelines. Production write-ups also show the opposite: for messy document workflows, LLMs can replace brittle classical systems and improve automation.

The right conclusion is narrower than the slogan.

Not: "Most LLM calls are waste."

Rather: production LLM traffic should be decomposed by task type. A frontier model call may contain reasoning, retrieval, extraction, validation, formatting, lookup, and arithmetic in the same prompt. Treating that bundle as one primitive hides the parts that could be served by a smaller model, cache, retriever, parser, or deterministic transform.

This post proposes a trace-level audit for estimating how much of an LLM bill is really ETL-shaped work.

The null result matters

I looked for a clean public answer to a deceptively simple question:

I did not find one.

That absence is informative. The industry measures token spend by provider, model, feature, customer, and sometimes prompt template. It rarely publishes distributions by task semantics. There are plenty of dashboards that say "you spent $18,742 on GPT-4-class models last month." There are fewer that say:

• 19% was classification over a closed label set.
• 14% was extraction into a recurring schema.
• 8% was normalization or format conversion.
• 6% was lookup against a bounded catalog.
• 3% was numeric computation wrapped in prose.
• The rest was open-ended reasoning, drafting, summarization, or genuine ambiguity.

The second report is harder to produce because it requires interpreting traces as programs, not just counting tokens.

What routing papers reveal

The best quantitative evidence is not about ETL. It is about routing.

RouteNLP, a 2026 preprint on closed-loop LLM routing, describes an enterprise financial-services partner whose NLP serving costs exceeded $200K per month. The authors report that more than 70% of queries were routine enough not to require frontier-model capabilities, and that across finance, customer-service, and legal scenarios only 25-35% of queries required frontier models. In an eight-week customer-service pilot processing roughly 5,000 queries per day, they report a 58% inference-cost reduction, 91% response acceptance, and a reduction in p99 latency from 1,847 ms to 387 ms.

That result is not a census of LLM usage. It is a reported deployment and a routing system. The replacement target is often a cheaper model, not SQL or Python. But it gives the crucial shape: workload difficulty is not uniform.

FrugalGPT reached the same shape from another direction. It studies cascades across LLM APIs and reports that carefully routing across cheaper and more expensive models can match the best individual model with up to 98% cost reduction in its experiments. RouteLLM reports more than 85% cost reduction on MT Bench, 45% on MMLU, and 35% on GSM8K while retaining about 95% of GPT-4 performance.

These papers do not prove that most LLM calls are ETL. They prove a weaker and more useful fact: many calls do not require the maximum available inference engine.

The ETL question begins there.

A small cost model

The easiest mistake is to measure only average cost per request. That hides the economic structure.

For a workflow cluster k, the monthly cost is approximately:

cost_k = calls_k * (input_tokens_k * input_price + output_tokens_k * output_price + retry_cost_k)

But the optimization target is not cost_k. It is the replaceable portion:

replaceable_value_k =
  calls_k
  * coverage_k
  * (frontier_cost_k - alternative_cost_k)
  - maintenance_cost_k
  - error_cost_k

The important term is coverage_k. A replacement path does not need to handle the whole distribution. It needs to handle the high-confidence slice where correctness is measurable. The long tail can remain on the frontier model.

This turns the migration question from ideology into arithmetic. A deterministic path that handles 40% of a very expensive workflow may be more valuable than a classifier that handles 95% of a cheap workflow. A path that saves tokens but increases severe error risk is negative value. A path that saves little cost but removes p99 latency from a critical UI may still be worth building.

Three ratios matter more than the headline bill:

1. Economic concentration: what fraction of spend sits in the top N workflow clusters?
2. Measurable coverage: what fraction of each cluster can be scored with a task-specific metric?
3. Safe deferral: what fraction can be routed to a fallback without harming the product?

The third ratio is why LLM replacement is different from a normal rewrite. You do not have to delete the old implementation. You can place a cheaper path in front of it and make the frontier model the residual path.

A frontier call is often a bundle

A production LLM call is rarely a pure act of language generation. It is usually a bundle of sub-computations because prompts are the easiest place to put work that has not yet been formalized.

Consider a support automation prompt:

Given this support thread:
1. Determine the customer intent.
2. Extract the account id and product.
3. Decide whether the issue violates the SLA.
4. Normalize any dates to ISO-8601.
5. Return JSON matching this schema.
6. If confidence is low, explain why.

That looks like one LLM call. Operationally, it is a pipeline:

thread text
  -> intent classification
  -> field extraction
  -> entity resolution
  -> date normalization
  -> SLA computation
  -> schema validation
  -> ambiguity handling

Some stages are semantic. Some are lookup. Some are arithmetic. Some are pure formatting. A frontier model can do all of them because it is a permissive universal adapter. That does not mean all of them are language-model tasks.

The bill is therefore ambiguous. It could be buying:

• inference for reasoning,
• flexibility during schema discovery,
• a substitute for missing parsers,
• a substitute for missing classifiers,
• a substitute for missing reference-data joins,
• or a substitute for missing engineering time.

Only the first two are strong long-term reasons to keep a frontier model in the hot path.

A task taxonomy for the invoice

The useful unit is not the request. It is the subtask.

Most repeated LLM traffic can be mapped into a small set of task families:

1. Open-ended reasoning and generation: synthesis, planning, drafting, nuanced summarization, multi-step tool choice.
2. Classification: mapping text to a label from a bounded set.
3. Sequence labeling: finding spans, entities, clauses, or key phrases in the input.
4. Structured extraction: populating a recurring JSON schema or table.
5. Retrieval and entity resolution: mapping a messy mention to a known customer, ticker, product, policy, or account.
6. Similarity and clustering: duplicate detection, grouping, topic assignment, semantic matching.
7. Normalization and transformation: dates, units, names, casing, whitespace, JSON reshaping, enum mapping.
8. Numeric and analytical computation: totals, ratios, thresholds, date math, SLA deltas.

The first family plausibly belongs on a frontier model. The middle families often belong on smaller models, embedding systems, rerankers, or classical NLP. The last two often belong in code.

This is not a hierarchy of dignity. It is a measurement scheme.

If an LLM call is expensive, repeated, and commercially important, ask which of these families it contains. Then price each family separately.

Structured output is a clue, not the endpoint

Structured output support has improved enough that many teams can now get reliable JSON directly from a model. OpenAI's Structured Outputs launch is explicit about the motivation: developers were using prompting, open-source tooling, and repeated retries to make model output fit system schemas, and structured outputs constrain models to developer-supplied schemas. OpenAI reported that gpt-4o-2024-08-06 with Structured Outputs reached 100% reliability on their complex JSON-schema-following evals, compared with less than 40% for gpt-4-0613 on the same evals (source).

A broader 2025 paper, Generating Structured Outputs from Language Models: Benchmark and Studies, describes constrained decoding as a method that masks invalid tokens so the model can sample only outputs conforming to a specified structure. The authors introduce JSONSchemaBench, a benchmark of 10K real-world JSON schemas, and report that constrained decoding can speed generation by 50% compared with unconstrained decoding, while support for real-world schemas varies substantially across frameworks.

This is good engineering. It also changes the interpretation of the workload.

If a prompt requires a JSON schema, the system has already admitted that the output is not arbitrary prose. If the schema is stable over many calls, the trace cluster now has a machine-readable contract. That contract is exactly what a smaller model, parser, extractor, validator, or deterministic transform would need.

Structured outputs are therefore both a runtime improvement and an audit signal.

They answer one question:

They do not answer the next question:

Consider a structured extraction response:

{
  "vendor_name": "Acme Ltd.",
  "invoice_date": "2026-03-14",
  "total_usd": 12842.17,
  "payment_terms": "net_30",
  "risk_flag": "low"
}

Those fields do not have equal semantics. vendor_name may be entity resolution. invoice_date may be OCR plus date normalization. total_usd may be table extraction plus arithmetic. payment_terms may be clause extraction. risk_flag may be classification. A single JSON schema can hide five different replacement candidates.

That is why structured output should be treated as the beginning of decomposition, not the end of it.

Classification is the easiest place to see the split

Text classification has become the hello-world of wasteful LLM architecture, because it moves through predictable phases.

During discovery, a frontier model is useful. It lets a team invent labels before it has a taxonomy. It handles messy examples. It gives product managers something that works before there is a training set.

After the label set stabilizes, the task changes. It becomes a supervised learning problem with a small output space and an evaluable metric.

A 2025 paper, Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?, argues against treating LLMs as the default for classification. The paper reports that BERT-like models often remain stronger for pattern-driven classification tasks, while LLMs have advantages when deep semantic understanding or world knowledge is needed.

That is the correct dividing line. "Classification" is not one task. A 3-way routing decision over templated tickets is not the same as classifying subtle legal risk in a rare fact pattern.

The audit should ask:

• Is the label vocabulary small and stable?
• Is the output evaluated by exact match, F1, or acceptance rate?
• Do identical or near-identical inputs produce stable labels?
• Is there enough historical traffic to create a training or validation set?
• Does the model rely on external world knowledge, or mostly on local text?

If the answers point toward stability, the frontier call is probably scaffolding. The durable system may be a classifier with an LLM fallback.

Extraction is where slogans fail

Extraction is the most interesting counterexample because both stories are true.

One story: many extraction prompts are schemas in disguise. "Return {invoice_no, total, due_date}" is often a request for a document parser plus a validator. If the documents are templated and the schema is stable, a frontier model may be an expensive way to avoid writing the parser.

The other story: real documents are hostile. They contain layout variation, OCR noise, missing fields, handwritten annotations, language drift, vendor-specific formats, and exceptions that destroy brittle rules.

Alan's production write-up on French insurance documents is a useful antidote to glib anti-LLM arguments. They report moving from roughly 40% to 70% document-processing automation after adopting a more unified LLM-based approach, while still keeping classification, transcription, retrieval, examples, validation, and evaluation in the pipeline (source).

The lesson is not that extraction should or should not use LLMs. The lesson is that extraction decomposes.

A single document workflow may contain:

• OCR or transcription,
• document classification,
• page or region selection,
• field extraction,
• field normalization,
• cross-field validation,
• confidence scoring,
• human review routing.

The LLM may be essential for one stage and wasteful for another. The goal is to find the boundary, not declare the whole pipeline pure AI or pure ETL.

The workflow-versus-agent distinction matters

Anthropic's "Building Effective AI Agents" makes a useful distinction between workflows and agents. Workflows orchestrate LLM and tool calls through predefined code paths; agents decide their own process dynamically using environmental feedback. The article recommends evaluator-optimizer loops when there are clear evaluation criteria and iterative refinement measurably improves outputs (source).

That maps cleanly onto the ETL question.

A workflow with a known control path is a better candidate for decomposition than an agent with open-ended planning. If the system always does:

classify -> extract -> validate -> write

then the control flow is already code-shaped. The question is which nodes still need a model.

If the system instead asks the model to decide which tools to call, when to stop, and how to revise its plan, replacement is harder. The trace still helps, but the unit of analysis becomes the trajectory, not a single transformation.

This suggests a useful rule:

That rule is not perfect. Some fixed workflows contain difficult language understanding. Some agents repeatedly perform simple operations. But as a first pass, it predicts where to search.

Trace features that expose ETL-shaped work

The provider invoice is too low resolution. You need traces.

At minimum, each request should preserve enough metadata to reconstruct the work:

• Prompt family: template, route, feature, product action, workflow name.
• Input class: ticket, transcript, invoice, contract, support thread, filing, CRM note, row batch.
• Output type: prose, label, span list, JSON record, catalog id, boolean, number.
• Output schema: field names, field types, optionality, schema version, parser failures.
• Reference data: bounded catalog, account table, product list, policy corpus, ticker universe.
• Cost and latency: input tokens, output tokens, cached input, retries, time to first token, total duration.
• Temperature and constraints: response format, JSON schema, tool use, max output, stop conditions.
• Downstream fate: accepted, edited, retried, rejected, manually corrected, written to a database.

Once those fields exist, the analysis becomes empirical rather than rhetorical.

For each trace cluster, compute:

1. Output entropy: how many distinct outputs appear, and how concentrated are they?
2. Schema stability: how often does the output conform to the same structure?
3. Functional stability: do similar inputs map to similar outputs?
4. Reference dependence: can the answer be derived from a bounded corpus or catalog?
5. Validation strength: can correctness be scored by exact match, F1, retrieval metrics, or numeric tolerance?
6. Fallback rate: what fraction of items are ambiguous enough to need the frontier model?

The candidate ETL share is the spend attached to clusters with low entropy, stable schemas, strong validation metrics, and derivability from local input plus bounded reference data.

Validation is task-specific

The hardest part of reducing model usage is not finding cheaper alternatives. It is knowing when they are correct.

Generic "LLM quality" metrics are too blunt. Each task family has its own validation surface:

Google's GenAI evaluation guidance frames the production transition in similar terms: moving from prototype to production requires evaluation, data, and metrics rather than "vibes-based" testing (source). Microsoft describes LLMOps as the end-to-end process for developing, deploying, and maintaining LLM applications, with an inner loop for develop-test-refine and an outer loop for production deployment and management (source).

The ETL audit is a specialization of that general LLMOps principle. It says: before you optimize, identify the task family and choose the metric that can falsify the replacement.

A measurement protocol

Here is a concrete protocol for decomposing an LLM bill.

1. Cluster by workflow, not endpoint

An endpoint often contains many workloads. A workload may span many endpoints. Use prompt template, route, embedding similarity, output schema, and downstream action to group traces.

2. Infer the output contract

For each cluster, classify the output as one or more of:

prose | label | span[] | record{schema} | catalog_ref | number | boolean

This single step changes the discussion. A record{schema} or catalog_ref output has a different replacement path than free prose.

3. Rank by economic mass

Sort clusters by monthly cost, but keep call count and latency visible. A low-cost high-frequency cluster may matter for reliability. A high-cost low-frequency cluster may not be worth touching.

4. Build a replay set

Sample historical inputs and accepted outputs. Include human corrections, retries, parser failures, and downstream edits when available. The replay set is the local truth source. Without it, replacement is speculation.

5. Test cheaper hypotheses

Try replacement classes in increasing order of risk:

exact cache
deterministic code
bounded lookup / retrieval
small classifier or extractor
smaller generative model
frontier model

The point is not to force everything downward. The point is to discover the lowest layer that clears the metric.

Each class has a different failure mode.

Exact cache. The safest optimization. It works when the same input, or a canonicalized equivalent, appears repeatedly and the output is not time-sensitive. The hard parts are invalidation, workspace isolation, and avoiding accidental caching of private or policy-sensitive data.

Deterministic code. The right target for normalization, arithmetic, format conversion, schema validation, and simple business rules. The risk is not model quality. The risk is incomplete requirements: code is only correct for the cases the team has named.

Bounded lookup or retrieval. The right target when the answer should be a member of a known set: customer, product, ticker, vendor, policy, clause, account, merchant. The risk is reference-data quality. A bad catalog creates confident wrong answers.

Small classifier or extractor. The right target when outputs are stable but still require pattern recognition. The risk is distribution shift. Monitoring matters because a classifier can silently decay as product behavior changes.

Smaller generative model. The right target when the task still benefits from generation but not frontier capability. The risk is shallow evals: a cheaper model can match easy tests while failing rare cases that matter.

Frontier model. The right target for ambiguity, synthesis, planning, high-consequence edge cases, or cold-start workflows where the contract is not yet known.

The migration is therefore not a ladder from "bad" to "good." It is a set of hypotheses about the true computation.

6. Report coverage, not replacement

The result should look like this:

"Coverage" is the important column. A compiled or deterministic path does not need to serve every case. It needs to serve enough high-confidence cases to justify its maintenance cost while preserving the fallback.

Promotion gates

A cheaper path should not be promoted because it looks plausible on a few examples. It should pass gates.

Gate 1: contract stability

The input and output contracts should be stable over a meaningful window. If the schema or label set is still changing weekly, the prompt may be the best place to absorb churn.

Gate 2: offline replay

Run the candidate path against historical traces. Report per-field and per-class metrics, not only aggregate accuracy. For classification, inspect confusion matrices. For extraction, inspect field-level failures. For entity resolution, inspect false positives separately from false negatives.

Gate 3: calibrated confidence

The replacement path needs a confidence score that predicts correctness. A high average accuracy is not enough. The system must know which examples to refuse.

Gate 4: shadow traffic

Run the replacement path beside the existing LLM path without serving it. Compare outputs, latency, and failure modes under current traffic. This catches distribution drift that historical replay misses.

Gate 5: limited serving

Serve only high-confidence cases first. Keep logging disagreements and downstream edits. Expand coverage only when the served slice remains stable.

Gate 6: rollback and residual policy

Define what happens when confidence is low, parsing fails, reference data is stale, or downstream validation rejects the output. The residual path may be the frontier model, a human queue, a retry, or a degraded response. It should not be an unhandled exception.

This is standard production engineering, but it is often skipped in LLM systems because the prompt is treated as the implementation and the model as the fallback. Once a replacement path is introduced, the old model should be treated as a dependency with explicit routing policy, not as a vague safety blanket.

The architecture that falls out

The architecture is a cascade, but not only across models.

request
  -> exact cache
  -> deterministic transform
  -> parser / validator
  -> bounded retrieval or catalog match
  -> small classifier / extractor
  -> frontier model

The cascade should be boring. It should have thresholds, metrics, logs, and rollback. It should also be allowed to say "no." Some tasks are genuinely open-ended. Some are too low volume to justify migration. Some lack the reference data required for a deterministic replacement. Some fail replay.

This is where the ETL analogy becomes precise. Mature data pipelines are not cheaper because they are less capable. They are cheaper because the work has been made explicit:

• the input contract is known,
• the output contract is known,
• the reference tables are known,
• the validation checks are known,
• the failure modes are known.

The frontier model is valuable before those facts are known. It becomes expensive after they are.

Where the ETL analogy breaks

ETL is a useful analogy because it emphasizes contracts, lineage, validation, and repeatability. It is also incomplete.

Traditional ETL usually assumes the transformation is known before execution. Many LLM workflows begin before the transformation is known. The model is used precisely because the input distribution is messy, the schema is provisional, and the business process has not yet hardened.

That means the lifecycle matters:

1. Exploration: the LLM absorbs ambiguity while the team discovers the task.
2. Instrumentation: traces reveal repeated shapes, output contracts, and failure modes.
3. Decomposition: repeated clusters are split into task families.
4. Replacement: stable sub-tasks move to cheaper paths.
5. Residual reasoning: the LLM remains for cases outside the replacement contract.

In that lifecycle, the LLM is not the enemy of the data pipeline. It is often how the data pipeline is discovered.

The technical debt begins when discovery never ends.

Security and compliance implications

There is another reason to decompose the bill: governance.

If a model prompt contains extraction rules, validation logic, routing decisions, and downstream write instructions, then the prompt is part of the control plane. It should be versioned and reviewed like code. It should have tests. It should have access controls. It should have observability.

Moving repeated sub-tasks into explicit pipelines can reduce governance risk:

• deterministic code can be reviewed,
• schemas can be versioned,
• reference data can be audited,
• validators can be tested,
• fallback thresholds can be documented,
• human-review queues can be scoped to genuinely ambiguous cases.

But replacement can also increase risk if done carelessly. A deterministic system can be confidently wrong. A classifier trained on historical outputs can reproduce historical bias. A cached response can leak stale policy. A catalog lookup can encode outdated entity mappings. The point is not that non-LLM systems are automatically safer. The point is that their failure modes are often easier to name.

Cost observability is necessary, not sufficient

Cost attribution is improving. Honeycomb's guide to token-cost tracking argues for request-level attribution by feature, customer, model, prompt, retries, and latency rather than a single monthly bill (source). Braintrust makes the related point that cost comparisons must preserve token categories such as cached input, cache writes, batch usage, service tier, retries, and output length (source).

That work is foundational. But cost attribution answers only the accounting question:

The ETL audit asks the systems question:

Those are different questions. The first needs billing metadata. The second needs task semantics.

A note on latency

Cost is easier to measure than latency, but latency often reveals the same structure.

A frontier model call has several components:

gateway overhead
  + queueing / provider latency
  + prefill over input tokens
  + decode over output tokens
  + retry / parsing overhead

ETL-shaped replacements attack different terms. Caching removes the provider call. Deterministic code removes prefill and decode. Structured outputs can reduce retries and output length. Smaller models may reduce both prefill and decode but still leave a model call in the path. A retrieval or catalog match may add its own index latency while removing generation.

This is why p95 and p99 matter. A workflow whose mean latency is acceptable may still have a tail problem caused by retries, schema failures, or provider variance. Replacing the stable slice can improve tail latency even when average cost savings are modest.

The measurement should therefore track:

• time to first token,
• total duration,
• retry count,
• parse failure rate,
• fallback rate,
• and downstream correction rate.

An LLM bill is partly a latency bill. The user experiences it as waiting.

Failure modes

The ETL hypothesis can fail in several ways.

The task is open-world

If the output depends on information not present in the input or bounded reference data, local deterministic code cannot recover it. A frontier model or retrieval-augmented generation path may be necessary.

The metric is weak

If correctness cannot be measured, replacement is dangerous. Exact match works for dates. Field-level F1 works for extraction. Retrieval metrics work for lookup. Human preference is weaker and more expensive.

The schema is unstable

If product requirements are still changing weekly, a prompt may be the right abstraction. Rewriting unstable behavior as code just moves churn from prompt text to source files.

The tail dominates

A cheap path that handles 80% of requests may still be a bad system if the remaining 20% carries all the business risk. Coverage must be weighted by consequence, not only count.

The LLM improved the process

The Alan case matters because it shows a real migration in the other direction. Sometimes the classical pipeline is the brittle one. The LLM is not always technical debt; sometimes it is the first implementation that actually survives the data distribution.

A better question than the headline

"How much of your LLM bill is just ETL?" is deliberately provocative. The more precise question is:

That is less catchy. It is also the version an engineer can measure.

The answer will differ by company. A code assistant, a legal research product, and an invoice-processing workflow will have very different distributions. But the method is portable:

trace -> cluster -> infer contract -> test cheaper implementation -> shadow -> promote -> retain fallback

The end state is not an LLM-free system. It is a system in which the frontier model is used where its marginal capability is actually needed.

The rest should look suspiciously like software.

Sources

• RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization
• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
• RouteLLM: Learning to Route LLMs with Preference Data
• Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?
• Introducing Structured Outputs in the API
• Generating Structured Outputs from Language Models: Benchmark and Studies
• Building Effective AI Agents
• Master Generative AI Evaluation: From Single Prompts to Complex Agents
• LLMOps - Operational management of LLMs
• What 300+ Engineers from Netflix, Amazon, and Instacart Asked Us About AI Engineering
• How we reached 70% document processing automation at Alan
• How to Track Token Cost Across LLM Workflows
• How to track LLM costs: a playbook for per-user, per-feature, and per-model attribution