Before you fine-tune, compile: lessons from a payment-risk experiment

We found a payment-risk implementation with higher F1 and recall at roughly 27× lower inference cost than Luna and 124× lower than Sol, without training custom weights. Then we asked it to match Sol's precision. It couldn't. Both results matter to the case for Seldon: making an AI function cheaper is useful only when its measured tradeoffs fit the application.

Coinbase's recent fraud-agent results inspired this experiment. In its Owning Intelligence post, Coinbase reports post-training Qwen3.5-9B on proprietary fraud outcomes. Against Opus 4.5, it gained 7.5 percentage points of precision, 12.0 points of recall, 9.6 points of F1, and 35.4 points of dollar-weighted recall. Its separate online serving evaluation reported 55% lower median LLM-request latency.

That is a compelling demonstration of specialization. It also raises a question for enterprises: how much improvement can you get by changing the program that performs the task, before changing a model's weights?

We built a payment-risk demo to investigate. We wanted to test the case for Seldon as an alternative starting point to fine-tuning: define the task, evaluate different implementations, and serve the one that best fits the business objective.

We did not train a competing Qwen model or use Coinbase's private data. Our experiment compares Seldon with direct Sol and Luna calls on a synthetic benchmark. It supports a case for trying compilation first; it does not establish that Seldon beats fine-tuning in a head-to-head test.

The bigger idea: keep the function, change its implementation

A frontier model is a convenient first implementation of an AI feature. You can describe what you want in a prompt and get something working. Once that feature becomes a repeated backend operation, a different question becomes important: what is the best way to execute this particular function?

Consider assess_payment(payment, history) → risk. The application needs a decision with a defined meaning and output shape. It need not depend forever on the model that made the first prototype possible. The implementation might eventually be a smaller LLM, a specialist decision model, deterministic operations, or a combination.

This is Seldon's broader thesis, Semantic Compute: separate the behavior an application needs from the machinery that executes it. The long-term direction is an AI compiler and runtime that can make those implementation choices explicit and measurable. Our thesis explores that direction; today's Functions workflow makes a bounded version practical. A developer supplies an existing call or short workflow, Seldon searches supported alternatives, and the developer chooses which measured revision to activate behind a stable interface.

The compiler analogy has a limit. An ordinary compiler is expected to preserve program semantics. Replacing a learned judgment requires empirical evaluation; matching the JSON schema does not prove matching behavior. In payment risk, catching more fraud while flagging more legitimate payments is a change the business must explicitly assess.

The experiment is a small test of that larger idea: can changing the implementation produce an attractive option before we train new weights?

The task and the four approaches

The task was deliberately narrow: inspect a card payment and its prior transaction history, then return a structured risk classification of low, medium, or high. Our fixed decision policy treated only high as a fraud-positive decision. Amount, recent velocity, merchant novelty, location, and historical behavior were available in the case file. Fraud labels were excluded from inference inputs.

The source was IBM's TabFormer synthetic card-transaction dataset. We used recorded simulator outcomes for scoring and rendered only history preceding the payment under review. These are simulated transactions, not customer payments or evidence of real-world loss prevention.

We compared four approaches across the main benchmark and a follow-up experiment:

1. Direct Sol: the original prompt and output schema sent to gpt-5.6-sol.
2. Direct Luna: the same application task sent to gpt-5.6-luna.
3. Seldon without a precision floor: compile the function, select an alternative, and evaluate the activated program against both baselines.
4. Seldon with a precision floor: explicitly maximize binary F1 while targeting at least Sol's previously measured precision, 65.17%.

The main comparison used the same 4,500 cases for all three approaches, including 900 frauds across a cohort of 460 users. Detection metrics were standardized to 5% fraud prevalence, so oversampling fraud did not artificially inflate reported precision. Costs and latency describe the actual executed workload.

The constrained follow-up evaluated three alternatives on a separate 500-case selection cohort containing 50 frauds. None qualified for the final benchmark. It therefore has selection results, not a fourth matched 4,500-case result.

The results

The constrained row failed its precision requirement. It is the best-F1 candidate returned by that run, not an accepted constrained solution. The highest-precision alternative in the same run reached 32.14% precision and 19.50% F1, also below the 65.17% floor. Its different evaluation cohort prevents treating the table as a controlled estimate of the effect of adding the constraint.

On the matched 4,500-case comparison, unconstrained Seldon improved F1 by 9.00 percentage points over Sol and 10.12 points over Luna. Recall rose by 46.23 and 44.29 points respectively. Fraud-dollar recall measures the share of fraudulent transaction value detected; Seldon's observed 87.37% was considerably higher than either baseline.

What began as a prompt to a general-purpose LLM became a function built around a specialist decision model: Jev (typesafe/jev-1.13.0). The application still asked the same payment-risk question, but a different kind of model supplied the decision. We did not train custom weights. A function first implemented with a frontier LLM does not have to remain a frontier-LLM call.

Getting there involved more than choosing a model name. Seldon configured how the function's inputs reached the decision model, exposed executable alternatives and their measured tradeoffs, and hosted the revision we selected. Two Jev configurations in the candidate comparison produced markedly different recall and F1 despite using the same model. That is the value Seldon aims to make repeatable: turning an available model capability into a measured, callable implementation of the function your application needs.

Observed inference spending was about 99.2% lower than Sol and 96.3% lower than Luna. P95 decision latency was about 47% lower than Sol and 33% lower than Luna on that same benchmark. Seldon completed all 4,500 cases; Sol had three failed decisions and Luna had two. Those failures remained in the experiment's accounting and scoring policy.

The cost ranges preserve unresolved charges at conservative upper bounds. These are measured inference costs, normalized per 1,000 scheduled cases. They exclude compilation, candidate selection, earlier attempts, engineering, and downstream review costs. They are not an estimate of total enterprise savings.

Higher F1 did not mean better on every business objective

Seldon caught more fraud, but its 21.64% precision meant many more legitimate payments were classified as high risk. Sol's 65.17% precision came with much lower recall. Luna sat between them on precision and close to Sol on recall.

That is a material operating tradeoff. A team looking for candidates for additional investigation may value broader detection differently from a team making automatic blocking decisions. Our experiment did not measure review capacity, customer friction, false-decline cost, or net financial benefit. Higher F1 alone cannot settle that deployment choice.

We therefore ran the fourth approach with a stricter objective: improve F1 while matching Sol's precision. The first attempt exposed a bug that lost the labeled task objective during evaluator editing. We fixed it, deployed the correction, and reran with byte-identical inputs. The results above come from that corrected run.

The compiler preserved the objective and reported all three published alternatives as missing the precision target. Our separate selection check agreed, and the experiment activated none of them. The demo's explicit acceptance rule enforced this stop; Seldon's comparison report can still expose executable alternatives with unmet preferences for a developer to inspect.

There was also a lesson about sample size. Some saved internal training evaluations contained just 32 cases and two frauds. One correct fraud prediction with no false positives produced an apparent 100% precision. On the subsequent 40-case development evaluation, the three published alternatives measured 19.72%, 0%, and 0%. All were already below the target before the 500-case check.

The search was bounded, GEPA refinement was disabled to fit the attempt limit, and the run ended partially measured. This establishes that the tested alternatives failed the requirement. It does not establish that the requirement is unattainable with a broader search, better evidence, or fine-tuning.

The operational difference: a function to improve, without a new checkpoint to operate

Fine-tuning can be inexpensive. Coinbase reports less than $100 of training compute. Its implementation also includes training infrastructure, evaluation, model serving and a continuing update lifecycle. That is sensible engineering when owning the model is valuable. The training bill alone, however, does not describe everything a team must maintain. Coinbase's account makes both the benefits and the responsibilities concrete.

The practical comparison is which work your team takes on:

Managed fine-tuning and hosting can absorb much of the infrastructure work in the middle column. Changing a precision target also does not inherently require retraining: threshold or policy changes may suffice. Seldon's argument is that implementation search, measurement and hosted execution belong in one workflow, so application teams can explore specialization without starting a model-training project.

That convenience does not remove the hard work of defining correctness. Seldon can start with generated evaluation examples, but scores on those examples do not establish performance on customer traffic. Our demo supplied labeled outcomes and used an additional scoring harness. Representative evidence, data governance and production monitoring remain the enterprise's responsibility whichever approach it chooses.

The hosted approach also trades away some of the ownership and infrastructure control available from self-hosting open weights. Seldon still relies on providers and serving infrastructure; those responsibilities are managed behind the interface rather than eliminated. Teams that need model ownership, strict infrastructure control, or adaptation to proprietary behavior may reasonably choose fine-tuning.

For everyone else, there is value in discovering that an existing specialist is sufficient. The payment-risk result illustrates that possibility, although it did not satisfy our stricter precision requirement. A simple winner is useful: the application benefits from the measured implementation, regardless of how elaborate the compiled program is.

Optimize against an operating target, then decide what to deploy

“Make the model better” leaves the business decision underspecified. Our fourth experiment used a much more useful request:

That made the tradeoff testable. The unconstrained winner was attractive on F1, recall, cost and latency, but unacceptable under this particular requirement. The failed search gave us a clear decision: keep looking rather than activate it for that use.

Seldon's optimization preferences generalize this approach. A team can prioritize task quality, cost or latency, and specify targets for metrics such as precision, measured cost and P95 response time. SLO-based optimization begins with those targets and a defined evaluation workload. It lets the same function be explored under different operating requirements without assuming a particular model must remain its implementation.

Today, these are search preferences and reported measurements, not continuously enforced production SLOs. New comparison reports retain executable alternatives even when they miss a target; a developer still chooses what to activate. Our experiment imposed its own hard acceptance rule and activated nothing from the constrained run. Neither a reported P95 nor a precision score guarantees future traffic will behave the same way.

Changing preferences quotes a new optimization run. Compatible evidence can be reused, and the existing function alias keeps serving its current revision until another is explicitly activated. This is the practical benefit of separating contract from implementation: the application keeps calling the function while the team evaluates a different way to execute it. Initial integration is explicit; Seldon does not silently intercept unrelated calls. Detected execution failures may fall back to the original workflow within runtime limits, but silently wrong answers remain possible.

What this experiment establishes

The strongest result is specific: on our synthetic payment-risk benchmark, a Seldon-selected program achieved higher F1 and recall, lower observed inference cost, and lower P95 latency than direct Sol and Luna, while sacrificing precision. Our additional precision-constrained search did not remove that sacrifice.

The 4,500-case cohort had been reused in earlier experiments. The original confirmatory qualification also failed its predeclared calibration and power requirements. We therefore treat these results as descriptive. The recorded analysis classified F1 and recall gains over both baselines as superior within that analysis, but it does not turn this reused benchmark into independent confirmation. The predeclared cost test remained inconclusive despite the large observed spending difference, and the fraud-dollar recall comparison with Sol also remained statistically inconclusive.

The model environments differ from Coinbase's, and we did not run a fine-tuned baseline. A direct claim of superiority over fine-tuning would require a matched comparison using the same data, decision policy, evaluation cohort, and accounting for adaptation and serving costs.

We also did not include a matched direct-Jev baseline. These results therefore do not isolate a performance advantage over manually configuring that specialist, or establish a benefit from composing multiple models.

For an enterprise deciding what to try next, start with one repeated, bounded function. Specify the business metric and acceptable tradeoffs, compare against the current implementation and a simple cheaper-model baseline, then test whether compilation produces a useful option. Include compilation, evaluation, serving and operational costs in the decision. Invest in fine-tuning when the remaining quality gap, control requirements and expected workload justify it.

Specialization can happen in the program as well as in the weights. Our experiment found a dramatic inference-cost reduction and a different quality tradeoff without custom weight training. It also found a requirement that the tested alternatives could not meet. Seldon's larger ambition is to make that process routine: keep the function the application needs, measure better ways to implement it, and put the deployment choice in the developer's hands.

Sources and experiment record

• Coinbase: How Coinbase Built, Evaluated, and Post-Trained a Fraud Agent, Part 3. Coinbase's figures describe its own private task and online serving evaluation.
• Payment-risk demo and methodology. Includes the pinned IBM TabFormer source, case rendering, decision policy, and independent scoring implementation.
• Main comparison: protocol v19. Production image dc4bfe22; matched 4,500-case Sol, Luna, and Seldon results.
• Corrected constrained experiment: protocol v21. Production image c8e3a186; three alternatives, 500 cases each, no qualifying candidate and no final benchmark execution.

The direct baselines used a 2,048-token output allowance; compiled execution used its bounded 8,192-token policy. Latencies are observed decision times through the experiment's execution paths, not complete checkout latency or a standardized provider speed test. Historical evidence and failed attempts are retained; the four approaches summarized here are not four independent replications.