Archive← Back to ArticlesDiscoveryUpdated October 14, 2026·Published October 6, 20267 min readCopy article
Exploring Decision Models with a Lightweight Benchmark
CPU inference in tens of milliseconds, with no per-request provider fee. Compare its tradeoffs with 9 decision APIs on accuracy, speed and charge.
Developed by Robert E. Beckner III (Merlin) | rbeckner.com
Some AI tasks are decisions: choose a function, route a request or flag it for review. A small model trained for one of those jobs can run on a CPU you already have.In our separate EmbeddingGemma 2 serving test, a warm encoder pass took 37–48 ms median using 2–4 CPU threads, with approximately 1.3–1.6 GB of memory. Inference needed no GPU and carried no per-request model-provider fee. On spare CPU capacity, that can make the incremental cost very small.I also asked Codex to compare 9 hosted decision APIs on the same 24 messages. Luna returned the complete request fastest; Decider V1.1 charged least. Jev combined a perfect primary score with full accepted coverage under the tested application's confidence checks. Together, these experiments show the tradeoff between a broad decision API and a fast specialist.
How we benchmarked 9 decision models on 21 questions#
Each API request contained 21 typed questions, including request type, record ownership, function selection and dates. The function question offered 56 options. The 24 frozen synthetic messages comprised 12 record requests and 12 that should escalate, including actions, analysis and combined reads. Every model received identical inputs.We compared Jev 1.13, Luna Decisions, Decider V1, Decider V1.1, Microsoft Decision-1, Liquid d1, Drex v1.5, Kev 4B and Clef Flash through native decision endpoints.The main score measures correct function selection or refusal, with agreeing request and ownership answers. An accepted answer also passes the application's confidence and agreement checks, allowing the application to use it. A correct choice can fail those checks and require another model or review.Codex authored and reviewed the labels. Calls ran serially through the same NYC gateway, with 1 physical attempt each. The initial 4 candidates were interleaved; later models ran as separate arms through October 9. Each arm included 2 short context probes beyond the 24 scored cases. The samples are small and agent-labelled; dates, complete record lookups, delivered answers and burst capacity were not independently evaluated. Provider load between arms remains uncontrolled.
These confidence cuts were developed for Jev and reused unchanged. The table measures compatibility with that policy; another application may choose different cuts after calibrating on separate data.Drex v1.5 selected 1 function for requests requiring 2 reads. The checks blocked 1 error but accepted another as a calendar lookup, omitting the second read.
The speed index is 100 × Luna's median divided by each model's median. A model taking twice as long scores 50. Timings include network and gateway overhead; they measure complete API responses, not throughput. With 24 observations, p95 is exploratory.
Reported provider charge per 1K requests scales each model's average charge across the 24 cases. The chart expresses the same measurement as requests per $1. It projects equivalent work, not capacity or a future bill.Decider V1.1 charged 54.5% less than Jev. Luna returned responses 25.4% quicker, at 1.99× the provider charge. Accepted coverage shows how much of that work the application could use.Token price alone did not predict the bill. Jev counted 274,501 input tokens across this cohort, versus Liquid d1's 182,505 for the same payload. Compare the reported charge for the complete request.Solar Decide Flash could not run this benchmark: its route allowed 26 candidates per choice question, while our questions required 52 and 56. It is excluded from the charts. Changing the options would create a different test.
Self-hosting: a fast CPU specialist with no per-request provider fee#
EmbeddingGemma 2 offers a different way to supply a decision: fine-tune its text encoder and add a classification head. Once hosted, each inference uses local compute instead of buying another model call.
CPU threads
Median encoder time
p95 encoder time
2
48 ms
54 ms
4
37 ms
42 ms
It ran in bf16 on an AMD Ryzen 7 7700, using approximately 1.3–1.6 GB of memory. Options were prepared in advance. These are warm encoding times; HTTP and fallback time are outside the measurement. For context, Jev took 330 ms median and 459 ms p95 on the specialist's English cases, returning 22 answers per request. The timing boundaries differ, so a speed ratio would be misleading.
A separate function-routing experiment used 204 English-language synthetic cases, selected by their existing language labels: 127 requests for a function and 77 that should escalate. The model chose from 55 functions plus escalation. Frozen and fine-tuned encoders trained on identical data; fine-tuning also updated the text tower's weights.
Function-routing arm
Correct selections / 204
Accepted at confidence ≥0.98 / 204
Wrong among accepted
Jev
173 (84.8%)
103
1
Frozen EmbeddingGemma 2 encoder + head
171 (83.8%)
28
0
Fine-tuned encoder + head, seed 0
176 (86.3%)
32
0
Fine-tuned encoder + head, seed 1
180 (88.2%)
32
0
Fine-tuned encoder + head, seed 2
179 (87.7%)
38
0
Fine-tuning averaged 87.4% correctness, up 3.6 percentage points from the frozen encoder. At the strict 0.98 cut, the 3 seeds accepted 32–38 cases with zero observed errors. Jev accepted 103 with 1 error. Accepted here means a function choice at the cut; escalation outputs are excluded. Zero observed errors on this sample does not establish an error-free service.The seeds repeat 204 cases; the frozen baseline is 1 run. Timing used a different checkpoint and corpus. These specialist measurements sit outside the API leaderboard; function selection does not prove dates, slots or actions.This is where self-hosting can pay: a stable, repeated classification task that a small model can finish on spare CPU capacity. There is no per-request inference-provider fee; electricity, allocated server capacity and maintenance remain system costs. We did not measure a total cost per decision. Selecting a function locally saves an API call only when no other answers from that call are needed. Count fallback calls and their added time too.
We also tested Drex DLM Q8_0 on Apple Silicon through its custom Metal runtime. It answered the original 21 questions. A resource guard stopped 1 request. On the 23 completed inputs, it scored 21/23, versus Jev's 23/23, and accepted 4/11 eligible requests.Its 2 errors reduced combined requests to a single function; the confidence checks blocked both. Its timing and zero provider invoice are outside the API charts. The weights carry CC BY-NC 4.0, so this was noncommercial research. Our Apple Silicon serving tests cover the separate question of local inference capacity.
For a decision API, compare the correct answers your application can accept, complete response time and provider charge. A model's confidence scores need calibration on your task before they can govern action.A CPU-hosted specialist becomes attractive when one bounded task repeats often and spare compute is available. Fine-tuning improved the encoder's score, and warm inference took tens of milliseconds. Its narrower accepted coverage means the economics depend on how often it finishes the decision and avoids an API call.For either approach, fix the confidence policy before testing fresh cases and measure the complete path, including review or fallback. All 9 APIs passed our 2 short context probes, but those probes do not establish long-document reliability; Clef's listed truncation boundary remains unverified by this study. Small benchmark wins are starting points for a workload-specific comparison.