Skip to content

6. Results

jev, the hosted reference, answered 65 of the 67 router queries correctly in both modes, at 466 ms per query. In September, one local method matched it on translated text: rune-26b-a4b at Q3_K_M on llama-server scored 65/67 at 367 ms from a 13.5 GB file, but only 57/67 on the original text. Rune Q5 on pcdServer now matches jev in both modes (65/67 each), using CPU weights at 962 ms/query from 19.13 GB. The best local model near 90 ms remains Qwen3.5-4B-Hmm on pcdServer: 64/67 on translated text and 65/67 on direct text, at 88 ms per query from a 4.5 GB file, or 93 ms from the 2.7 GB Q4_K_M file. The dedicated decider-0.8b on dohnuts (Metal) scored 62/67 and 61/67 at 52 ms from the 0.81 GB DreamBlooms Q8_0 file that ornotto registers. Among encoders, GLiNER2.5-Decide scored 61/67 on translated text in 21.9 ms on Core ML CPU/GPU, and laya 52/67 in 7 ms. 320 configurations finished; failed and refused alternatives are excluded from the result tables.

The jev and openrouter@typesafe/jev-1.13 rows are two measurements of the same TypeSafe Jev model through OpenRouter. TypeSafe's model page listed no other version in September 2026, and both public aliases, jev-latest and jev-preview, pointed to it when we checked on 30 September.1 Chapter 1 explains why the version behind an alias matters. Every benchmark number in this chapter comes from our own 67-query routing test. Local inference ran on one Mac; remote calls used that Mac as the client, with provider hardware undisclosed. Numbers that other people published about the same models are collected separately, in Numbers others report, and are never mixed into our tables.

Native Laya MLX and xDecision

Three new configurations completed the same 106 distinct inputs and both 67-query modes on 3 October. All 302 earlier configurations and their frozen translations are retained. Each model ran alone on MLX with GPU selected; HTTP and tokenization use CPU. This is configured placement, not an operation trace. The native heads return choice, noul and score; the router uses choice.

native runtime / storage translated direct translated ms direct ms weights GB
Laya MLX 0.3.0 / FP16 52/67 49/67 11.4 11.2 0.64
xDecision / F16 44/67 43/67 14.9 15.1 0.70
xDecision / Q8_0 45/67 43/67 14.3 14.2 0.40

xDecision Q8_0 expands to dense FP16 at load. Its size column measures the stored file, not resident RAM. Its runtime is pinned to 4082689 and weights to d97731bc72664e06722e5ca64fcdbc8e0e2f9c95. Laya's native checkpoint is pinned to f2b4faf51023039425946074e2cf1361d2db11d5. These are separate runtime configurations, not replacements for earlier Laya rows. Calibration on this routing task is unverified. Both runtimes are now available through the package's optional native engines.

OpenRouter: nine remote models, ten runs

The fresh typesafe/jev-1.13 run scored 65/67 in both modes, at 364.4 ms translated and 375.1 ms direct. Together Tev1 scored 64/67 translated at 339.3 ms and 65/67 direct at 335.1 ms. Kev 4B also scored 65/67 in both, at 1,653.0 ms translated and 1,525.0 ms direct. The earlier jev measurement and typesafe/jev-1.13 are the same TypeSafe Jev model called through OpenRouter at different times. Both measurements are retained; differences in end-to-end timing across dates are not a controlled model-speed comparison.

All nine requested endpoints were measured on the same 67 queries, in translated and direct modes, on 3 October 2026. Translations and local results are retained from the earlier runs. Each endpoint received 106 distinct inputs, sequentially, with duplicate inputs reused across modes. Latency includes the network trip and provider processing; R means the provider did not disclose CPU/GPU placement or weight precision.

Six endpoints use a native five-way choice. Respan's three endpoints support noul only: each request contains five independent yes/no questions using the same task descriptions. We choose the largest P(true), then normalize the five values for the routing report. This noul OVR readout is labelled separately; its values are not calibrated exclusive-choice probabilities. Choose OpenRouter API in the explorer's engine filter to select ten recorded runs of nine endpoints. Its table shows the readout for each run.

Full 67-query runs in each mode; network and provider processing included. R: undisclosed remote CPU/GPU.
ModelRunResolved versionC/GReadoutTranslatedDirectTranslated msDirect ms
typesafe/jev-1.133 October 2026typesafe/jev-1.13-20260917Rnative choice65/6765/67364.4375.1
typesafe/jev-1.13earlier runnot loggedRnative choice65/6765/67466.1455.2
jaredpalmer/kev-4b3 October 2026jaredpalmer/kev-4b-20260924Rnative choice65/6765/671,653.01,525.0
togethercomputer/tev1-4b-experimental3 October 2026togethercomputer/tev1-4b-experimental-20260923Rnative choice64/6765/67339.3335.1
inception/mercury-decide:free3 October 2026inception/mercury-decide-20260930Rnative choice64/6764/67408.3404.5
liquid/d13 October 2026liquid/d1-20260930Rnative choice64/6764/67566.2583.6
upstage/solar-decide3 October 2026upstage/solar-decide-20260928Rnative choice62/6763/671,253.31,292.3
respan/span-01-lite:free3 October 2026respan/span-01-lite-20260925Rnoul OVR56/6760/67436.0474.6
respan/span-01-lite3 October 2026respan/span-01-lite-20260925Rnoul OVR56/6760/67447.5462.6
respan/span-013 October 2026respan/span-01-20260925Rnoul OVR56/6760/67487.4566.3

Issue 102: October additions

The 32 additions were measured on 2 October 2026. They use the same 67 queries and recorded translation choices as the earlier round. Each model ran alone. The 258 earlier result rows are unchanged.

method translated direct ms/query weights GB C/G
pcdserver@rune-mradermacher-26b-a4b-q5 65/67 65/67 962.3 19.13 C+G
pcdserver@rune-mradermacher-26b-a4b-q3 64/67 64/67 146.1 13.29 G
dohnuts-metal@jpt-4b-q8 64/67 64/67 335.6 4.48 G
pcdserver@rune-mradermacher-26b-a4b-q8 64/67 64/67 758.1 26.86 C
pcdserver@jpt-4b-q8 63/67 64/67 86.3 4.48 G
pcdserver@jpt-4b-q4 63/67 64/67 93.0 2.71 G
pcdserver@jpt-9b-q5 63/67 64/67 155.2 6.47 G
dohnuts-metal@jpt-4b-q5 63/67 64/67 367.0 3.07 G
pcdserver@rune-mradermacher-26b-a4b-q4-metal 63/67 64/67 124.8 16.80 G
pcdserver@rune-mradermacher-26b-a4b-q4 63/67 64/67 487.1 16.80 C+G
dohnuts-metal@jpt-9b-q5 63/67 64/67 619.6 6.47 G
dohnuts-metal@jpt-9b-q4 63/67 64/67 619.8 5.63 G
dohnuts-metal@jpt-9b-q8 63/67 64/67 624.7 9.53 G
pcdserver@jpt-4b-q5 62/67 63/67 99.4 3.07 G
dohnuts-metal@jet-4b-q8 62/67 63/67 281.4 4.48 G
dohnuts-metal@jpt-4b-q4 62/67 62/67 290.6 2.71 G
dohnuts-metal@jpt-9b-q3 61/67 64/67 636.7 4.62 G
pcdserver@jpt-9b-q4 61/67 63/67 137.3 5.63 G
pcdserver@jpt-9b-q8 61/67 63/67 141.8 9.53 G
dohnuts-metal@jpt-4b-q3 61/67 63/67 315.0 2.26 G
coreml@kev-0.8b-fp16-l512 60/67 62/67 121.3 1.51 C+G
pcdserver@jpt-9b-q3 60/67 62/67 139.2 4.62 G
pcdserver@jpt-4b-q3 60/67 60/67 93.7 2.26 G
mlx@tev1-0.8b-4bit 60/67 59/67 64.8 0.42 G
dohnuts-metal@tev1-0.8b-q8 59/67 62/67 61.5 0.81 G
dohnuts-metal@this-that-model-1.2-q3 57/67 56/67 113.1 1.10 G
systemone@kev-ggmlc-0.8b-q8 57/67 45/67 199.8 0.83 G
systemone@bosun-1.7b-q5 55/67 55/67 130.8 1.26 G
dohnuts-metal@this-that-model-1.2-q5 54/67 57/67 117.4 1.41 G
systemone@kev-ggmlc-0.8b-q4 54/67 43/67 207.2 0.96 G
dohnuts-metal@this-that-model-1.2-q8 53/67 57/67 107.8 2.01 G
dohnuts-metal@this-that-model-1.2-q4 51/67 51/67 110.5 1.27 G
dohnuts-metal@this-that-model-1.2-q2 21/67 18/67 112.2 0.97 G

Kev uses the FluidInference Core ML export, the L512/K16 single-question row on CPU and GPU. This is not the fused multi-question path. Its fitted temperature is already in the graph. Tev1 uses the native decision prompt with thinking disabled and a single forward pass over option-letter logits, on dohnuts Q8 and MLX 4-bit. Its release licence is unresolved, and temperature 1.0 does not establish calibration.

Bosun Q5_K_M runs on llama-cpp-python with Metal. Its native prompt shuffles candidates into stable slots. We verified the prompt against the pinned source renderer and tokenizer ChatML template, checked all 256 decision-token IDs, read final-position logits for the valid slots and restored the caller's order. Temperature is 1.0; these measurements do not establish calibration. Hanno's server implements native Bosun through Transformers; its generic llama backend does not implement this GGUF readout.

The mys Kev Q4 and Q8 files ran on ggmlc with Metal. Both fail to load in current dohnuts with unknown model architecture: 'ggmlc'. This is an observed runtime incompatibility, so the two measurements use the compiler's own runtime. The Q4 file is larger than Q8 because its dynamic quantization preserves additional tensors at higher precision.

Rune's mradermacher conversions now run on pcdServer through the fork's Jinja chat-template fallback. Q5_K_M scores 65/67 in both modes at 962 ms/query from 19.13 GB. Q3_K_M scores 64/67 in both modes at 146 ms from 13.29 GB. These are separate conversions and a separate readout from the earlier surogate Rune rows. The original Q4 and Q5 runs keep weights on CPU with Metal computation enabled; Q8 runs entirely on CPU after device-offload attempts exceeded the guards. The fresh Q4 full-Metal rerun scores 63/64 at 124.8 ms (3.9× faster than the earlier Q4 at 487.1 ms), and is retained as a separate row. The new Q5 full-Metal run is separate: 64/64 at 153.3 ms translated and 152.8 ms direct, 6.3× faster than mixed Q5 but one fewer correct answer in each mode. The retained mixed Q5 and CPU Q8 timings are not a pure quantization comparison with the GPU runs. Swap grew during the larger-model attempts; each run remained guarded and each completed measurement ran alone.

JPT 4B and JPT 9B ran at Q3, Q4, Q5 and Q8 on both requested engines. The repositories publish no Q8: we converted their pinned BF16 files locally with llama-quantize Q8_0. The conversion manifests record source revisions, input/output hashes, converter hash, sizes and tensor types. JPT is CC BY-NC 4.0. Native dohnuts uses source-matched chat prompts, contextual answer tokens and the affirmative yes/no label; its temperatures are 1.036 (4B) and 1.087 (9B). The 4B Q8 native readout scores 64/67 in both modes; pcdServer is faster but scores 63/64.

Jet Q8 scores 62/67 translated and 63/67 direct on dohnuts, in 281 ms. Its score questions read digit logits and yes/no questions read the corresponding word logits. ThisThat 1.2 ran on dohnuts at every requested precision. Q3 scores 57/56, while Q2 falls to 21/18. Their native prompts and label mappings are part of the updated bundled engine, and response checks cover choice, yes/no and score in addition to the router benchmark.

All 24 requested weight variants are covered by these 32 measurements. The two mys Kev measurements use the documented ggmlc alternative, and the four JPT Q8 engine measurements use the documented local conversions. The earlier 258 exported rows remain field-identical.

The top of the table

The fifteen best methods, ordered by translated accuracy, then direct accuracy, then speed:

The 15 best methods by translated score, then direct score, then speed. Click to sort.
EngineC/GModelFamilyQuantGBms/queryTranslatedDirect
OpenRouter APIRtypesafe/jev-1.13dedicated364.465/6765/67
pcdServerC+Grune-mradermacher-26b-a4bdedicatedq519.13962.365/6765/67
OpenRouter APIRjaredpalmer/kev-4bdedicated1,653.065/6765/67
slot (llama-server)Grune-26b-a4bdedicatedq313.52366.665/6757/67
pcdServerGqwen3.5-4b-hmmfine-tunedq84.4887.664/6765/67
pcdServerGqwen3.5-4b-hmmfine-tunedq42.7192.664/6765/67
pcdServerGqwen3.5-4b-hmmfine-tunedbf168.4294.864/6765/67
OpenRouter APIRtogethercomputer/tev1-4b-experimentaldedicated339.364/6765/67
ollayaGwinnow-12bdedicatedq8_012.67891.164/6765/67
pcdServerGrune-mradermacher-26b-a4bdedicatedq313.29146.164/6764/67
pcdServerGrune-mradermacher-26b-a4bdedicatedq519.13153.364/6764/67
dohnuts (Metal)Gdecider-35b-a3bdedicatedq421.17266.364/6764/67
System One serverGneohorse-4bdedicatedq85.17273.164/6764/67
slot (llama-server)Gdecider-35b-a3bdedicatedq421.17299.164/6764/67
dohnuts (Metal)Gjpt-4bdedicatedq84.48335.664/6764/67

Five findings stand out from the first round of 208 methods, three more from the next 50, and the October additions bring Rune Q5 to 65/67 in both modes.

  • A small, dedicated model gets within three queries of jev. decider-0.8b answers 62 of 67 through its own readout, from a file a fifth the size of the Hmm model's Q8 build. decider-0.8b's Q8_0 conversions are the fastest methods to score 62 or more (51 ms for mradermacher's, 52 ms for DreamBlooms'), and its probabilities are calibrated.
  • A tuned 4B chat model on pcdServer matches the 35B decider. Qwen3.5-4B-Hmm ties decider-35b-a3b on dohnuts on translated text (64 each) and beats it by one answer on direct text (65 against 64). Its 2.7 GB Q4 build is an eighth the size of the 35B file and runs three times faster. Its accuracy does not move from Q4 to BF16.
  • Vanilla chat models are close behind. unsloth's Qwen3.5-4B at Q8 scored 63/64 at 85 ms, one query behind the Hmm tune, with no task-specific training. Qwen3.5-2B at Q3_K_M scored 61/62 at 43 ms from 1.2 GB.
  • Size does not rescue an old model. The best Qwen1.5 method, at 4B, scored 42/67 in 248 ms. Qwen3.5 at 0.8B scored 57.
  • Speed and accuracy separate at the extremes. laya runs in 4 to 7 ms on Apple silicon but tops out at 52. kev-4b reaches 62 on translated text but takes 1.1 seconds per query.
  • The later dedicated models join the top, at a price in latency or memory. rune (65/67 translated, 13.5 GB), winnow in ollaya (64/67 and 65/67, 891 ms), Jev-Omni (64/67 in both modes, 1.2 seconds) and NeoHorse-Jev-4B (64/67 in both modes, 273 ms from 5.2 GB) all reach the top fifteen. None of them answers as fast as Qwen3.5-4B-Hmm on pcdServer, which scores 64/67 and 65/67 in 88 ms.
  • An English-only encoder gets within four answers of jev on translated text. GLiNER2.5-Decide scored 61/67 on Core AI, ONNX Runtime and Core ML alike, nine answers above laya, in 27 ms on Core AI. It cannot read the original text of a non-English query, so it needs the translator.
  • A new readout is no guarantee. The contrastive CLM scored 33 to 39, the entailment readouts 20 (ModernBERT NLI) and 51 (semif), GLiClass 26, and the small encoders Julia-1, Pulse Decide and decima-small 31 to 37. Apart from semif, they score no better than the old Qwen1.5 models, whose best was 42.

Every method

All 320 methods are in the table below. Click a column header to sort by it, and click again to reverse the order. Type in the box to keep only the rows that contain every word you type: pcdServer hmm keeps the Hmm model on pcdServer, and dedicated Metal keeps the dedicated models on dohnuts (Metal). Chapter 5 defines every column. An empty direct cell means the model was trained on English only and was sent the translated text alone. A method marked † carries a licence note, which its tooltip shows.

All 329 methods, best first. Click to sort; an empty cell was not measured, or does not apply (the direct score of a model trained on English only).
MethodEngineC/GModelFamilyQuantGBms/queryTranslatedDirectEnglishOther, translatedOther, directLoad ms
openrouter@typesafe/jev-1.13 †OpenRouter APIRtypesafe/jev-1.13dedicated364.465/6765/6726/2839/3939/39
slot@rune-26b-a4b-q3slot (llama-server)Grune-26b-a4bdedicatedq313.52366.665/6757/6726/2839/3931/39608
jev †OpenRouter APIRtypesafe/jev-1.13dedicated466.165/6765/6726/2839/3939/39
pcdserver@rune-mradermacher-26b-a4b-q5 †pcdServerC+Grune-mradermacher-26b-a4bdedicatedq519.13962.365/6765/6726/2839/3939/394,324
openrouter@jaredpalmer/kev-4b †OpenRouter APIRjaredpalmer/kev-4bdedicated1,653.065/6765/6726/2839/3939/39
pcdserver@qwen3.5-4b-hmm-q8pcdServerGqwen3.5-4b-hmmfine-tunedq84.4887.664/6765/6726/2838/3939/39462
pcdserver@qwen3.5-4b-hmm-q4pcdServerGqwen3.5-4b-hmmfine-tunedq42.7192.664/6765/6726/2838/3939/39296
pcdserver@qwen3.5-4b-hmm-bf16pcdServerGqwen3.5-4b-hmmfine-tunedbf168.4294.864/6765/6726/2838/3939/39341
pcdserver@rune-mradermacher-26b-a4b-q3pcdServerGrune-mradermacher-26b-a4bdedicatedq313.29146.164/6764/6725/2839/3939/39354
pcdserver@rune-mradermacher-26b-a4b-q5-metal †pcdServerGrune-mradermacher-26b-a4bdedicatedq519.13153.364/6764/6725/2839/3939/39507
dohnuts-metal@decider-35b-a3b-q4dohnuts (Metal)Gdecider-35b-a3bdedicatedq421.17266.364/6764/6726/2838/3938/39
systemone@neohorse-4b-q8System One serverGneohorse-4bdedicatedq85.17273.164/6764/6725/2839/3939/39
slot@decider-35b-a3b-q4slot (llama-server)Gdecider-35b-a3bdedicatedq421.17299.164/6764/6726/2838/3938/39375
slot@rune-26b-a4b-q4slot (llama-server)Grune-26b-a4bdedicatedq417.03335.364/6755/6726/2838/3929/39439
dohnuts-metal@jpt-4b-q8 †dohnuts (Metal)Gjpt-4bdedicatedq84.48335.664/6764/6725/2839/3939/39
openrouter@togethercomputer/tev1-4b-experimental †OpenRouter APIRtogethercomputer/tev1-4b-experimentaldedicated339.364/6765/6726/2838/3939/39
openrouter@inception/mercury-decide:free †OpenRouter APIRinception/mercury-decide:freededicated408.364/6764/6725/2839/3939/39
openrouter@liquid/d1 †OpenRouter APIRliquid/d1dedicated566.264/6764/6725/2839/3939/39
pcdserver@rune-mradermacher-26b-a4b-q8 †pcdServerCrune-mradermacher-26b-a4bdedicatedq826.86758.164/6764/6725/2839/3939/399,856
ollaya@winnow-12bollayaGwinnow-12bdedicatedq8_012.67891.164/6765/6726/2838/3939/3936,488
gguf-head@jev-omni-q3-last-mlx †llama.cpp + MLX headGjev-omnidedicatedq36.09908.464/6764/6726/2838/3938/391,193
gguf-head@jev-omni-q3llama.cpp embeddings + headC+Gjev-omnidedicatedq36.091,222.364/6764/6726/2838/3938/391,411
pcdserver@qwen3.5-4b-unsloth-q8pcdServerGqwen3.5-4b-unslothvanillaq84.4885.063/6764/6725/2838/3939/39346
pcdserver@jpt-4b-q8 †pcdServerGjpt-4bdedicatedq84.4886.363/6764/6725/2838/3939/39284
pcdserver@jpt-4b-q4 †pcdServerGjpt-4bdedicatedq42.7193.063/6764/6725/2838/3939/39318
pcdserver@qwen3.5-4b-unsloth-q5pcdServerGqwen3.5-4b-unslothvanillaq53.1494.563/6764/6725/2838/3939/39354
pcdserver@rune-mradermacher-26b-a4b-q4-metal †pcdServerGrune-mradermacher-26b-a4bdedicatedq416.80124.863/6764/6725/2838/3939/39332
pcdserver@jpt-9b-q5 †pcdServerGjpt-9bdedicatedq56.47155.263/6764/6725/2838/3939/39495
mlx@openjev-35b-a3b-4bitMLXGopenjev-35b-a3bdedicated4bit20.99250.063/6762/6724/2839/3938/3927,426
systemone@neohorse-4b-q3System One serverGneohorse-4bdedicatedq32.95296.163/6763/6724/2839/3939/39
pcdserver@openjev-35b-a3b-q4pcdServerGopenjev-35b-a3bdedicatedq421.17296.863/6764/6725/2838/3939/39338
pcdserver@decider-35b-a3b-q4pcdServerGdecider-35b-a3bdedicatedq421.17296.863/6763/6724/2839/3939/39799
slot@openjev-35b-a3b-q4slot (llama-server)Gopenjev-35b-a3bdedicatedq421.17311.363/6762/6724/2839/3938/39388
systemone@jeb-35b-a3b-q3System One serverGjeb-35b-a3bdedicatedq317.17333.263/6763/6724/2839/3939/39
torch@decider-4b-bf16PyTorchGdecider-4bdedicatedbf16348.263/6765/6726/2837/3939/3916,802
dohnuts-metal@jpt-4b-q5 †dohnuts (Metal)Gjpt-4bdedicatedq53.07367.063/6764/6725/2838/3939/39
pcdserver@rune-mradermacher-26b-a4b-q4 †pcdServerC+Grune-mradermacher-26b-a4bdedicatedq416.80487.163/6764/6725/2838/3939/393,170
systemone@clef-flash-mlx4 †Clef joint head (MLX)Gclef-flashdedicatedq45.28554.863/6763/6724/2839/3939/39
systemone@clef-flash-q4 †llama-server (System One)Gclef-flashdedicatedq4_k_m6.33610.763/6764/6725/2838/3939/39
systemone@clef-flash-q6 †llama-server (System One)Gclef-flashdedicatedq6_k8.28610.863/6764/6725/2838/3939/39
dohnuts-metal@jpt-9b-q5 †dohnuts (Metal)Gjpt-9bdedicatedq56.47619.663/6764/6725/2838/3939/39
dohnuts-metal@jpt-9b-q4 †dohnuts (Metal)Gjpt-9bdedicatedq45.63619.863/6764/6725/2838/3939/39
dohnuts-metal@jpt-9b-q8 †dohnuts (Metal)Gjpt-9bdedicatedq89.53624.763/6764/6725/2838/3939/39
systemone@clef-flash-q5 †llama-server (System One)Gclef-flashdedicatedq5_k_m7.36644.863/6764/6725/2838/3939/39
gguf-head@jev-omni-q8llama.cpp embeddings + headC+Gjev-omnidedicatedq812.671,124.163/6763/6725/2838/3938/391,157
gguf-head@jev-omni-q5llama.cpp embeddings + headC+Gjev-omnidedicatedq58.551,212.763/6763/6725/2838/3938/391,243
systemone@lev-4bSystem One serverGlev-4bdedicated9.492,203.863/6764/6725/2838/3939/39
dohnuts-metal@decider-0.8b-q8dohnuts (Metal)Gdecider-0.8bdedicatedq80.8151.462/6761/6724/2838/3937/39
dohnuts-metal@decider-0.8b-dreamblooms-q8dohnuts (Metal)Gdecider-0.8b-dreambloomsdedicatedq80.8152.262/6761/6724/2838/3937/39
pcdserver@qwen2.5-3b-q8pcdServerGqwen2.5-3bvanillaq83.2953.362/6757/6723/2839/3935/39178
pcdserver@qwen2.5-3b-f16pcdServerGqwen2.5-3bvanillaf166.1853.562/6760/6723/2839/3937/39252
dohnuts-metal@decider-0.8b-q6dohnuts (Metal)Gdecider-0.8bdedicatedq60.6353.562/6762/6724/2838/3938/39
dohnuts-metal@decider-0.8b-q5dohnuts (Metal)Gdecider-0.8bdedicatedq50.5855.262/6760/6724/2838/3936/39
pcdserver@qwen2.5-3b-q4pcdServerGqwen2.5-3bvanillaq41.9359.262/6760/6723/2839/3937/39339
pcdserver@qwen2.5-3b-q6pcdServerGqwen2.5-3bvanillaq62.5462.062/6760/6723/2839/3937/39213
slot@decider-0.8b-q8slot (llama-server)Gdecider-0.8bdedicatedq80.8164.062/6761/6724/2838/3937/39428
slot@decider-0.8b-dreamblooms-q8slot (llama-server)Gdecider-0.8b-dreambloomsdedicatedq80.8165.162/6761/6724/2838/3937/39689
slot@decider-0.8b-q5slot (llama-server)Gdecider-0.8bdedicatedq50.5867.962/6760/6724/2838/3936/39142
slot@decider-0.8b-q6slot (llama-server)Gdecider-0.8bdedicatedq60.6368.162/6762/6724/2838/3938/39216
pcdserver@qwen3.5-4b-q4pcdServerGqwen3.5-4bvanillaq43.0189.862/6763/6724/2838/3939/39311
pcdserver@qwen3.5-4b-unsloth-q6pcdServerGqwen3.5-4b-unslothvanillaq63.5393.662/6763/6724/2838/3939/39291
pcdserver@qwen3.8-4b-distill-q4pcdServerGqwen3.8-4b-distillfine-tunedq42.7893.662/6763/6724/2838/3939/39284
pcdserver@jpt-4b-q5 †pcdServerGjpt-4bdedicatedq53.0799.462/6763/6724/2838/3939/39345
pcdserver@qwen3.8-9b-distill-q4pcdServerGqwen3.8-9b-distillfine-tunedq45.78137.862/6764/6725/2837/3939/39449
systemone@system-one-minicpm5-2b-q8 †System One (MLX)Gsystem-one-minicpm5-2b-q8dedicatedq82.67141.862/6759/6724/2838/3935/39
dohnuts@decider-0.8b-dreamblooms-q8dohnuts (CPU)Cdecider-0.8b-dreambloomsdedicatedq80.81275.262/6760/6724/2838/3936/39
dohnuts-metal@jet-4b-q8dohnuts (Metal)Gjet-4bdedicatedq84.48281.462/6763/6724/2838/3939/39
slot@qwen3.5-4b-hmm-bf16slot (llama-server)Gqwen3.5-4b-hmmfine-tunedbf168.42284.262/6763/6724/2838/3939/39334
slot@qwen3.5-4b-hmm-q8slot (llama-server)Gqwen3.5-4b-hmmfine-tunedq84.48287.362/6763/6724/2838/3939/39285
dohnuts-metal@jpt-4b-q4 †dohnuts (Metal)Gjpt-4bdedicatedq42.71290.662/6762/6724/2838/3938/39
slot@qwen3.5-4b-hmm-q4slot (llama-server)Gqwen3.5-4b-hmmfine-tunedq42.71303.162/6763/6725/2837/3938/39293
splash@ornith-1.5-35b-a3b-splash †Splash (Metal)Gornith-1.5-35b-a3b-splashfine-tunedsplash-packed-q4-moe20.95322.462/6763/6724/2838/3939/39
slot@jevk5-4b-q4slot (llama-server)Gjevk5-4bdedicatedq42.71350.562/6763/6724/2838/3939/39429
systemone@imajev-4bSystem One serverGimajev-4bdedicated9.81363.162/6764/6725/2837/3939/39
pcdserver@bonsai-27b-q1pcdServerGbonsai-27bvanillaq13.80513.562/6763/6724/2838/3939/391,827
systemone@clef-flash-q3 †llama-server (System One)Gclef-flashdedicatedq3_k_m4.97668.362/6764/6725/2837/3939/39
kev-4b-gguflaya.cppGkev-4bdedicated1,103.162/6755/6724/2838/3931/39
openrouter@upstage/solar-decide †OpenRouter APIRupstage/solar-decidededicated1,253.362/6763/6724/2838/3939/39
coreml@gliner25-decide-fp16-l256-cpu-gpu †Core MLC+Ggliner25-decidededicatedfp160.9321.961/6722/2839/3913,151
coreai@gliner25-decideCore AIAgliner25-decidededicatedfp160.8726.961/6722/2839/393,378
pcdserver@qwen3.5-2b-q3pcdServerGqwen3.5-2bvanillaq31.2242.761/6762/6723/2838/3939/39203
dohnuts-metal@decider-0.8b-f16dohnuts (Metal)Gdecider-0.8bdedicatedf161.5251.961/6760/6723/2838/3937/39
dohnuts-metal@decider-0.8b-q4dohnuts (Metal)Gdecider-0.8bdedicatedq40.5353.361/6759/6723/2838/3936/39
pcdserver@qwen2.5-3b-q5pcdServerGqwen2.5-3bvanillaq52.2261.761/6757/6722/2839/3936/39300
slot@decider-0.8b-f16slot (llama-server)Gdecider-0.8bdedicatedf161.5263.661/6760/6723/2838/3937/39670
onnx@gliner25-decide-fp32ONNX RuntimeCgliner25-decidededicatedfp321.7467.361/6722/2839/394,607
slot@decider-0.8b-q4slot (llama-server)Gdecider-0.8bdedicatedq40.5367.461/6759/6723/2838/3936/39231
pcdserver@qwen3-4b-unsloth-bf16pcdServerGqwen3-4b-unslothvanillabf168.0571.561/6762/6723/2838/3939/39259
pcdserver@qwen3-4b-q4pcdServerGqwen3-4bvanillaq42.5075.561/6762/6723/2838/3939/39230
pcdserver@qwen3-4b-unsloth-q4pcdServerGqwen3-4b-unslothvanillaq42.5076.461/6761/6723/2838/3938/39247
pcdserver@qwen3.5-4b-supercoder-q4pcdServerGqwen3.5-4b-supercoderfine-tunedq42.6184.261/6762/6723/2838/3939/39351
pcdserver@jevk5-4b-q8pcdServerGjevk5-4bdedicatedq84.4884.261/6762/6723/2838/3939/39292
executorch@decider-0.8b-fp16ExecuTorch (MLX)Gdecider-0.8bdedicatedfp1689.761/6760/6723/2838/3937/39
pcdserver@jevk5-4b-q4pcdServerGjevk5-4bdedicatedq42.7189.961/6762/6723/2838/3939/39332
pcdserver@qwen3.5-4b-unsloth-iq3pcdServerGqwen3.5-4b-unslothvanillaiq31.9590.261/6762/6723/2838/3939/39800
pcdserver@qwen3.5-4b-unsloth-q2pcdServerGqwen3.5-4b-unslothvanillaq21.9493.561/6761/6722/2839/3939/39829
pcdserver@qwen3.5-4b-unsloth-q4pcdServerGqwen3.5-4b-unslothvanillaq42.7493.661/6762/6723/2838/3939/39297
pcdserver@qwen2.5-coder-7b-q4pcdServerGqwen2.5-coder-7bfine-tunedq44.68108.961/6762/6723/2838/3939/39386
pcdserver@qwen3-8b-q4pcdServerGqwen3-8bvanillaq45.03116.061/6762/6723/2838/3939/39367
pcdserver@qwen3.5-9b-unsloth-q4pcdServerGqwen3.5-9b-unslothvanillaq45.68136.961/6763/6724/2837/3939/39452
pcdserver@jpt-9b-q4 †pcdServerGjpt-9bdedicatedq45.63137.361/6763/6724/2837/3939/39478
pcdserver@jpt-9b-q8 †pcdServerGjpt-9bdedicatedq89.53141.861/6763/6724/2837/3939/39451
ollaya@decider-0.8bollayaCdecider-0.8bdedicatedfp321.53284.861/6760/6723/2838/3937/397,924
dohnuts-metal@jpt-4b-q3 †dohnuts (Metal)Gjpt-4bdedicatedq32.26315.061/6763/6724/2837/3939/39
slot@jevk5-4b-q8slot (llama-server)Gjevk5-4bdedicatedq84.48333.961/6763/6724/2837/3939/39372
coreml@gliner25-decide-fp16-l256Core MLAgliner25-decidededicatedfp160.93349.161/6722/2839/3982,062
dohnuts-metal@jpt-9b-q3 †dohnuts (Metal)Gjpt-9bdedicatedq34.62636.761/6764/6726/2835/3938/39
mlx@jev-omni-4bitMLXGjev-omnidedicated4bit6.81780.161/6761/6724/2837/3937/39
pcdserver@decider-2b-f16pcdServerGdecider-2bdedicatedf163.7842.260/6760/6722/2838/3938/39166
pcdserver@qwen2.5-coder-3b-q4pcdServerGqwen2.5-coder-3bfine-tunedq42.1060.260/6759/6724/2836/3935/39295
mlx@tev1-0.8b-4bit †MLXGtev1-0.8bdedicated4bit0.4264.860/6759/6721/2839/3938/393,482
pcdserver@qwen3-4b-unsloth-q8pcdServerGqwen3-4b-unslothvanillaq84.2870.360/6762/6723/2837/3939/39306
pcdserver@qwen3-4b-unsloth-q3pcdServerGqwen3-4b-unslothvanillaq32.0878.360/6761/6722/2838/3939/39706
pcdserver@qwen3-4b-unsloth-q6pcdServerGqwen3-4b-unslothvanillaq63.3179.360/6761/6722/2838/3939/39276
pcdserver@qwen3-4b-unsloth-q5pcdServerGqwen3-4b-unslothvanillaq52.8979.460/6762/6723/2837/3939/39321
pcdserver@qwen3.5-4b-unsloth-q3pcdServerGqwen3.5-4b-unslothvanillaq32.2992.760/6761/6722/2838/3939/39358
pcdserver@jpt-4b-q3 †pcdServerGjpt-4bdedicatedq32.2693.760/6760/6722/2838/3938/39374
coreml@kev-0.8b-fp16-l512 †Core MLC+Gkev-0.8bdedicatedfp161.51121.360/6762/6724/2836/3938/3911,368
pcdserver@jpt-9b-q3 †pcdServerGjpt-9bdedicatedq34.62139.260/6762/6723/2837/3939/39482
ollaya@kev-0.8bollayaCkev-0.8bdedicatedfp321.82268.260/6762/6724/2836/3938/394,848
pcdserver@decider-2b-dreamblooms-q8pcdServerGdecider-2b-dreambloomsdedicatedq82.0140.159/6760/6722/2837/3938/39198
dohnuts-metal@tev1-0.8b-q8 †dohnuts (Metal)Gtev1-0.8bdedicatedq80.8161.559/6762/6723/2836/3939/39
pcdserver@qwen3-4b-unsloth-iq2pcdServerGqwen3-4b-unslothvanillaiq21.5372.759/6757/6721/2838/3936/39605
dohnuts-metal@decider-2b-f16dohnuts (Metal)Gdecider-2bdedicatedf163.78100.359/6762/6724/2835/3938/39
dohnuts-metal@decider-2b-dreamblooms-q8dohnuts (Metal)Gdecider-2b-dreambloomsdedicatedq82.01103.759/6762/6724/2835/3938/39
slot@decider-2b-f16slot (llama-server)Gdecider-2bdedicatedf163.78112.459/6762/6724/2835/3938/39153
slot@decider-2b-dreamblooms-q8slot (llama-server)Gdecider-2b-dreambloomsdedicatedq82.01114.459/6762/6724/2835/3938/39199
executorch@decider-2b-fp16ExecuTorch (MLX)Gdecider-2bdedicatedfp16251.359/6762/6724/2835/3938/39
pcdserver@decider-0.8b-q5pcdServerGdecider-0.8bdedicatedq50.5829.957/6757/6722/2836/3935/39180
pcdserver@qwen3.5-2b-q8pcdServerGqwen3.5-2bvanillaq82.0839.557/6759/6720/2838/3939/39189
pcdserver@qwen3.5-2b-unsloth-q8pcdServerGqwen3.5-2b-unslothvanillaq82.0139.557/6759/6720/2838/3939/39126
pcdserver@decider-0.8b-q4pcdServerGdecider-0.8bdedicatedq40.5340.257/6757/6722/2836/3935/39583
pcdserver@decider-2b-iq4pcdServerGdecider-2bdedicatediq41.2441.457/6757/6721/2837/3937/39167
pcdserver@qwen3.5-2b-unsloth-bf16pcdServerGqwen3.5-2b-unslothvanillabf163.7842.257/6759/6720/2838/3939/39328
pcdserver@qwen3.5-2b-unsloth-q4pcdServerGqwen3.5-2b-unslothvanillaq41.2844.857/6757/6719/2839/3938/39139
dohnuts-metal@decider-0.8b-q3dohnuts (Metal)Gdecider-0.8bdedicatedq30.4754.557/6755/6724/2834/3931/39
pcdserver@qwen2.5-3b-q3pcdServerGqwen2.5-3bvanillaq31.5960.657/6751/6720/2838/3931/39672
ollaya@von-1.1ollayaCvon-1.1dedicatedfp321.5962.857/6722/2836/3914,254
slot@decider-0.8b-q3slot (llama-server)Gdecider-0.8bdedicatedq30.4769.757/6755/6724/2834/3931/39101
dohnuts-metal@this-that-model-1.2-q3dohnuts (Metal)Gthis-that-model-1.2dedicatedq31.10113.157/6756/6723/2835/3933/39
systemone@leo-1.7bSystem One serverGleo-1.7bdedicated3.52136.257/6762/6723/2835/3939/39
systemone@kev-ggmlc-0.8b-q8 †ggmlc (Metal)Gkev-0.8bdedicatedq80.83199.857/6745/6724/2834/3921/39
kev-0.8b-gguflaya.cppGkev-0.8bdedicated231.557/6745/6724/2834/3921/39
pcdserver@qwen3.5-0.8b-q5pcdServerGqwen3.5-0.8bvanillaq50.6532.157/6757/6721/2836/3937/39284
pcdserver@qwen3-1.7b-unsloth-q8pcdServerGqwen3-1.7b-unslothvanillaq81.8335.157/6759/6721/2836/3938/3995
pcdserver@qwen3-1.7b-q8pcdServerGqwen3-1.7bvanillaq82.1735.257/6759/6721/2836/3938/3994
pcdserver@qwen3-1.7b-bf16pcdServerGqwen3-1.7bvanillabf164.0736.257/6759/6721/2836/3938/39206
pcdserver@qwen3-1.7b-q5pcdServerGqwen3-1.7bvanillaq51.4738.757/6759/6721/2836/3938/39148
pcdserver@qwen3-1.7b-unsloth-q5pcdServerGqwen3-1.7b-unslothvanillaq51.2638.757/6759/6721/2836/3938/39170
pcdserver@qwen3-1.7b-unsloth-q6pcdServerGqwen3-1.7b-unslothvanillaq61.4239.257/6759/6721/2836/3938/39158
pcdserver@qwen3-1.7b-q6pcdServerGqwen3-1.7bvanillaq61.6739.357/6759/6721/2836/3938/39150
pcdserver@qwen3.5-2b-unsloth-iq3pcdServerGqwen3.5-2b-unslothvanillaiq30.9340.257/6757/6721/2836/3937/39435
pcdserver@qwen3.5-2b-unsloth-q3pcdServerGqwen3.5-2b-unslothvanillaq31.1141.957/6759/6720/2837/3939/39165
pcdserver@qwen3.5-2b-q5pcdServerGqwen3.5-2bvanillaq51.5742.357/6757/6719/2838/3939/39196
pcdserver@qwen3.5-2b-q6pcdServerGqwen3.5-2bvanillaq61.7042.657/6757/6719/2838/3939/39134
pcdserver@qwen3.5-2b-unsloth-q5pcdServerGqwen3.5-2b-unslothvanillaq51.4443.557/6757/6719/2838/3938/39142
pcdserver@qwen3.5-2b-q4pcdServerGqwen3.5-2bvanillaq41.4047.257/6757/6719/2838/3939/39217
onnx@von-1.2ONNX RuntimeCvon-1.2dedicatedfp321.5852.657/6722/2835/392,485
ollaya@decision-eosollayaCdecision-eosdedicatedfp321.54322.957/6756/6723/2834/3933/395,638
pcdserver@qwen3.5-0.8b-unsloth-q5pcdServerGqwen3.5-0.8b-unslothvanillaq50.5929.656/6757/6721/2835/3936/39184
pcdserver@qwen3-1.7b-unsloth-q4pcdServerGqwen3-1.7b-unslothvanillaq41.1140.256/6757/6720/2836/3938/39194
pcdserver@qwen3.5-2b-unsloth-q6pcdServerGqwen3.5-2b-unslothvanillaq61.5743.856/6757/6719/2837/3939/39132
dohnuts-metal@decider-2b-iq4dohnuts (Metal)Gdecider-2bdedicatediq41.24106.756/6757/6723/2833/3935/39
slot@decider-2b-iq4slot (llama-server)Gdecider-2bdedicatediq41.24126.856/6757/6723/2833/3935/39719
openrouter@respan/span-01-lite:free †OpenRouter APIRrespan/span-01-lite:freededicated436.056/6760/6721/2835/3939/39
openrouter@respan/span-01-lite †OpenRouter APIRrespan/span-01-litededicated447.556/6760/6721/2835/3939/39
openrouter@respan/span-01 †OpenRouter APIRrespan/span-01dedicated487.456/6760/6721/2835/3939/39
pcdserver@decider-0.8b-dreamblooms-q8pcdServerGdecider-0.8b-dreambloomsdedicatedq80.8128.355/6757/6720/2835/3937/3978
pcdserver@qwen3.5-0.8b-unsloth-q8pcdServerGqwen3.5-0.8b-unslothvanillaq80.8128.755/6757/6720/2835/3937/39177
pcdserver@qwen3.5-0.8b-q6pcdServerGqwen3.5-0.8bvanillaq60.6929.055/6757/6720/2835/3937/39450
pcdserver@qwen3.5-0.8b-unsloth-q6pcdServerGqwen3.5-0.8b-unslothvanillaq60.6429.255/6757/6720/2835/3937/39186
pcdserver@decider-0.8b-q6pcdServerGdecider-0.8bdedicatedq60.6329.855/6757/6720/2835/3937/39280
pcdserver@qwen3.5-0.8b-bf16pcdServerGqwen3.5-0.8bvanillabf161.5630.055/6757/6720/2835/3937/39407
pcdserver@decider-0.8b-q8pcdServerGdecider-0.8bdedicatedq80.8130.255/6757/6720/2835/3937/39208
pcdserver@qwen3.5-0.8b-ggml-bf16pcdServerGqwen3.5-0.8b-ggmlvanillabf161.5630.355/6757/6720/2835/3937/39337
pcdserver@decider-0.8b-f16pcdServerGdecider-0.8bdedicatedf161.5230.755/6756/6719/2836/3937/39438
pcdserver@qwen3.5-0.8b-unsloth-bf16pcdServerGqwen3.5-0.8b-unslothvanillabf161.5231.055/6757/6720/2835/3937/39208
pcdserver@qwen3.5-0.8b-q8pcdServerGqwen3.5-0.8bvanillaq80.8331.255/6757/6720/2835/3937/39252
pcdserver@qwen3.5-0.8b-unsloth-q4pcdServerGqwen3.5-0.8b-unslothvanillaq40.5350.155/6754/6720/2835/3934/39255
systemone@bosun-1.7b-q5 †llama-cpp-python (Metal)Gbosun-1.7bdedicatedq51.26130.855/6755/6722/2833/3933/39
pcdserver@pulse-0.6b-q4pcdServerGpulse-0.6bfine-tunedq40.4027.954/6752/6721/2833/3931/39196
pcdserver@bonsai-1.7b-q2pcdServerGbonsai-1.7bvanillaq20.4934.954/6755/6719/2835/3936/39259
pcdserver@qwen3-1.7b-q2pcdServerGqwen3-1.7bvanillaq20.8837.554/6750/6721/2833/3929/39104
pcdserver@qwen3-1.7b-q4pcdServerGqwen3-1.7bvanillaq41.2840.754/6757/6720/2834/3938/39343
gguf-head@laya-neutron-mlllama.cpp embeddings + headC+Glaya-multilingualdedicatedq4_00.2742.754/6751/6721/2833/3930/392,739
gguf-head@laya-neutron-ml-mps †llama.cpp + MPS headGlaya-multilingualdedicatedq4_00.2750.354/6751/6721/2833/3930/394,915
dohnuts-metal@this-that-model-1.2-q5dohnuts (Metal)Gthis-that-model-1.2dedicatedq51.41117.454/6757/6721/2833/3937/39
systemone@kev-ggmlc-0.8b-q4 †ggmlc (Metal)Gkev-0.8bdedicatedq40.96207.254/6743/6723/2831/3920/39
pcdserver@qwen3-0.6b-q8pcdServerGqwen3-0.6bvanillaq80.6425.553/6754/6718/2835/3936/39191
pcdserver@qwen3.5-0.8b-ggml-q4pcdServerGqwen3.5-0.8b-ggmlvanillaq40.5628.853/6755/6719/2834/3936/39801
pcdserver@qwen3-1.7b-unsloth-q3pcdServerGqwen3-1.7b-unslothvanillaq30.9437.653/6756/6718/2835/3938/39216
pcdserver@qwen3.5-2b-unsloth-q2pcdServerGqwen3.5-2b-unslothvanillaq20.9741.353/6748/6719/2834/3929/39255
mlx@laya-ml-mxfp8 †MLXGlaya-multilingualdedicatedmxfp80.3257.853/6749/6720/2833/3929/392,639
pcdserver@qwen2.5-3b-q2pcdServerGqwen2.5-3bvanillaq21.2758.353/6746/6717/2836/3929/39271
dohnuts-metal@this-that-model-1.2-q8dohnuts (Metal)Gthis-that-model-1.2dedicatedq82.01107.853/6757/6720/2833/3938/39
systemone@system-one-scorer-onnx †ONNX scalar (CPU)Csystem-one-scorer-onnxdedicatedq42.641,410.853/6723/2830/39
laya-mlxMLXGlaya-multilingualdedicated6.952/6749/6721/2831/3928/391,797
laya-coreaiCore AIAlaya-multilingualdedicated10.052/6749/6721/2831/3928/393,164
systemone@laya-mlx-package-fp16 †Laya MLX (native)Glaya-multilingualdedicatedfp160.6411.452/6749/6721/2831/3928/39
ollaya@laya-multilingualollayaGlaya-multilingualdedicatedfp320.6812.352/6749/6721/2831/3928/391,217
laya-coreai-mlbCore AIAlaya-multilingualdedicated19.452/6749/6721/2831/3928/392,404
laya-ml-onnx-fp32ONNX RuntimeClaya-multilingualdedicatedfp3223.452/6749/6721/2831/3928/39825
onnx@laya-ml-fp32ONNX RuntimeClaya-multilingualdedicatedfp321.2924.752/6749/6721/2831/3928/391,578
laya-ml-onnx-int8ONNX RuntimeClaya-multilingualdedicatedint825.552/6749/6721/2831/3928/39741
pcdserver@decider-0.8b-q3pcdServerGdecider-0.8bdedicatedq30.4729.352/6747/6718/2834/3929/39161
pcdserver@qwen3.5-0.8b-q4pcdServerGqwen3.5-0.8bvanillaq40.5830.252/6751/6717/2835/3934/39258
laya-wd-gguf-q8llama.cpp embeddingsC+Glaya-multilingualdedicatedq844.452/6749/6721/2831/3928/392,337
laya-wd-gguf-f16llama.cpp embeddingsC+Glaya-multilingualdedicatedf1644.452/6749/6721/2831/3928/392,437
laya-wd-gguf-q8-mps †llama.cpp + MPS headGlaya-multilingualdedicatedq80.3745.952/6749/6721/2831/3928/3986,773
laya-ml-gguflaya.cppClaya-multilingualdedicated57.652/6749/6721/2831/3928/39
onnx@laya-ml-fp16ONNX RuntimeClaya-multilingualdedicatedfp160.6557.652/6749/6721/2831/3928/391,067
laya-ml-gguf-f16laya.cppClaya-multilingualdedicatedf1661.452/6749/6721/2831/3928/39
pcdserver@qwen3.5-4b-unsloth-iq2pcdServerGqwen3.5-4b-unslothvanillaiq21.5287.052/6754/6716/2836/3938/39659
laya-wd-gguf-q6llama.cpp embeddingsC+Glaya-multilingualdedicatedq644.551/6747/6720/2831/3927/392,276
laya-ml-gguf-q4laya.cppClaya-multilingualdedicatedq455.051/6747/6718/2833/3929/39
dohnuts-metal@this-that-model-1.2-q4dohnuts (Metal)Gthis-that-model-1.2dedicatedq41.27110.551/6751/6719/2832/3932/39
mlx@semif-4b-nliMLXGsemif-4b-nlidedicated8.41120.651/6749/6719/2832/3930/3913,083
pcdserver@pulse-0.6b-q8pcdServerGpulse-0.6bfine-tunedq80.6423.250/6747/6720/2830/3927/3956
pcdserver@lq-decide-1.7b-q8pcdServerGlq-decide-1.7bfine-tunedq81.8335.150/6746/6720/2830/3926/39116
pcdserver@qwen3-1.7b-q3pcdServerGqwen3-1.7bvanillaq31.0737.750/6753/6716/2834/3937/39109
pcdserver@qwen3.5-2b-q2pcdServerGqwen3.5-2bvanillaq21.0741.549/6749/6719/2830/3930/39127
pcdserver@qwen3.8-2b-distill-q4pcdServerGqwen3.8-2b-distillfine-tunedq41.3142.949/6751/6719/2830/3932/39154
laya-wd-gguf-q3llama.cpp embeddingsC+Glaya-multilingualdedicatedq345.349/6743/6717/2832/3926/393,284
systemone@system-one-qwen3-0.6b †System One (MLX)Gsystem-one-qwen3-0.6bdedicatedbf16+adapter1.5247.749/6752/6721/2828/3931/39
pcdserver@qwen2.5-3b-iq2pcdServerGqwen2.5-3bvanillaiq21.1455.549/6749/6716/2833/3933/39427
laya-coreml-ane-w8-compactCore ML, Neural EngineC+Nlaya-multilingualdedicatedw84.548/6749/6717/2831/3932/3921,052
laya-typed-decisionsPyTorchGlaya-typed-decisionsdedicated17.748/6743/6718/2830/3925/394,793
pcdserver@qwen3.5-0.8b-unsloth-q3pcdServerGqwen3.5-0.8b-unslothvanillaq30.4730.148/6748/6717/2831/3931/39186
laya-coreml-ane-compactCore ML, Neural EngineC+Nlaya-multilingualdedicated4.247/6749/6717/2830/3932/3920,130
laya-mlx-compactMLXGlaya-multilingualdedicated7.947/6749/6717/2830/3932/3911,010
pcdserver@bonsai-1.7b-q1pcdServerGbonsai-1.7bvanillaq10.2534.947/6741/6714/2833/3927/39243
laya-wd-gguf-q5llama.cpp embeddingsC+Glaya-multilingualdedicatedq545.547/6743/6717/2830/3926/392,350
pcdserver@qwen3.5-0.8b-q3pcdServerGqwen3.5-0.8bvanillaq30.5231.346/6740/6715/2831/3925/39218
gguf-head@laya-neutron-enllama.cpp embeddings + headC+Glaya-endedicatedq4_00.2962.446/6716/2830/391,664
systemone@xdecision-q8-mlx †xDecision (MLX)GxDecisiondedicatedq80.4014.345/6743/6717/2828/3926/39
pcdserver@qwen3.8-2b-distill-q5pcdServerGqwen3.8-2b-distillfine-tunedq51.4546.145/6745/6717/2828/3928/39140
systemone@xdecision-f16-mlx †xDecision (MLX)GxDecisiondedicatedf160.7014.944/6743/6716/2828/3927/39
pcdserver@qwen3-1.7b-unsloth-q2pcdServerGqwen3-1.7b-unslothvanillaq20.7837.444/6746/6718/2826/3928/39276
pcdserver@qwen3.8-2b-distill-q8pcdServerGqwen3.8-2b-distillfine-tunedq82.0839.544/6745/6717/2827/3928/39129
pcdserver@qwen3.5-2b-unsloth-iq2pcdServerGqwen3.5-2b-unslothvanillaiq20.7740.844/6739/6715/2829/3924/39129
mlx@laya-ml-mxfp4 †MLXGlaya-multilingualdedicatedmxfp40.1657.944/6744/6716/2828/3928/392,438
pcdserver@qwen3.5-0.8b-unsloth-q2pcdServerGqwen3.5-0.8b-unslothvanillaq20.4228.543/6742/6714/2829/3928/39764
pcdserver@qwen3.8-2b-distill-q6pcdServerGqwen3.8-2b-distillfine-tunedq61.6143.543/6743/6716/2827/3927/39122
ollaya@laya-enollayaGlaya-endedicatedfp320.8519.442/6716/2826/391,804
systemone@system-one-distilled †Transformers NLI (MPS)Gsystem-one-distilleddedicatedfp320.2821.242/6717/2825/39
systemone@system-one-gold †Transformers NLI (MPS)Gsystem-one-golddedicatedfp320.2821.442/6717/2825/39
pcdserver@zerocoder-0.8b-q2pcdServerGzerocoder-0.8bfine-tunedq20.3834.742/6736/6715/2827/3921/39339
systemone@raz-nli-xsmall-openjev †Raz NLI (MPS)Graz-nli-xsmall-openjevdedicatedfp320.2849.442/6714/2828/39
pcdserver@qwen1.5-4b-q8pcdServerGqwen1.5-4bvanillaq84.20247.842/6743/6716/2826/3927/39322
pcdserver@lq-decide-1.7b-q4pcdServerGlq-decide-1.7bfine-tunedq41.1143.841/6742/6715/2826/3927/39108
pcdserver@qwen1.5-4b-q4pcdServerGqwen1.5-4bvanillaq42.46232.141/6743/6716/2825/3927/39391
pcdserver@qwen1.5-4b-q2pcdServerGqwen1.5-4bvanillaq21.62233.041/6742/6716/2825/3926/39444
pcdserver@qwen3-reranker-0.6b-q5pcdServerGqwen3-reranker-0.6bfine-tunedq50.4424.740/6744/6715/2825/3929/3958
pcdserver@qwen3.5-0.8b-q2pcdServerGqwen3.5-0.8bvanillaq20.4632.240/6742/6713/2827/3929/39541
pcdserver@qwen1.5-4b-q6pcdServerGqwen1.5-4bvanillaq63.25275.740/6743/6716/2824/3927/39309
systemone@clm-8b-q8System One serverC+Gclm-8bdedicatedq88.25107.639/6728/6714/2825/3915/39
pcdserver@qwen1.5-4b-q3pcdServerGqwen1.5-4bvanillaq32.03237.239/6743/6716/2823/3927/39441
pcdserver@qwen1.5-4b-q5pcdServerGqwen1.5-4bvanillaq52.84283.639/6742/6716/2823/3926/39424
systemone@clm-8b-mlx6 †CLM embeddings + head (MLX/CPU)C+Gclm-8bdedicatedq67.24415.739/6714/2825/39
ollaya@clm-8bollayaCclm-8bdedicatedbf1616.501,677.939/6730/6714/2825/3916/3949,184
systemone@clm-8b-mlx8System One serverC+Gclm-8bdedicatedmlx89.29100.438/6731/6714/2824/3917/39
systemone@clm-8b-q5System One serverC+Gclm-8bdedicatedq55.78119.238/6727/6713/2825/3914/39
systemone@clm-8b-q6-k †CLM embeddings + head (Metal/CPU)C+Gclm-8bdedicatedq6_k7.10490.738/6713/2825/39
systemone@clm-8b-q5-km †CLM embeddings + head (Metal/CPU)C+Gclm-8bdedicatedq5_k_m6.31509.938/6713/2825/39
onnx@decima-small-int8ONNX RuntimeCdecima-smalldedicatedint80.1312.437/6738/6713/2824/3925/392,290
systemone@system-one-zeroshot †Transformers NLI (MPS)Gsystem-one-zeroshotdedicatedfp320.2821.237/6714/2823/39
laya-wd-gguf-q4llama.cpp embeddingsC+Glaya-multilingualdedicatedq443.336/6738/6714/2822/3924/392,243
systemone@clm-8b-q4System One serverC+Gclm-8bdedicatedq45.03113.836/6728/6713/2823/3916/39
systemone@clm-8b-q4-km †CLM embeddings + head (Metal/CPU)C+Gclm-8bdedicatedq4_k_m5.57449.436/6713/2823/39
pcdserver@qwen1.5-0.5b-q3pcdServerGqwen1.5-0.5bvanillaq30.3521.235/6732/6711/2824/3921/39248
pcdserver@lyco-router-0.6b-q4pcdServerGlyco-router-0.6bfine-tunedq40.4826.435/6733/6712/2823/3921/39514
torch@pulse-decide-150m †PyTorchGpulse-decide-150mdedicated0.6017.934/6715/2819/3910,043
pcdserver@qwen3.5-0.8b-unsloth-iq3pcdServerGqwen3.5-0.8b-unslothvanillaiq30.4028.234/6731/6712/2822/3919/39246
pcdserver@qwen1.5-1.8b-q3pcdServerGqwen1.5-1.8bvanillaq31.0241.834/6735/6715/2819/3920/39510
pcdserver@qwen3-reranker-0.6b-q6pcdServerGqwen3-reranker-0.6bfine-tunedq60.4924.533/6737/6713/2820/3924/3961
pcdserver@qwen1.5-1.8b-q4pcdServerGqwen1.5-1.8bvanillaq41.2240.033/6739/6715/2818/3924/39343
torch@lumma-fev-0.6bPyTorchGlumma-fev-0.6bdedicated1.3069.533/6728/6714/2819/3914/396,272
systemone@clm-8b-mlx4System One serverC+Gclm-8bdedicatedmlx45.2089.633/6728/6711/2822/3918/39
pcdserver@zerocoder-0.8b-q3pcdServerGzerocoder-0.8bfine-tunedq30.4534.632/6730/6711/2821/3919/39291
pcdserver@qwen1.5-1.8b-q8pcdServerGqwen1.5-1.8bvanillaq81.9636.432/6736/6713/2819/3923/39106
pcdserver@qwen1.5-1.8b-q5pcdServerGqwen1.5-1.8bvanillaq51.3841.332/6735/6713/2819/3922/39327
mlx@julia-1MLXGjulia-1dedicated0.586.831/6731/6711/2820/3920/391,866
onnx@julia-1ONNX RuntimeCjulia-1dedicatedfp320.5812.231/6731/6711/2820/3920/39976
pcdserver@qwen1.5-1.8b-q6pcdServerGqwen1.5-1.8bvanillaq61.5839.831/6737/6714/2817/3923/39334
pcdserver@qwen1.5-0.5b-q6pcdServerGqwen1.5-0.5bvanillaq60.5122.630/6728/678/2822/3921/39197
pcdserver@qwen3-reranker-0.6b-q8pcdServerGqwen3-reranker-0.6bfine-tunedq80.6423.330/6733/6712/2818/3921/3957
pcdserver@qwen3-reranker-0.6b-f16pcdServerGqwen3-reranker-0.6bfine-tunedf161.2023.830/6732/6712/2818/3920/3953
pcdserver@qwen1.5-0.5b-q8pcdServerGqwen1.5-0.5bvanillaq80.6621.128/6728/677/2822/3921/39177
pcdserver@qwen1.5-0.5b-q5pcdServerGqwen1.5-0.5bvanillaq50.4621.728/6728/677/2822/3922/39238
pcdserver@qwen2.5-0.5b-q4pcdServerGqwen2.5-0.5bvanillaq40.4920.228/6726/6710/2818/3916/39605
pcdserver@qwen3-reranker-0.6b-q4pcdServerGqwen3-reranker-0.6bfine-tunedq40.4029.228/6734/6710/2818/3924/39339
pcdserver@qwen2.5-0.5b-pcb-q2pcdServerGqwen2.5-0.5b-pcbfine-tunedq20.3418.127/6723/679/2818/3914/39334
pcdserver@zerocoder-0.8b-f16pcdServerGzerocoder-0.8bfine-tunedf161.6433.027/6723/6712/2815/3911/39288
pcdserver@zerocoder-0.8b-q4pcdServerGzerocoder-0.8bfine-tunedq40.5346.427/6724/6712/2815/3912/39270
pcdserver@qwen2.5-0.5b-f16pcdServerGqwen2.5-0.5bvanillaf161.2717.826/6725/679/2817/3916/39158
pcdserver@qwen1.5-0.5b-q4pcdServerGqwen1.5-0.5bvanillaq40.4121.626/6728/678/2818/3921/39217
pcdserver@qwen3.5-0.8b-unsloth-iq2pcdServerGqwen3.5-0.8b-unslothvanillaiq20.3428.226/6723/679/2817/3914/39312
pcdserver@qwen1.5-1.8b-q2pcdServerGqwen1.5-1.8bvanillaq20.8639.826/6736/6710/2816/3926/39392
systemone@systemone-lite-0.5b †SystemOne Lite (MPS)Gsystemone-lite-0.5bdedicatedbf16/fp320.9956.426/6712/2814/39
ollaya@gliclass-largeollayaCgliclass-largededicatedfp321.7759.826/679/2817/393,794
pcdserver@qwen2.5-0.5b-q8pcdServerGqwen2.5-0.5bvanillaq80.6817.825/6724/678/2817/3916/3940
pcdserver@zerocoder-0.8b-q8pcdServerGzerocoder-0.8bfine-tunedq80.8734.425/6721/6711/2814/3910/39184
pcdserver@zerocoder-0.8b-q6pcdServerGzerocoder-0.8bfine-tunedq60.6835.324/6721/6711/2813/3910/39182
pcdserver@qwen2.5-0.5b-q6pcdServerGqwen2.5-0.5bvanillaq60.6522.223/6724/678/2815/3916/39134
pcdserver@qwen2.5-0.5b-q5pcdServerGqwen2.5-0.5bvanillaq50.5218.922/6722/677/2815/3915/39219
pcdserver@qwen1.5-0.5b-q2pcdServerGqwen1.5-0.5bvanillaq20.3021.121/6722/678/2813/3914/39247
pcdserver@qwen3-reranker-0.6b-q3pcdServerGqwen3-reranker-0.6bfine-tunedq30.3524.221/6721/678/2813/3913/3989
systemone@raz-nli-base †Raz NLI (MPS)Graz-nli-basededicatedfp320.7438.521/6710/2811/39
dohnuts-metal@this-that-model-1.2-q2dohnuts (Metal)Gthis-that-model-1.2dedicatedq20.97112.221/6718/678/2813/3910/39
pcdserver@qwen2.5-0.5b-pcb-q3pcdServerGqwen2.5-0.5b-pcbfine-tunedq30.3617.720/6720/677/2813/3913/39185
ollaya@nli-modernbert-largeollayaGnli-modernbert-largededicatedfp320.8022.720/679/2811/391,201
pcdserver@euaiact-0.5b-q4pcdServerGeuaiact-0.5bfine-tunedq40.4023.820/6720/678/2812/3912/39697
systemone@system-one-270m †Transformers slot (MPS)Gsystem-one-270mdedicatedbf16/fp320.5432.820/678/2812/39
pcdserver@qwen2.5-0.5b-pcb-f16pcdServerGqwen2.5-0.5b-pcbfine-tunedf160.9917.219/6720/677/2812/3913/39144
pcdserver@qwen2.5-0.5b-pcb-q6pcdServerGqwen2.5-0.5b-pcbfine-tunedq60.5117.519/6720/677/2812/3913/3979
pcdserver@qwen2.5-0.5b-pcb-q8pcdServerGqwen2.5-0.5b-pcbfine-tunedq80.5317.519/6720/677/2812/3913/3940
pcdserver@qwen2.5-0.5b-pcb-q5pcdServerGqwen2.5-0.5b-pcbfine-tunedq50.4217.619/6719/677/2812/3912/39259
pcdserver@qwen2.5-0.5b-q3pcdServerGqwen2.5-0.5bvanillaq30.4318.418/6719/675/2813/3914/39316
pcdserver@zerocoder-0.8b-q5pcdServerGzerocoder-0.8bfine-tunedq50.6036.218/6717/678/2810/399/39216
dohnuts-metal@decider-0.8b-q2dohnuts (Metal)Gdecider-0.8bdedicatedq20.4255.218/6717/677/2811/3910/39
slot@decider-0.8b-q2slot (llama-server)Gdecider-0.8bdedicatedq20.4264.718/6717/677/2811/3910/39108
pcdserver@qwen2.5-0.5b-pcb-q4pcdServerGqwen2.5-0.5b-pcbfine-tunedq40.4022.917/6719/676/2811/3913/39361
pcdserver@decider-0.8b-q2pcdServerGdecider-0.8bdedicatedq20.4229.317/6717/674/2813/3913/39577
systemone@oomu-systemone-0.6b †llama-cpp-python (Metal)Goomu-systemone-0.6bdedicatedq8_00.6465.217/6717/677/2810/3910/39
pcdserver@qwen2.5-0.5b-q2pcdServerGqwen2.5-0.5bvanillaq20.4217.615/6716/674/2811/3912/39477
pcdserver@qwen3-4b-unsloth-iq1pcdServerGqwen3-4b-unslothvanillaiq11.0872.513/6714/676/287/398/39594
pcdserver@lq-decide-0.6b-q8pcdServerGlq-decide-0.6bfine-tunedq80.6423.111/6711/674/287/397/39135
pcdserver@qwen3-reranker-0.6b-q2pcdServerGqwen3-reranker-0.6bfine-tunedq20.3023.910/6714/675/285/399/39241
pcdserver@lq-decide-0.6b-q4pcdServerGlq-decide-0.6bfine-tunedq40.4029.510/6711/674/286/397/39183

Some reading notes:

  • English is the hard part. The ten deliberately ambiguous queries are nine English and one Polish, so most models lose more of the 28 English queries than of the 39 others. jev got 26 of 28 English and 39 of 39 others.
  • Translation helps weak multilingual models and costs strong ones nothing. kev-0.8b scored 57 translated and 45 direct, Qwen2.5-3B at Q8 62 and 57, laya 52 and 49. The Hmm model scored one answer better without translation, 65 against 64, and so did the Qwen3.5-4B builds.
  • Load times differ by three orders of magnitude. A GGUF on pcdServer or llama-server loaded in 40 ms to 1.8 seconds. Core ML models compiled for the Neural Engine took 20 seconds, decider-4b in PyTorch 17 seconds, and the fixed-shape Core ML package of GLiNER2.5-Decide 82 seconds. The authors' System One servers were started before the timed run, so their load is not measured.
  • Quantization is flat until it falls off a cliff. Q4 to Q8 barely changes a score. Below Q3 scores drop fast: decider-0.8b at Q2 answered 17 or 18 of 67 on every engine, and Qwen3-4B at IQ1_S answered 13. Chapter 7 has the full sweeps.
  • Five methods failed. A Core ML laya export fixed at 64 tokens and four options could not take a five-way question. The BF16 build of unsloth's Qwen3.5-4B was not on disk when its turn came. The earlier pcdServer build answered every request for the two surogate Rune files with HTTP 422; the October fork separately measures mradermacher Rune through its Jinja fallback. The Q5_K_M file of jeb-35b-a3b took the machine's free memory below the benchmark's safety limit while it loaded, and the run was stopped. None of the five appears in the tables.

The October retry also measured Ornith on Splash: 62/67 translated at 322.4 ms, 63/67 direct at 328.5 ms. Only this Splash package completed. The explorer records the refused, failed and deferred runs, including ggmlc Laya Metal and the unresolved Swift loading incident.

Best method per engine and runtime

Most methods run on pcdServer, because it takes any chat GGUF: 173 of the 320. The other engines and runtimes ran only the models they were built for. The best method on each:

engine or runtime methods best method translated / direct ms/query C/G
Core AI 3 coreai@gliner25-decide 61 / – 26.9 A
Core ML 3 coreml@gliner25-decide-fp16-l256-cpu-gpu 61 / – 21.9 C+G
Core ML, Neural Engine 2 laya-coreml-ane-w8-compact 48 / 49 4.5 C+N
ExecuTorch (MLX) 2 executorch@decider-0.8b-fp16 61 / 60 89.7 G
Laya MLX (native) 1 systemone@laya-mlx-package-fp16 52 / 49 11.4 G
MLX 9 mlx@openjev-35b-a3b-4bit 63 / 62 250.0 G
ONNX Runtime 8 onnx@gliner25-decide-fp32 61 / – 67.3 C
OpenRouter API 10 openrouter@typesafe/jev-1.13 65 / 65 364.4 R
PyTorch 4 torch@decider-4b-bf16 63 / 65 348.2 G
Splash (Metal) 1 splash@ornith-1.5-35b-a3b-splash 62 / 63 322.4 G
System One server 11 systemone@neohorse-4b-q8 64 / 64 273.1 G
dohnuts (CPU) 1 dohnuts@decider-0.8b-dreamblooms-q8 62 / 60 275.2 C
dohnuts (Metal) 27 dohnuts-metal@decider-35b-a3b-q4 64 / 64 266.3 G
ggmlc (Metal) 2 systemone@kev-ggmlc-0.8b-q8 57 / 45 199.8 G
laya.cpp 5 kev-4b-gguf 62 / 55 1,103.1 G
llama-cpp-python (Metal) 1 systemone@bosun-1.7b-q5 55 / 55 130.8 G
llama.cpp + MLX head 1 gguf-head@jev-omni-q3-last-mlx 64 / 64 908.4 G
llama.cpp + MPS head 2 gguf-head@laya-neutron-ml-mps 54 / 51 50.3 G
llama.cpp embeddings 6 laya-wd-gguf-q8 52 / 49 44.4 C+G
llama.cpp embeddings + head 5 gguf-head@jev-omni-q3 64 / 64 1,222.3 C+G
ollaya 10 ollaya@winnow-12b 64 / 65 891.1 G
pcdServer 173 pcdserver@rune-mradermacher-26b-a4b-q5 65 / 65 962.3 C+G
slot (llama-server) 20 slot@rune-26b-a4b-q3 65 / 57 366.6 G
xDecision (MLX) 2 systemone@xdecision-q8-mlx 45 / 43 14.3 G

A dash is a direct score that does not apply: GLiNER2.5-Decide was trained on English only.

If you need an answer in under 10 ms on a Mac, only laya delivers it at a useful score, 52/67; Julia-1 on MLX answers in 6.8 ms but scores 31. Between 10 and 30 ms, laya in ollaya answers in 12 ms (52/67), and GLiNER2.5-Decide on Core ML CPU/GPU in 21.9 ms (61/67, translated text only). If you can spend 40 to 55 ms, pcdServer with a 2B chat model or dohnuts with decider-0.8b reaches 61 or 62. Every method above 62 took at least 85 ms; Rune Q5 reaches 65 in both modes at 962 ms with CPU weights.

The table also shows where the choice of engine is a choice of model. pcdServer ran 173 methods, including the October mradermacher Rune conversions through the new Jinja fallback. The earlier failed surogate Rune runs remain excluded. ollaya and the authors' System One servers ran the models they ship, and nothing else. The hosted jev and the five fastest local engines answer the same five-way question, but they do not compete for the same models. Chapter 3 lists which readout each engine uses, and chapter 12 turns this table into a choice.

The second round, model by model

The 50 methods added in the second round were mostly dedicated decision models that appeared on Hugging Face in the second half of September 2026. Their authors describe them against jev and against each other. Our numbers put them on one question and one machine, which is a narrower test than the authors' own, and a more even one.

rune-26b-a4b is the only local model that matched jev on translated text: 65/67 at Q3_K_M on slot, in 367 ms from 13.52 GB. On the original, untranslated text it scored 57/67, the largest gap between the two modes among the top methods. The Q4_K_M file scored 64/67 and 55/67 in 335 ms from 17.03 GB. The file we measured is version 1 of the model, from a gated revision of the repository. The later v3, which leads the Decision Index board, was not published as GGUF when we ran the benchmark, so the two are not the same weights.

winnow-12b in ollaya scored 64/67 translated and 65/67 direct, the same as the Hmm model on direct text, but at 891 ms per query from 12.67 GB. Its authors compare it to jev on a 231-item public subset of JevBench, where the Q8_0 build and jev both scored 85.71%.2 ollaya's own measurement on typed-decisions put winnow:12b at 0.702 and the smaller winnow:e4b at 0.722.3 On our set the 12B model is among the most accurate and among the slowest.

NeoHorse-Jev-4B scored 64/67 in both modes at Q8_0, in 273 ms from 5.17 GB, and 63/67 in both modes at Q3_K_M, in 296 ms from 2.95 GB. It is the smallest file in the benchmark to score 63 or more in both modes on its own server. We found no accuracy figure from its authors to set beside ours. The name also needs care: NeoHorse-1 is a general agentic and coding model, and only NeoHorse-Jev-4B is the decision model (chapter 4).

Jev-Omni scored 64/67 in both modes from its 6.09 GB Q3_K_M file, read through llama.cpp embeddings and its own head, but took 1,222 ms per query. The Q5_K_M and Q8_0 files each scored 63/67 in both modes. On MLX, the 4-bit build scored 61/67 in both modes in 780 ms. Its card reports 86.15% on 231 matched JevBench decisions.5

GLiNER2.5-Decide is the surprise among the encoders. It scored 61/67 on translated text on Core AI (27 ms), ONNX Runtime (67 ms) and Core ML (349 ms), nine answers above laya and within four of jev. Its card reports 60.2% exact match on the publisher's own 17-domain decision set.6 The two figures measure different things and cannot be compared, but they point the same way: a 340M English encoder with a label head can hold its own on a clean English question. It cannot read the original text of the 39 non-English queries, so it needs the translator in front of it.

Julia-1 runs the other way. Its card reports 73.15% on typed-decisions (1,463 of 2,000), above the 72.70% it quotes for jev.7 On our router set it scored 31/67 in both modes, on MLX in 6.8 ms and on ONNX Runtime in 12.2 ms. A model that does well on one public board can still do badly on a question it was not trained near, which is the reason this book measures on a fixed question of its own (chapter 5).

CLM, the contrastive readout, scored 33 to 39 translated across five builds, and 39/67 in ollaya at 1,678 ms. Its heads were trained on the states and actions of agents rather than on routing, and ollaya's own typed-decisions run scored it at 0.357, which ollaya's release notes call "close to chance there".3 Chapter 7 shows that no quantization rescues it.

Numbers others report

The same decision models are ranked on several public boards, and by their own publishers. None of these numbers comes from our run, and none of them can be compared with a score out of 67. They are here so that you can see where our result agrees with the outside view and where it does not.

Reported by others, not measured by us

The tables in this section copy published figures. Each names its board, its version where it has one, and the date the figure was measured or read. Numbers from different boards use different questions, different metrics and different clients, so a row in one table says nothing about a row in another.

typed-decisions

The Hugging Face dataset LocalLLaMA/typed-decisions was created on 16 September 2026. Its test split has 400 cases and 2,000 decisions from four workflows: agent-trace observability, customer service, invoice processing and security incidents. Its leaderboard, read on 30 September 2026:8

# Model Accuracy ↑ KL from gold ↓ Brier ↓ ECE ↓ p50 latency
1 meraGPT Decider 1 (sd-1), hosted 0.768 0.096 0.052 0.180 526 ms
2 Liquid AI d1 (d1:free), hosted, measured 2026-09-30 0.742 0.475 0.155 0.124 525 ms
3 TypeSafe jev 1.13.0, hosted, measured 2026-09-18 0.727 1.442 0.148 0.144 710 ms
4 Featherless Simple Jev, hosted 0.716 0.488 0.176 – –
– Prior (always the most common answer) 0.470 0.347 0.189 0.088 –

The card's own comment on jev: "Its accuracy is near the 0.735 ceiling, but it puts nearly all its probability on one answer, which is where the KL gap comes from." Chapter 9 returns to that gap. The latency column is the card's client against each hosted API, not a local measurement, and it is not the 466 ms we measured for jev from our own client.

jev's accuracy on this set is quoted three ways in September 2026: 0.727 on the card, 0.738 on ollaya's site (which cites Winnow's benchmark report for it), and 72.70% on the Julia-1 card.47 The differences come from different runs and dates. This book uses the card's 0.727 with its measurement date.

ollaya publishes its own runs on the same set, "measured by Ollaya", in its release notes between 24 and 28 September 2026:3

ollaya tag Accuracy on typed-decisions
kev:9b 0.722
winnow:e4b 0.722
winnow:12b 0.702
kev:4b 0.669
JevK5 0.625
decider:2b 0.591
nli 0.548
decider:0.8b 0.506
gliclass 0.477
kev:0.8b 0.460
laya:en 0.361
CLM 0.357

ollaya leaves out laya's own figure of 0.766, because laya "was fine-tuned on this dataset".4 The two lists agree with ours on some points and not on others. Both put winnow near the top and CLM near the bottom. On typed-decisions, decider:0.8b sits well below kev:4b; on our router set, decider-0.8b through its own readout (62/67) scored the same as kev-4b on translated text and above it on direct text. A board with four business workflows and a five-way router over FontLab tasks reward different things.

Decision 1.0

The vLLM Semantic Router project published its Decision 1.0 family on 21 and 22 September 2026, with its own benchmark of 54 tasks and 3,766 questions. Its card reports:9

Model Overall, Decision 1.0 benchmark
Decision-1.0-Eos-0.8B 61.89
Kev 0.8B 58.28
Qwen3.5-2B, letter readout 57.24
Laya English 51.03

We ran Eos through ollaya as decision:eos: 57/67 translated and 56/67 direct, in 323 ms. That is below kev-0.8b in ollaya (60/67 and 62/67) and below vanilla Qwen3.5-2B on pcdServer (61/67 and 62/67), the opposite order from the card. The board is the publisher's own, and so is the set of tasks it chose.

How it looked from the outside

On 26 September 2026 Lijuan Tang and Yuemeng Zheng posted an audit of 28 papers on typed decision models, all posted between 19 and 24 September, in the ten days after jev's launch. Their abstract concludes that "the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost".10

Our table points the same way from a different direction. The best local rows read label probabilities: letter logits on slot and dohnuts, first-token probabilities on pcdServer. A vanilla Qwen3.5-4B read that way scored 63/67 and 64/67 without any decision training.

What changed in September 2026

The first round of this benchmark had 208 methods, most of them vanilla and fine-tuned chat models on pcdServer. By the time the second round ran, at the end of September 2026, the field had changed shape. Nearly every decision model in the second round first appeared on Hugging Face between 16 and 29 September, and several public boards had appeared alongside them. Three things follow for this chapter.

  • The top of the table is more crowded, not higher. rune reached jev's 65 on translated text, and four other second-round models reached 64. None of them beat jev in both modes, and none beat Qwen3.5-4B-Hmm on direct text.
  • Hosted alternatives to jev appeared. meraGPT Decider 1 and Liquid AI d1 rank above jev on typed-decisions, and OpenAI announced a Decision API on 29 September.11 None of them is in our benchmark. Neither of the first two states its weights or licence.
  • Board numbers moved during the month. JevBench changed its scale more than once in two weeks, and jev's own typed-decisions figure is quoted three ways. Chapter 5 explains why this book reports one question on one date instead.

One set of weights, three scores

An engine is not a neutral container. The same GGUF file, read two ways, gives different answers:

Click a column header to sort.
ModelQuantEngineC/Gms/queryTranslatedDirect
decider-0.8bf16pcdServerG30.755/6756/67
decider-0.8bf16dohnuts (Metal)G51.961/6760/67
decider-0.8bf16slot (llama-server)G63.661/6760/67
decider-0.8bfp16ExecuTorch (MLX)G89.761/6760/67
decider-0.8bfp32ollayaC284.861/6760/67
decider-0.8bq2pcdServerG29.317/6717/67
decider-0.8bq2dohnuts (Metal)G55.218/6717/67
decider-0.8bq2slot (llama-server)G64.718/6717/67
decider-0.8bq3pcdServerG29.352/6747/67
decider-0.8bq3dohnuts (Metal)G54.557/6755/67
decider-0.8bq3slot (llama-server)G69.757/6755/67
decider-0.8bq4pcdServerG40.257/6757/67
decider-0.8bq4dohnuts (Metal)G53.361/6759/67
decider-0.8bq4slot (llama-server)G67.461/6759/67
decider-0.8bq5pcdServerG29.957/6757/67
decider-0.8bq5dohnuts (Metal)G55.262/6760/67
decider-0.8bq5slot (llama-server)G67.962/6760/67
decider-0.8bq6pcdServerG29.855/6757/67
decider-0.8bq6dohnuts (Metal)G53.562/6762/67
decider-0.8bq6slot (llama-server)G68.162/6762/67
decider-0.8bq8pcdServerG30.255/6757/67
decider-0.8bq8dohnuts (Metal)G51.462/6761/67
decider-0.8bq8slot (llama-server)G64.062/6761/67
decider-0.8b-dreambloomsq8pcdServerG28.355/6757/67
decider-0.8b-dreambloomsq8dohnuts (Metal)G52.262/6761/67
decider-0.8b-dreambloomsq8slot (llama-server)G65.162/6761/67
decider-0.8b-dreambloomsq8dohnuts (CPU)C275.262/6760/67
decider-2bf16pcdServerG42.260/6760/67
decider-2bf16dohnuts (Metal)G100.359/6762/67
decider-2bf16slot (llama-server)G112.459/6762/67
decider-2bfp16ExecuTorch (MLX)G251.359/6762/67
decider-2biq4pcdServerG41.457/6757/67
decider-2biq4dohnuts (Metal)G106.756/6757/67
decider-2biq4slot (llama-server)G126.856/6757/67
decider-2b-dreambloomsq8pcdServerG40.159/6760/67
decider-2b-dreambloomsq8dohnuts (Metal)G103.759/6762/67
decider-2b-dreambloomsq8slot (llama-server)G114.459/6762/67
decider-35b-a3bq4dohnuts (Metal)G266.364/6764/67
decider-35b-a3bq4pcdServerG296.863/6763/67
decider-35b-a3bq4slot (llama-server)G299.164/6764/67
decider-4bbf16PyTorchG348.263/6765/67

decider-0.8b scores 62/67 on dohnuts and 62/67 on slot. Both read the model where it was trained to answer: the logits of the option letters after Answer: (, divided by the checkpoint's temperature. The agreement is exact: on all 106 benchmark texts the two picked the same task, and their probabilities differed by at most 0.0004. They differ only in speed. dohnuts (Metal) took 52 ms per query, slot on llama-server 65 ms, and dohnuts on the CPU with eight threads 275 ms.

pcdServer scored the same file 55/67 translated and 57/67 direct, at 28 ms. pcdServer does not know the decider layout. It wraps the query in the model's chat template, lists the fields in a system prompt, and scores the first token of each allowed value. decider was never trained to answer that way, so seven answers go missing, although pcdServer is the fastest engine on this model.

The larger deciders narrow the gap. decider-2b's Q8_0 and F16 files scored 59 or 60 translated on every engine that ran them (the IQ4_NL file 56 or 57). On direct text they scored 62 through the readout and 60 through pcdServer. The 35B model scored 64/64 through its readout and 63/63 through pcdServer.

The same weights also ran in Mapika's own runtimes: ExecuTorch on MLX scored decider-0.8b at 61/60 in 90 ms, and decider-2b at 59/62 in 251 ms. ollaya runs decider-0.8b as a float32 ONNX graph on the CPU, with decider's own slot readout: 61/60 in 285 ms.

A runtime can also leave the score alone entirely. GLiNER2.5-Decide scored 61/67 on Core AI, ONNX Runtime and Core ML, and its three builds differ only in speed: 27 ms, 67 ms and 349 ms per query.

Reading a dedicated model through its own readout is worth several answers on small models. For a fine-tuned or vanilla chat model, pcdServer's chat-template readout is the one that works: the Hmm model scored 64/65 there and 62/63 through its letter readout on llama-server, and in a third of the time.

What the results do not say

A five-way router over 67 queries decides nothing about your task. A model that is one query behind another here is not measurably worse, and the ranking between models within two or three queries of each other would change with a different set of 67. The results are clear about the large gaps: dedicated or modern models against old ones, sensible quantizations against 1- and 2-bit ones, and a trained readout against a foreign one. Chapter 12 turns those gaps into recommendations. Chapter 8 explains where the milliseconds go.

The same caution applies to the public boards in Numbers others report. Each measures its own question, and a model's rank moves from one board to the next. If a board and this chapter disagree about a model, the disagreement is information about the two questions, not an error in either. The only way to know how a model will do on your decision is to run it on your own labelled examples, which is what chapter 5 describes and chapter 9 builds on.

Additional System One readouts (3 October 2026)

Eleven new configurations completed all 67 routing queries, using the frozen translations. MiniCPM5 Q8 through its native MLX runtime scored 62/67 translated and 59/67 direct, at 141.8/147.2 ms per query. The Qwen3 0.6B adapter scored 49/52, at 47.7 ms translated. The CPU ONNX Q4 scalar scorer scored 53/67, at 1410.8 ms. Full timings, weights and readout labels are in the explorer; historical rows are retained unchanged.

Gold, Distilled and ZeroShot are measured through standard Transformers NLI, with raw entailment logits and no author calibration. They scored 42, 42 and 37 of 67 translated queries. These are not measurements of the author's unavailable custom System One runtime. Raz XSmall scored 42, Raz Base 21, Lite 26, Gemma 270M 20, and OOMU 17; compatible protocol support does not guarantee good routing accuracy. English-only paths have no direct-mode score.

The Rune v3 native endpoint is implemented but not measured in this update. The earlier Rune Q3 slot score remains 65/67 translated, 57/67 direct. CoreAIKit's scalar scorer load was stopped by the memory watchdog at 29.1 GB RSS with 2 GB swap growth; no score is published. FluidInference's Gemma repository has no trained weights or Core ML artifact, and Nev Lite's custom runtime link is unavailable. The package reference records installation, readout differences and these remaining limitations.

Native Core AI extraction

On 3 October, the GLiNER2 PII Core AI conversion completed a separate 20-text synthetic entity-extraction suite on this Mac. Eight labels were supplied per call, with threshold 0.5. Four texts reproduce the upstream demo; sixteen add independent examples, three negative texts, four European-language texts and an emoji-prefixed text. One warm-up preceded the timed calls.

PII extraction metric Measured value
Exact entity sets 20/20 texts
True / false positive / false negative entities 27 / 0 / 0
Entity precision / recall / F1 1.000 / 1.000 / 1.000
Warm mean / median 12.5 / 11.9 ms/text
Native model load 3.05 s

The task counts exact (label, matched text) pairs, ignoring entity order. It is a small smoke benchmark, not a general privacy evaluation. It cannot be compared with the 67-query router accuracy or used to claim that PII redaction is complete for arbitrary documents. CPU handles tokenization and span decoding; the Core AI graph is configured for GPU. Per-operation device placement was not profiled. Both word- and token-budget rejection probes passed, and the managed native process exited after the run. The measured responses and expected entity sets include timings, native spans and redactions.

This benchmark edition closes with 329 router configurations and the separate 20-text PII extraction suite. Clef-Flash Core AI FP16 and int8mix are skipped for this edition after the memory preflight refused execution: 36 and 30 GiB available RAM required, respectively; 25 GiB available at the final check. Their native joint-schema integrations remain experimental, with no measured Core AI Clef accuracy or timing in the rankings.

Historical measurements remain available after local storage consolidation. Where complete alternatives existed, the cleanup preferred MLX and retained one quantization using saved accuracy and memory fit; without measurements, approximately four-bit precision was the storage/quality default. Distinct fine-tunes, shared components and benchmark evidence were preserved. A removed weight variant must be downloaded again before reproducing its historical run. All 329 router records and frozen translations remain unchanged. The package chapter explains how to run both integrations.

  1. TypeSafe, "Models", documentation page, read 2026-09-30. https://docs.typesafe.ai/models ↩

  2. EldanRing, "Winnow-12B" model card, created 2026-09-20, read 2026-09-30. https://huggingface.co/EldanRing/Winnow-12B ↩

  3. ollaya-dev, "ollaya" release notes v0.3.0 to v0.7.4, 2026-09-24 to 2026-09-28. https://github.com/ollaya-dev/ollaya/releases ↩↩↩

  4. ollaya, "ollaya.dev", project site, read 2026-09-30. https://ollaya.dev ↩↩

  5. akhilaaa3, "Jev-Omni" model card, created 2026-09-20, read 2026-09-30. https://huggingface.co/akhilaaa3/Jev-Omni ↩

  6. fastino, "GLiNER2.5-Decide" model card, created 2026-09-23, read 2026-09-30. https://huggingface.co/fastino/GLiNER2.5-Decide ↩

  7. SupersonicLabs, "Julia-1" model card, measured 2026-09-24, read 2026-09-30. https://huggingface.co/SupersonicLabs/Julia-1 ↩↩

  8. LocalLLaMA, "typed-decisions" dataset card and leaderboard, created 2026-09-16, read 2026-09-30. https://huggingface.co/datasets/LocalLLaMA/typed-decisions ↩

  9. vLLM Semantic Router, "Decision-1.0-Eos-0.8B" model card, created 2026-09-21, read 2026-09-30. https://huggingface.co/llm-semantic-router/Decision-1.0-Eos-0.8B ↩

  10. Lijuan Tang and Yuemeng Zheng, "Typed Decision Models: An Early Evidence Audit and Evaluation Checklist", arXiv 2609.32160, 2026-09-26. https://arxiv.org/abs/2609.32160 ↩

  11. The New Stack, "OpenAI Decision API on Luna", 2026-09-29. https://thenewstack.io/openai-decision-api-luna/ ↩

Copyright © 2026 Fontlab Ltd. Text and data CC BY 4.0, code Apache-2.0.