Skip to content

Results: every table

This page collects every table the benchmark produced, with enough context to read each one on its own. The chapters that discuss them carry the same tables beside the argument. Local measurements come from distinct saved runs on one 48 GB Mac; remote rows include network/provider latency, on one 67-query router question; chapter 5 describes the run. For the headline figures, start at Results: the short version.

Scores count correct answers out of 67. Translated means the 39 non-English queries were first translated into English; direct means the classifier got the original text. ms/query is the mean wall-clock time per query, without model loading. Chapter 5 defines every column.

Explore the interactive Pareto diagrams to find the fastest model above your own accuracy floor.

C/G labels every classifier run: C = CPU, G = GPU, C+G = mixed CPU/GPU, C+N = CPU/Neural Engine, A = automatic CPU/GPU/Neural Engine placement, R = remote. Hover over the code for execution settings and evidence.

Most tables sort when you click a column header; click again to reverse the order.

OpenRouter: nine remote models

All nine requested endpoints were measured on the same 67 queries, in translated and direct modes, on 3 October 2026. Translations and local results are retained from the earlier runs. Each endpoint received 106 distinct inputs, sequentially, with duplicate inputs reused across modes. Latency includes the network trip and provider processing; R means the provider did not disclose CPU/GPU placement or weight precision.

Six endpoints use a native five-way choice. Respan's three endpoints support noul only: each request contains five independent yes/no questions using the same task descriptions. We choose the largest P(true), then normalize the five values for the routing report. This noul OVR readout is labelled separately; its values are not calibrated exclusive-choice probabilities. Choose OpenRouter API in the explorer's engine filter to select all nine endpoints. Its table shows the readout for each run.

Full 67-query runs in each mode; network and provider processing included. R: undisclosed remote CPU/GPU.
ModelRunResolved versionC/GReadoutTranslatedDirectTranslated msDirect ms
typesafe/jev-1.133 October 2026typesafe/jev-1.13-20260917Rnative choice65/6765/67364.4375.1
typesafe/jev-1.13earlier runnot loggedRnative choice65/6765/67466.1455.2
jaredpalmer/kev-4b3 October 2026jaredpalmer/kev-4b-20260924Rnative choice65/6765/671,653.01,525.0
togethercomputer/tev1-4b-experimental3 October 2026togethercomputer/tev1-4b-experimental-20260923Rnative choice64/6765/67339.3335.1
inception/mercury-decide:free3 October 2026inception/mercury-decide-20260930Rnative choice64/6764/67408.3404.5
liquid/d13 October 2026liquid/d1-20260930Rnative choice64/6764/67566.2583.6
upstage/solar-decide3 October 2026upstage/solar-decide-20260928Rnative choice62/6763/671,253.31,292.3
respan/span-01-lite:free3 October 2026respan/span-01-lite-20260925Rnoul OVR56/6760/67436.0474.6
respan/span-01-lite3 October 2026respan/span-01-lite-20260925Rnoul OVR56/6760/67447.5462.6
respan/span-013 October 2026respan/span-01-20260925Rnoul OVR56/6760/67487.4566.3

All 320 methods

A method is one model file, read one way, on one engine or runtime. This is every method that finished, best first. Incomplete and failed attempts are left out; the latest exclusions complement the earlier failures; chapter 6 says why.

Type in the box to keep only the rows that contain every word you type: pcdServer hmm keeps the Hmm model on pcdServer, and dedicated Metal keeps the dedicated models on dohnuts (Metal).

How to read this table

  • An empty direct cell means the model was trained on English only and received the translated text alone. An empty GB or load cell means the value was not measured or does not apply, for example the size of a hosted model.
  • † after a method marks an execution or licence note. Hover over the method to read it.
  • English counts the 28 English queries in translated mode. Other, translated and other, direct count the 39 non-English queries in each mode.
All 329 methods, best first. Click to sort; an empty cell was not measured, or does not apply (the direct score of a model trained on English only).
MethodEngineC/GModelFamilyQuantGBms/queryTranslatedDirectEnglishOther, translatedOther, directLoad ms
openrouter@typesafe/jev-1.13 †OpenRouter APIRtypesafe/jev-1.13dedicated364.465/6765/6726/2839/3939/39
slot@rune-26b-a4b-q3slot (llama-server)Grune-26b-a4bdedicatedq313.52366.665/6757/6726/2839/3931/39608
jev †OpenRouter APIRtypesafe/jev-1.13dedicated466.165/6765/6726/2839/3939/39
pcdserver@rune-mradermacher-26b-a4b-q5 †pcdServerC+Grune-mradermacher-26b-a4bdedicatedq519.13962.365/6765/6726/2839/3939/394,324
openrouter@jaredpalmer/kev-4b †OpenRouter APIRjaredpalmer/kev-4bdedicated1,653.065/6765/6726/2839/3939/39
pcdserver@qwen3.5-4b-hmm-q8pcdServerGqwen3.5-4b-hmmfine-tunedq84.4887.664/6765/6726/2838/3939/39462
pcdserver@qwen3.5-4b-hmm-q4pcdServerGqwen3.5-4b-hmmfine-tunedq42.7192.664/6765/6726/2838/3939/39296
pcdserver@qwen3.5-4b-hmm-bf16pcdServerGqwen3.5-4b-hmmfine-tunedbf168.4294.864/6765/6726/2838/3939/39341
pcdserver@rune-mradermacher-26b-a4b-q3pcdServerGrune-mradermacher-26b-a4bdedicatedq313.29146.164/6764/6725/2839/3939/39354
pcdserver@rune-mradermacher-26b-a4b-q5-metal †pcdServerGrune-mradermacher-26b-a4bdedicatedq519.13153.364/6764/6725/2839/3939/39507
dohnuts-metal@decider-35b-a3b-q4dohnuts (Metal)Gdecider-35b-a3bdedicatedq421.17266.364/6764/6726/2838/3938/39
systemone@neohorse-4b-q8System One serverGneohorse-4bdedicatedq85.17273.164/6764/6725/2839/3939/39
slot@decider-35b-a3b-q4slot (llama-server)Gdecider-35b-a3bdedicatedq421.17299.164/6764/6726/2838/3938/39375
slot@rune-26b-a4b-q4slot (llama-server)Grune-26b-a4bdedicatedq417.03335.364/6755/6726/2838/3929/39439
dohnuts-metal@jpt-4b-q8 †dohnuts (Metal)Gjpt-4bdedicatedq84.48335.664/6764/6725/2839/3939/39
openrouter@togethercomputer/tev1-4b-experimental †OpenRouter APIRtogethercomputer/tev1-4b-experimentaldedicated339.364/6765/6726/2838/3939/39
openrouter@inception/mercury-decide:free †OpenRouter APIRinception/mercury-decide:freededicated408.364/6764/6725/2839/3939/39
openrouter@liquid/d1 †OpenRouter APIRliquid/d1dedicated566.264/6764/6725/2839/3939/39
pcdserver@rune-mradermacher-26b-a4b-q8 †pcdServerCrune-mradermacher-26b-a4bdedicatedq826.86758.164/6764/6725/2839/3939/399,856
ollaya@winnow-12bollayaGwinnow-12bdedicatedq8_012.67891.164/6765/6726/2838/3939/3936,488
gguf-head@jev-omni-q3-last-mlx †llama.cpp + MLX headGjev-omnidedicatedq36.09908.464/6764/6726/2838/3938/391,193
gguf-head@jev-omni-q3llama.cpp embeddings + headC+Gjev-omnidedicatedq36.091,222.364/6764/6726/2838/3938/391,411
pcdserver@qwen3.5-4b-unsloth-q8pcdServerGqwen3.5-4b-unslothvanillaq84.4885.063/6764/6725/2838/3939/39346
pcdserver@jpt-4b-q8 †pcdServerGjpt-4bdedicatedq84.4886.363/6764/6725/2838/3939/39284
pcdserver@jpt-4b-q4 †pcdServerGjpt-4bdedicatedq42.7193.063/6764/6725/2838/3939/39318
pcdserver@qwen3.5-4b-unsloth-q5pcdServerGqwen3.5-4b-unslothvanillaq53.1494.563/6764/6725/2838/3939/39354
pcdserver@rune-mradermacher-26b-a4b-q4-metal †pcdServerGrune-mradermacher-26b-a4bdedicatedq416.80124.863/6764/6725/2838/3939/39332
pcdserver@jpt-9b-q5 †pcdServerGjpt-9bdedicatedq56.47155.263/6764/6725/2838/3939/39495
mlx@openjev-35b-a3b-4bitMLXGopenjev-35b-a3bdedicated4bit20.99250.063/6762/6724/2839/3938/3927,426
systemone@neohorse-4b-q3System One serverGneohorse-4bdedicatedq32.95296.163/6763/6724/2839/3939/39
pcdserver@openjev-35b-a3b-q4pcdServerGopenjev-35b-a3bdedicatedq421.17296.863/6764/6725/2838/3939/39338
pcdserver@decider-35b-a3b-q4pcdServerGdecider-35b-a3bdedicatedq421.17296.863/6763/6724/2839/3939/39799
slot@openjev-35b-a3b-q4slot (llama-server)Gopenjev-35b-a3bdedicatedq421.17311.363/6762/6724/2839/3938/39388
systemone@jeb-35b-a3b-q3System One serverGjeb-35b-a3bdedicatedq317.17333.263/6763/6724/2839/3939/39
torch@decider-4b-bf16PyTorchGdecider-4bdedicatedbf16348.263/6765/6726/2837/3939/3916,802
dohnuts-metal@jpt-4b-q5 †dohnuts (Metal)Gjpt-4bdedicatedq53.07367.063/6764/6725/2838/3939/39
pcdserver@rune-mradermacher-26b-a4b-q4 †pcdServerC+Grune-mradermacher-26b-a4bdedicatedq416.80487.163/6764/6725/2838/3939/393,170
systemone@clef-flash-mlx4 †Clef joint head (MLX)Gclef-flashdedicatedq45.28554.863/6763/6724/2839/3939/39
systemone@clef-flash-q4 †llama-server (System One)Gclef-flashdedicatedq4_k_m6.33610.763/6764/6725/2838/3939/39
systemone@clef-flash-q6 †llama-server (System One)Gclef-flashdedicatedq6_k8.28610.863/6764/6725/2838/3939/39
dohnuts-metal@jpt-9b-q5 †dohnuts (Metal)Gjpt-9bdedicatedq56.47619.663/6764/6725/2838/3939/39
dohnuts-metal@jpt-9b-q4 †dohnuts (Metal)Gjpt-9bdedicatedq45.63619.863/6764/6725/2838/3939/39
dohnuts-metal@jpt-9b-q8 †dohnuts (Metal)Gjpt-9bdedicatedq89.53624.763/6764/6725/2838/3939/39
systemone@clef-flash-q5 †llama-server (System One)Gclef-flashdedicatedq5_k_m7.36644.863/6764/6725/2838/3939/39
gguf-head@jev-omni-q8llama.cpp embeddings + headC+Gjev-omnidedicatedq812.671,124.163/6763/6725/2838/3938/391,157
gguf-head@jev-omni-q5llama.cpp embeddings + headC+Gjev-omnidedicatedq58.551,212.763/6763/6725/2838/3938/391,243
systemone@lev-4bSystem One serverGlev-4bdedicated9.492,203.863/6764/6725/2838/3939/39
dohnuts-metal@decider-0.8b-q8dohnuts (Metal)Gdecider-0.8bdedicatedq80.8151.462/6761/6724/2838/3937/39
dohnuts-metal@decider-0.8b-dreamblooms-q8dohnuts (Metal)Gdecider-0.8b-dreambloomsdedicatedq80.8152.262/6761/6724/2838/3937/39
pcdserver@qwen2.5-3b-q8pcdServerGqwen2.5-3bvanillaq83.2953.362/6757/6723/2839/3935/39178
pcdserver@qwen2.5-3b-f16pcdServerGqwen2.5-3bvanillaf166.1853.562/6760/6723/2839/3937/39252
dohnuts-metal@decider-0.8b-q6dohnuts (Metal)Gdecider-0.8bdedicatedq60.6353.562/6762/6724/2838/3938/39
dohnuts-metal@decider-0.8b-q5dohnuts (Metal)Gdecider-0.8bdedicatedq50.5855.262/6760/6724/2838/3936/39
pcdserver@qwen2.5-3b-q4pcdServerGqwen2.5-3bvanillaq41.9359.262/6760/6723/2839/3937/39339
pcdserver@qwen2.5-3b-q6pcdServerGqwen2.5-3bvanillaq62.5462.062/6760/6723/2839/3937/39213
slot@decider-0.8b-q8slot (llama-server)Gdecider-0.8bdedicatedq80.8164.062/6761/6724/2838/3937/39428
slot@decider-0.8b-dreamblooms-q8slot (llama-server)Gdecider-0.8b-dreambloomsdedicatedq80.8165.162/6761/6724/2838/3937/39689
slot@decider-0.8b-q5slot (llama-server)Gdecider-0.8bdedicatedq50.5867.962/6760/6724/2838/3936/39142
slot@decider-0.8b-q6slot (llama-server)Gdecider-0.8bdedicatedq60.6368.162/6762/6724/2838/3938/39216
pcdserver@qwen3.5-4b-q4pcdServerGqwen3.5-4bvanillaq43.0189.862/6763/6724/2838/3939/39311
pcdserver@qwen3.5-4b-unsloth-q6pcdServerGqwen3.5-4b-unslothvanillaq63.5393.662/6763/6724/2838/3939/39291
pcdserver@qwen3.8-4b-distill-q4pcdServerGqwen3.8-4b-distillfine-tunedq42.7893.662/6763/6724/2838/3939/39284
pcdserver@jpt-4b-q5 †pcdServerGjpt-4bdedicatedq53.0799.462/6763/6724/2838/3939/39345
pcdserver@qwen3.8-9b-distill-q4pcdServerGqwen3.8-9b-distillfine-tunedq45.78137.862/6764/6725/2837/3939/39449
systemone@system-one-minicpm5-2b-q8 †System One (MLX)Gsystem-one-minicpm5-2b-q8dedicatedq82.67141.862/6759/6724/2838/3935/39
dohnuts@decider-0.8b-dreamblooms-q8dohnuts (CPU)Cdecider-0.8b-dreambloomsdedicatedq80.81275.262/6760/6724/2838/3936/39
dohnuts-metal@jet-4b-q8dohnuts (Metal)Gjet-4bdedicatedq84.48281.462/6763/6724/2838/3939/39
slot@qwen3.5-4b-hmm-bf16slot (llama-server)Gqwen3.5-4b-hmmfine-tunedbf168.42284.262/6763/6724/2838/3939/39334
slot@qwen3.5-4b-hmm-q8slot (llama-server)Gqwen3.5-4b-hmmfine-tunedq84.48287.362/6763/6724/2838/3939/39285
dohnuts-metal@jpt-4b-q4 †dohnuts (Metal)Gjpt-4bdedicatedq42.71290.662/6762/6724/2838/3938/39
slot@qwen3.5-4b-hmm-q4slot (llama-server)Gqwen3.5-4b-hmmfine-tunedq42.71303.162/6763/6725/2837/3938/39293
splash@ornith-1.5-35b-a3b-splash †Splash (Metal)Gornith-1.5-35b-a3b-splashfine-tunedsplash-packed-q4-moe20.95322.462/6763/6724/2838/3939/39
slot@jevk5-4b-q4slot (llama-server)Gjevk5-4bdedicatedq42.71350.562/6763/6724/2838/3939/39429
systemone@imajev-4bSystem One serverGimajev-4bdedicated9.81363.162/6764/6725/2837/3939/39
pcdserver@bonsai-27b-q1pcdServerGbonsai-27bvanillaq13.80513.562/6763/6724/2838/3939/391,827
systemone@clef-flash-q3 †llama-server (System One)Gclef-flashdedicatedq3_k_m4.97668.362/6764/6725/2837/3939/39
kev-4b-gguflaya.cppGkev-4bdedicated1,103.162/6755/6724/2838/3931/39
openrouter@upstage/solar-decide †OpenRouter APIRupstage/solar-decidededicated1,253.362/6763/6724/2838/3939/39
coreml@gliner25-decide-fp16-l256-cpu-gpu †Core MLC+Ggliner25-decidededicatedfp160.9321.961/6722/2839/3913,151
coreai@gliner25-decideCore AIAgliner25-decidededicatedfp160.8726.961/6722/2839/393,378
pcdserver@qwen3.5-2b-q3pcdServerGqwen3.5-2bvanillaq31.2242.761/6762/6723/2838/3939/39203
dohnuts-metal@decider-0.8b-f16dohnuts (Metal)Gdecider-0.8bdedicatedf161.5251.961/6760/6723/2838/3937/39
dohnuts-metal@decider-0.8b-q4dohnuts (Metal)Gdecider-0.8bdedicatedq40.5353.361/6759/6723/2838/3936/39
pcdserver@qwen2.5-3b-q5pcdServerGqwen2.5-3bvanillaq52.2261.761/6757/6722/2839/3936/39300
slot@decider-0.8b-f16slot (llama-server)Gdecider-0.8bdedicatedf161.5263.661/6760/6723/2838/3937/39670
onnx@gliner25-decide-fp32ONNX RuntimeCgliner25-decidededicatedfp321.7467.361/6722/2839/394,607
slot@decider-0.8b-q4slot (llama-server)Gdecider-0.8bdedicatedq40.5367.461/6759/6723/2838/3936/39231
pcdserver@qwen3-4b-unsloth-bf16pcdServerGqwen3-4b-unslothvanillabf168.0571.561/6762/6723/2838/3939/39259
pcdserver@qwen3-4b-q4pcdServerGqwen3-4bvanillaq42.5075.561/6762/6723/2838/3939/39230
pcdserver@qwen3-4b-unsloth-q4pcdServerGqwen3-4b-unslothvanillaq42.5076.461/6761/6723/2838/3938/39247
pcdserver@qwen3.5-4b-supercoder-q4pcdServerGqwen3.5-4b-supercoderfine-tunedq42.6184.261/6762/6723/2838/3939/39351
pcdserver@jevk5-4b-q8pcdServerGjevk5-4bdedicatedq84.4884.261/6762/6723/2838/3939/39292
executorch@decider-0.8b-fp16ExecuTorch (MLX)Gdecider-0.8bdedicatedfp1689.761/6760/6723/2838/3937/39
pcdserver@jevk5-4b-q4pcdServerGjevk5-4bdedicatedq42.7189.961/6762/6723/2838/3939/39332
pcdserver@qwen3.5-4b-unsloth-iq3pcdServerGqwen3.5-4b-unslothvanillaiq31.9590.261/6762/6723/2838/3939/39800
pcdserver@qwen3.5-4b-unsloth-q2pcdServerGqwen3.5-4b-unslothvanillaq21.9493.561/6761/6722/2839/3939/39829
pcdserver@qwen3.5-4b-unsloth-q4pcdServerGqwen3.5-4b-unslothvanillaq42.7493.661/6762/6723/2838/3939/39297
pcdserver@qwen2.5-coder-7b-q4pcdServerGqwen2.5-coder-7bfine-tunedq44.68108.961/6762/6723/2838/3939/39386
pcdserver@qwen3-8b-q4pcdServerGqwen3-8bvanillaq45.03116.061/6762/6723/2838/3939/39367
pcdserver@qwen3.5-9b-unsloth-q4pcdServerGqwen3.5-9b-unslothvanillaq45.68136.961/6763/6724/2837/3939/39452
pcdserver@jpt-9b-q4 †pcdServerGjpt-9bdedicatedq45.63137.361/6763/6724/2837/3939/39478
pcdserver@jpt-9b-q8 †pcdServerGjpt-9bdedicatedq89.53141.861/6763/6724/2837/3939/39451
ollaya@decider-0.8bollayaCdecider-0.8bdedicatedfp321.53284.861/6760/6723/2838/3937/397,924
dohnuts-metal@jpt-4b-q3 †dohnuts (Metal)Gjpt-4bdedicatedq32.26315.061/6763/6724/2837/3939/39
slot@jevk5-4b-q8slot (llama-server)Gjevk5-4bdedicatedq84.48333.961/6763/6724/2837/3939/39372
coreml@gliner25-decide-fp16-l256Core MLAgliner25-decidededicatedfp160.93349.161/6722/2839/3982,062
dohnuts-metal@jpt-9b-q3 †dohnuts (Metal)Gjpt-9bdedicatedq34.62636.761/6764/6726/2835/3938/39
mlx@jev-omni-4bitMLXGjev-omnidedicated4bit6.81780.161/6761/6724/2837/3937/39
pcdserver@decider-2b-f16pcdServerGdecider-2bdedicatedf163.7842.260/6760/6722/2838/3938/39166
pcdserver@qwen2.5-coder-3b-q4pcdServerGqwen2.5-coder-3bfine-tunedq42.1060.260/6759/6724/2836/3935/39295
mlx@tev1-0.8b-4bit †MLXGtev1-0.8bdedicated4bit0.4264.860/6759/6721/2839/3938/393,482
pcdserver@qwen3-4b-unsloth-q8pcdServerGqwen3-4b-unslothvanillaq84.2870.360/6762/6723/2837/3939/39306
pcdserver@qwen3-4b-unsloth-q3pcdServerGqwen3-4b-unslothvanillaq32.0878.360/6761/6722/2838/3939/39706
pcdserver@qwen3-4b-unsloth-q6pcdServerGqwen3-4b-unslothvanillaq63.3179.360/6761/6722/2838/3939/39276
pcdserver@qwen3-4b-unsloth-q5pcdServerGqwen3-4b-unslothvanillaq52.8979.460/6762/6723/2837/3939/39321
pcdserver@qwen3.5-4b-unsloth-q3pcdServerGqwen3.5-4b-unslothvanillaq32.2992.760/6761/6722/2838/3939/39358
pcdserver@jpt-4b-q3 †pcdServerGjpt-4bdedicatedq32.2693.760/6760/6722/2838/3938/39374
coreml@kev-0.8b-fp16-l512 †Core MLC+Gkev-0.8bdedicatedfp161.51121.360/6762/6724/2836/3938/3911,368
pcdserver@jpt-9b-q3 †pcdServerGjpt-9bdedicatedq34.62139.260/6762/6723/2837/3939/39482
ollaya@kev-0.8bollayaCkev-0.8bdedicatedfp321.82268.260/6762/6724/2836/3938/394,848
pcdserver@decider-2b-dreamblooms-q8pcdServerGdecider-2b-dreambloomsdedicatedq82.0140.159/6760/6722/2837/3938/39198
dohnuts-metal@tev1-0.8b-q8 †dohnuts (Metal)Gtev1-0.8bdedicatedq80.8161.559/6762/6723/2836/3939/39
pcdserver@qwen3-4b-unsloth-iq2pcdServerGqwen3-4b-unslothvanillaiq21.5372.759/6757/6721/2838/3936/39605
dohnuts-metal@decider-2b-f16dohnuts (Metal)Gdecider-2bdedicatedf163.78100.359/6762/6724/2835/3938/39
dohnuts-metal@decider-2b-dreamblooms-q8dohnuts (Metal)Gdecider-2b-dreambloomsdedicatedq82.01103.759/6762/6724/2835/3938/39
slot@decider-2b-f16slot (llama-server)Gdecider-2bdedicatedf163.78112.459/6762/6724/2835/3938/39153
slot@decider-2b-dreamblooms-q8slot (llama-server)Gdecider-2b-dreambloomsdedicatedq82.01114.459/6762/6724/2835/3938/39199
executorch@decider-2b-fp16ExecuTorch (MLX)Gdecider-2bdedicatedfp16251.359/6762/6724/2835/3938/39
pcdserver@decider-0.8b-q5pcdServerGdecider-0.8bdedicatedq50.5829.957/6757/6722/2836/3935/39180
pcdserver@qwen3.5-2b-q8pcdServerGqwen3.5-2bvanillaq82.0839.557/6759/6720/2838/3939/39189
pcdserver@qwen3.5-2b-unsloth-q8pcdServerGqwen3.5-2b-unslothvanillaq82.0139.557/6759/6720/2838/3939/39126
pcdserver@decider-0.8b-q4pcdServerGdecider-0.8bdedicatedq40.5340.257/6757/6722/2836/3935/39583
pcdserver@decider-2b-iq4pcdServerGdecider-2bdedicatediq41.2441.457/6757/6721/2837/3937/39167
pcdserver@qwen3.5-2b-unsloth-bf16pcdServerGqwen3.5-2b-unslothvanillabf163.7842.257/6759/6720/2838/3939/39328
pcdserver@qwen3.5-2b-unsloth-q4pcdServerGqwen3.5-2b-unslothvanillaq41.2844.857/6757/6719/2839/3938/39139
dohnuts-metal@decider-0.8b-q3dohnuts (Metal)Gdecider-0.8bdedicatedq30.4754.557/6755/6724/2834/3931/39
pcdserver@qwen2.5-3b-q3pcdServerGqwen2.5-3bvanillaq31.5960.657/6751/6720/2838/3931/39672
ollaya@von-1.1ollayaCvon-1.1dedicatedfp321.5962.857/6722/2836/3914,254
slot@decider-0.8b-q3slot (llama-server)Gdecider-0.8bdedicatedq30.4769.757/6755/6724/2834/3931/39101
dohnuts-metal@this-that-model-1.2-q3dohnuts (Metal)Gthis-that-model-1.2dedicatedq31.10113.157/6756/6723/2835/3933/39
systemone@leo-1.7bSystem One serverGleo-1.7bdedicated3.52136.257/6762/6723/2835/3939/39
systemone@kev-ggmlc-0.8b-q8 †ggmlc (Metal)Gkev-0.8bdedicatedq80.83199.857/6745/6724/2834/3921/39
kev-0.8b-gguflaya.cppGkev-0.8bdedicated231.557/6745/6724/2834/3921/39
pcdserver@qwen3.5-0.8b-q5pcdServerGqwen3.5-0.8bvanillaq50.6532.157/6757/6721/2836/3937/39284
pcdserver@qwen3-1.7b-unsloth-q8pcdServerGqwen3-1.7b-unslothvanillaq81.8335.157/6759/6721/2836/3938/3995
pcdserver@qwen3-1.7b-q8pcdServerGqwen3-1.7bvanillaq82.1735.257/6759/6721/2836/3938/3994
pcdserver@qwen3-1.7b-bf16pcdServerGqwen3-1.7bvanillabf164.0736.257/6759/6721/2836/3938/39206
pcdserver@qwen3-1.7b-q5pcdServerGqwen3-1.7bvanillaq51.4738.757/6759/6721/2836/3938/39148
pcdserver@qwen3-1.7b-unsloth-q5pcdServerGqwen3-1.7b-unslothvanillaq51.2638.757/6759/6721/2836/3938/39170
pcdserver@qwen3-1.7b-unsloth-q6pcdServerGqwen3-1.7b-unslothvanillaq61.4239.257/6759/6721/2836/3938/39158
pcdserver@qwen3-1.7b-q6pcdServerGqwen3-1.7bvanillaq61.6739.357/6759/6721/2836/3938/39150
pcdserver@qwen3.5-2b-unsloth-iq3pcdServerGqwen3.5-2b-unslothvanillaiq30.9340.257/6757/6721/2836/3937/39435
pcdserver@qwen3.5-2b-unsloth-q3pcdServerGqwen3.5-2b-unslothvanillaq31.1141.957/6759/6720/2837/3939/39165
pcdserver@qwen3.5-2b-q5pcdServerGqwen3.5-2bvanillaq51.5742.357/6757/6719/2838/3939/39196
pcdserver@qwen3.5-2b-q6pcdServerGqwen3.5-2bvanillaq61.7042.657/6757/6719/2838/3939/39134
pcdserver@qwen3.5-2b-unsloth-q5pcdServerGqwen3.5-2b-unslothvanillaq51.4443.557/6757/6719/2838/3938/39142
pcdserver@qwen3.5-2b-q4pcdServerGqwen3.5-2bvanillaq41.4047.257/6757/6719/2838/3939/39217
onnx@von-1.2ONNX RuntimeCvon-1.2dedicatedfp321.5852.657/6722/2835/392,485
ollaya@decision-eosollayaCdecision-eosdedicatedfp321.54322.957/6756/6723/2834/3933/395,638
pcdserver@qwen3.5-0.8b-unsloth-q5pcdServerGqwen3.5-0.8b-unslothvanillaq50.5929.656/6757/6721/2835/3936/39184
pcdserver@qwen3-1.7b-unsloth-q4pcdServerGqwen3-1.7b-unslothvanillaq41.1140.256/6757/6720/2836/3938/39194
pcdserver@qwen3.5-2b-unsloth-q6pcdServerGqwen3.5-2b-unslothvanillaq61.5743.856/6757/6719/2837/3939/39132
dohnuts-metal@decider-2b-iq4dohnuts (Metal)Gdecider-2bdedicatediq41.24106.756/6757/6723/2833/3935/39
slot@decider-2b-iq4slot (llama-server)Gdecider-2bdedicatediq41.24126.856/6757/6723/2833/3935/39719
openrouter@respan/span-01-lite:free †OpenRouter APIRrespan/span-01-lite:freededicated436.056/6760/6721/2835/3939/39
openrouter@respan/span-01-lite †OpenRouter APIRrespan/span-01-litededicated447.556/6760/6721/2835/3939/39
openrouter@respan/span-01 †OpenRouter APIRrespan/span-01dedicated487.456/6760/6721/2835/3939/39
pcdserver@decider-0.8b-dreamblooms-q8pcdServerGdecider-0.8b-dreambloomsdedicatedq80.8128.355/6757/6720/2835/3937/3978
pcdserver@qwen3.5-0.8b-unsloth-q8pcdServerGqwen3.5-0.8b-unslothvanillaq80.8128.755/6757/6720/2835/3937/39177
pcdserver@qwen3.5-0.8b-q6pcdServerGqwen3.5-0.8bvanillaq60.6929.055/6757/6720/2835/3937/39450
pcdserver@qwen3.5-0.8b-unsloth-q6pcdServerGqwen3.5-0.8b-unslothvanillaq60.6429.255/6757/6720/2835/3937/39186
pcdserver@decider-0.8b-q6pcdServerGdecider-0.8bdedicatedq60.6329.855/6757/6720/2835/3937/39280
pcdserver@qwen3.5-0.8b-bf16pcdServerGqwen3.5-0.8bvanillabf161.5630.055/6757/6720/2835/3937/39407
pcdserver@decider-0.8b-q8pcdServerGdecider-0.8bdedicatedq80.8130.255/6757/6720/2835/3937/39208
pcdserver@qwen3.5-0.8b-ggml-bf16pcdServerGqwen3.5-0.8b-ggmlvanillabf161.5630.355/6757/6720/2835/3937/39337
pcdserver@decider-0.8b-f16pcdServerGdecider-0.8bdedicatedf161.5230.755/6756/6719/2836/3937/39438
pcdserver@qwen3.5-0.8b-unsloth-bf16pcdServerGqwen3.5-0.8b-unslothvanillabf161.5231.055/6757/6720/2835/3937/39208
pcdserver@qwen3.5-0.8b-q8pcdServerGqwen3.5-0.8bvanillaq80.8331.255/6757/6720/2835/3937/39252
pcdserver@qwen3.5-0.8b-unsloth-q4pcdServerGqwen3.5-0.8b-unslothvanillaq40.5350.155/6754/6720/2835/3934/39255
systemone@bosun-1.7b-q5 †llama-cpp-python (Metal)Gbosun-1.7bdedicatedq51.26130.855/6755/6722/2833/3933/39
pcdserver@pulse-0.6b-q4pcdServerGpulse-0.6bfine-tunedq40.4027.954/6752/6721/2833/3931/39196
pcdserver@bonsai-1.7b-q2pcdServerGbonsai-1.7bvanillaq20.4934.954/6755/6719/2835/3936/39259
pcdserver@qwen3-1.7b-q2pcdServerGqwen3-1.7bvanillaq20.8837.554/6750/6721/2833/3929/39104
pcdserver@qwen3-1.7b-q4pcdServerGqwen3-1.7bvanillaq41.2840.754/6757/6720/2834/3938/39343
gguf-head@laya-neutron-mlllama.cpp embeddings + headC+Glaya-multilingualdedicatedq4_00.2742.754/6751/6721/2833/3930/392,739
gguf-head@laya-neutron-ml-mps †llama.cpp + MPS headGlaya-multilingualdedicatedq4_00.2750.354/6751/6721/2833/3930/394,915
dohnuts-metal@this-that-model-1.2-q5dohnuts (Metal)Gthis-that-model-1.2dedicatedq51.41117.454/6757/6721/2833/3937/39
systemone@kev-ggmlc-0.8b-q4 †ggmlc (Metal)Gkev-0.8bdedicatedq40.96207.254/6743/6723/2831/3920/39
pcdserver@qwen3-0.6b-q8pcdServerGqwen3-0.6bvanillaq80.6425.553/6754/6718/2835/3936/39191
pcdserver@qwen3.5-0.8b-ggml-q4pcdServerGqwen3.5-0.8b-ggmlvanillaq40.5628.853/6755/6719/2834/3936/39801
pcdserver@qwen3-1.7b-unsloth-q3pcdServerGqwen3-1.7b-unslothvanillaq30.9437.653/6756/6718/2835/3938/39216
pcdserver@qwen3.5-2b-unsloth-q2pcdServerGqwen3.5-2b-unslothvanillaq20.9741.353/6748/6719/2834/3929/39255
mlx@laya-ml-mxfp8 †MLXGlaya-multilingualdedicatedmxfp80.3257.853/6749/6720/2833/3929/392,639
pcdserver@qwen2.5-3b-q2pcdServerGqwen2.5-3bvanillaq21.2758.353/6746/6717/2836/3929/39271
dohnuts-metal@this-that-model-1.2-q8dohnuts (Metal)Gthis-that-model-1.2dedicatedq82.01107.853/6757/6720/2833/3938/39
systemone@system-one-scorer-onnx †ONNX scalar (CPU)Csystem-one-scorer-onnxdedicatedq42.641,410.853/6723/2830/39
laya-mlxMLXGlaya-multilingualdedicated6.952/6749/6721/2831/3928/391,797
laya-coreaiCore AIAlaya-multilingualdedicated10.052/6749/6721/2831/3928/393,164
systemone@laya-mlx-package-fp16 †Laya MLX (native)Glaya-multilingualdedicatedfp160.6411.452/6749/6721/2831/3928/39
ollaya@laya-multilingualollayaGlaya-multilingualdedicatedfp320.6812.352/6749/6721/2831/3928/391,217
laya-coreai-mlbCore AIAlaya-multilingualdedicated19.452/6749/6721/2831/3928/392,404
laya-ml-onnx-fp32ONNX RuntimeClaya-multilingualdedicatedfp3223.452/6749/6721/2831/3928/39825
onnx@laya-ml-fp32ONNX RuntimeClaya-multilingualdedicatedfp321.2924.752/6749/6721/2831/3928/391,578
laya-ml-onnx-int8ONNX RuntimeClaya-multilingualdedicatedint825.552/6749/6721/2831/3928/39741
pcdserver@decider-0.8b-q3pcdServerGdecider-0.8bdedicatedq30.4729.352/6747/6718/2834/3929/39161
pcdserver@qwen3.5-0.8b-q4pcdServerGqwen3.5-0.8bvanillaq40.5830.252/6751/6717/2835/3934/39258
laya-wd-gguf-q8llama.cpp embeddingsC+Glaya-multilingualdedicatedq844.452/6749/6721/2831/3928/392,337
laya-wd-gguf-f16llama.cpp embeddingsC+Glaya-multilingualdedicatedf1644.452/6749/6721/2831/3928/392,437
laya-wd-gguf-q8-mps †llama.cpp + MPS headGlaya-multilingualdedicatedq80.3745.952/6749/6721/2831/3928/3986,773
laya-ml-gguflaya.cppClaya-multilingualdedicated57.652/6749/6721/2831/3928/39
onnx@laya-ml-fp16ONNX RuntimeClaya-multilingualdedicatedfp160.6557.652/6749/6721/2831/3928/391,067
laya-ml-gguf-f16laya.cppClaya-multilingualdedicatedf1661.452/6749/6721/2831/3928/39
pcdserver@qwen3.5-4b-unsloth-iq2pcdServerGqwen3.5-4b-unslothvanillaiq21.5287.052/6754/6716/2836/3938/39659
laya-wd-gguf-q6llama.cpp embeddingsC+Glaya-multilingualdedicatedq644.551/6747/6720/2831/3927/392,276
laya-ml-gguf-q4laya.cppClaya-multilingualdedicatedq455.051/6747/6718/2833/3929/39
dohnuts-metal@this-that-model-1.2-q4dohnuts (Metal)Gthis-that-model-1.2dedicatedq41.27110.551/6751/6719/2832/3932/39
mlx@semif-4b-nliMLXGsemif-4b-nlidedicated8.41120.651/6749/6719/2832/3930/3913,083
pcdserver@pulse-0.6b-q8pcdServerGpulse-0.6bfine-tunedq80.6423.250/6747/6720/2830/3927/3956
pcdserver@lq-decide-1.7b-q8pcdServerGlq-decide-1.7bfine-tunedq81.8335.150/6746/6720/2830/3926/39116
pcdserver@qwen3-1.7b-q3pcdServerGqwen3-1.7bvanillaq31.0737.750/6753/6716/2834/3937/39109
pcdserver@qwen3.5-2b-q2pcdServerGqwen3.5-2bvanillaq21.0741.549/6749/6719/2830/3930/39127
pcdserver@qwen3.8-2b-distill-q4pcdServerGqwen3.8-2b-distillfine-tunedq41.3142.949/6751/6719/2830/3932/39154
laya-wd-gguf-q3llama.cpp embeddingsC+Glaya-multilingualdedicatedq345.349/6743/6717/2832/3926/393,284
systemone@system-one-qwen3-0.6b †System One (MLX)Gsystem-one-qwen3-0.6bdedicatedbf16+adapter1.5247.749/6752/6721/2828/3931/39
pcdserver@qwen2.5-3b-iq2pcdServerGqwen2.5-3bvanillaiq21.1455.549/6749/6716/2833/3933/39427
laya-coreml-ane-w8-compactCore ML, Neural EngineC+Nlaya-multilingualdedicatedw84.548/6749/6717/2831/3932/3921,052
laya-typed-decisionsPyTorchGlaya-typed-decisionsdedicated17.748/6743/6718/2830/3925/394,793
pcdserver@qwen3.5-0.8b-unsloth-q3pcdServerGqwen3.5-0.8b-unslothvanillaq30.4730.148/6748/6717/2831/3931/39186
laya-coreml-ane-compactCore ML, Neural EngineC+Nlaya-multilingualdedicated4.247/6749/6717/2830/3932/3920,130
laya-mlx-compactMLXGlaya-multilingualdedicated7.947/6749/6717/2830/3932/3911,010
pcdserver@bonsai-1.7b-q1pcdServerGbonsai-1.7bvanillaq10.2534.947/6741/6714/2833/3927/39243
laya-wd-gguf-q5llama.cpp embeddingsC+Glaya-multilingualdedicatedq545.547/6743/6717/2830/3926/392,350
pcdserver@qwen3.5-0.8b-q3pcdServerGqwen3.5-0.8bvanillaq30.5231.346/6740/6715/2831/3925/39218
gguf-head@laya-neutron-enllama.cpp embeddings + headC+Glaya-endedicatedq4_00.2962.446/6716/2830/391,664
systemone@xdecision-q8-mlx †xDecision (MLX)GxDecisiondedicatedq80.4014.345/6743/6717/2828/3926/39
pcdserver@qwen3.8-2b-distill-q5pcdServerGqwen3.8-2b-distillfine-tunedq51.4546.145/6745/6717/2828/3928/39140
systemone@xdecision-f16-mlx †xDecision (MLX)GxDecisiondedicatedf160.7014.944/6743/6716/2828/3927/39
pcdserver@qwen3-1.7b-unsloth-q2pcdServerGqwen3-1.7b-unslothvanillaq20.7837.444/6746/6718/2826/3928/39276
pcdserver@qwen3.8-2b-distill-q8pcdServerGqwen3.8-2b-distillfine-tunedq82.0839.544/6745/6717/2827/3928/39129
pcdserver@qwen3.5-2b-unsloth-iq2pcdServerGqwen3.5-2b-unslothvanillaiq20.7740.844/6739/6715/2829/3924/39129
mlx@laya-ml-mxfp4 †MLXGlaya-multilingualdedicatedmxfp40.1657.944/6744/6716/2828/3928/392,438
pcdserver@qwen3.5-0.8b-unsloth-q2pcdServerGqwen3.5-0.8b-unslothvanillaq20.4228.543/6742/6714/2829/3928/39764
pcdserver@qwen3.8-2b-distill-q6pcdServerGqwen3.8-2b-distillfine-tunedq61.6143.543/6743/6716/2827/3927/39122
ollaya@laya-enollayaGlaya-endedicatedfp320.8519.442/6716/2826/391,804
systemone@system-one-distilled †Transformers NLI (MPS)Gsystem-one-distilleddedicatedfp320.2821.242/6717/2825/39
systemone@system-one-gold †Transformers NLI (MPS)Gsystem-one-golddedicatedfp320.2821.442/6717/2825/39
pcdserver@zerocoder-0.8b-q2pcdServerGzerocoder-0.8bfine-tunedq20.3834.742/6736/6715/2827/3921/39339
systemone@raz-nli-xsmall-openjev †Raz NLI (MPS)Graz-nli-xsmall-openjevdedicatedfp320.2849.442/6714/2828/39
pcdserver@qwen1.5-4b-q8pcdServerGqwen1.5-4bvanillaq84.20247.842/6743/6716/2826/3927/39322
pcdserver@lq-decide-1.7b-q4pcdServerGlq-decide-1.7bfine-tunedq41.1143.841/6742/6715/2826/3927/39108
pcdserver@qwen1.5-4b-q4pcdServerGqwen1.5-4bvanillaq42.46232.141/6743/6716/2825/3927/39391
pcdserver@qwen1.5-4b-q2pcdServerGqwen1.5-4bvanillaq21.62233.041/6742/6716/2825/3926/39444
pcdserver@qwen3-reranker-0.6b-q5pcdServerGqwen3-reranker-0.6bfine-tunedq50.4424.740/6744/6715/2825/3929/3958
pcdserver@qwen3.5-0.8b-q2pcdServerGqwen3.5-0.8bvanillaq20.4632.240/6742/6713/2827/3929/39541
pcdserver@qwen1.5-4b-q6pcdServerGqwen1.5-4bvanillaq63.25275.740/6743/6716/2824/3927/39309
systemone@clm-8b-q8System One serverC+Gclm-8bdedicatedq88.25107.639/6728/6714/2825/3915/39
pcdserver@qwen1.5-4b-q3pcdServerGqwen1.5-4bvanillaq32.03237.239/6743/6716/2823/3927/39441
pcdserver@qwen1.5-4b-q5pcdServerGqwen1.5-4bvanillaq52.84283.639/6742/6716/2823/3926/39424
systemone@clm-8b-mlx6 †CLM embeddings + head (MLX/CPU)C+Gclm-8bdedicatedq67.24415.739/6714/2825/39
ollaya@clm-8bollayaCclm-8bdedicatedbf1616.501,677.939/6730/6714/2825/3916/3949,184
systemone@clm-8b-mlx8System One serverC+Gclm-8bdedicatedmlx89.29100.438/6731/6714/2824/3917/39
systemone@clm-8b-q5System One serverC+Gclm-8bdedicatedq55.78119.238/6727/6713/2825/3914/39
systemone@clm-8b-q6-k †CLM embeddings + head (Metal/CPU)C+Gclm-8bdedicatedq6_k7.10490.738/6713/2825/39
systemone@clm-8b-q5-km †CLM embeddings + head (Metal/CPU)C+Gclm-8bdedicatedq5_k_m6.31509.938/6713/2825/39
onnx@decima-small-int8ONNX RuntimeCdecima-smalldedicatedint80.1312.437/6738/6713/2824/3925/392,290
systemone@system-one-zeroshot †Transformers NLI (MPS)Gsystem-one-zeroshotdedicatedfp320.2821.237/6714/2823/39
laya-wd-gguf-q4llama.cpp embeddingsC+Glaya-multilingualdedicatedq443.336/6738/6714/2822/3924/392,243
systemone@clm-8b-q4System One serverC+Gclm-8bdedicatedq45.03113.836/6728/6713/2823/3916/39
systemone@clm-8b-q4-km †CLM embeddings + head (Metal/CPU)C+Gclm-8bdedicatedq4_k_m5.57449.436/6713/2823/39
pcdserver@qwen1.5-0.5b-q3pcdServerGqwen1.5-0.5bvanillaq30.3521.235/6732/6711/2824/3921/39248
pcdserver@lyco-router-0.6b-q4pcdServerGlyco-router-0.6bfine-tunedq40.4826.435/6733/6712/2823/3921/39514
torch@pulse-decide-150m †PyTorchGpulse-decide-150mdedicated0.6017.934/6715/2819/3910,043
pcdserver@qwen3.5-0.8b-unsloth-iq3pcdServerGqwen3.5-0.8b-unslothvanillaiq30.4028.234/6731/6712/2822/3919/39246
pcdserver@qwen1.5-1.8b-q3pcdServerGqwen1.5-1.8bvanillaq31.0241.834/6735/6715/2819/3920/39510
pcdserver@qwen3-reranker-0.6b-q6pcdServerGqwen3-reranker-0.6bfine-tunedq60.4924.533/6737/6713/2820/3924/3961
pcdserver@qwen1.5-1.8b-q4pcdServerGqwen1.5-1.8bvanillaq41.2240.033/6739/6715/2818/3924/39343
torch@lumma-fev-0.6bPyTorchGlumma-fev-0.6bdedicated1.3069.533/6728/6714/2819/3914/396,272
systemone@clm-8b-mlx4System One serverC+Gclm-8bdedicatedmlx45.2089.633/6728/6711/2822/3918/39
pcdserver@zerocoder-0.8b-q3pcdServerGzerocoder-0.8bfine-tunedq30.4534.632/6730/6711/2821/3919/39291
pcdserver@qwen1.5-1.8b-q8pcdServerGqwen1.5-1.8bvanillaq81.9636.432/6736/6713/2819/3923/39106
pcdserver@qwen1.5-1.8b-q5pcdServerGqwen1.5-1.8bvanillaq51.3841.332/6735/6713/2819/3922/39327
mlx@julia-1MLXGjulia-1dedicated0.586.831/6731/6711/2820/3920/391,866
onnx@julia-1ONNX RuntimeCjulia-1dedicatedfp320.5812.231/6731/6711/2820/3920/39976
pcdserver@qwen1.5-1.8b-q6pcdServerGqwen1.5-1.8bvanillaq61.5839.831/6737/6714/2817/3923/39334
pcdserver@qwen1.5-0.5b-q6pcdServerGqwen1.5-0.5bvanillaq60.5122.630/6728/678/2822/3921/39197
pcdserver@qwen3-reranker-0.6b-q8pcdServerGqwen3-reranker-0.6bfine-tunedq80.6423.330/6733/6712/2818/3921/3957
pcdserver@qwen3-reranker-0.6b-f16pcdServerGqwen3-reranker-0.6bfine-tunedf161.2023.830/6732/6712/2818/3920/3953
pcdserver@qwen1.5-0.5b-q8pcdServerGqwen1.5-0.5bvanillaq80.6621.128/6728/677/2822/3921/39177
pcdserver@qwen1.5-0.5b-q5pcdServerGqwen1.5-0.5bvanillaq50.4621.728/6728/677/2822/3922/39238
pcdserver@qwen2.5-0.5b-q4pcdServerGqwen2.5-0.5bvanillaq40.4920.228/6726/6710/2818/3916/39605
pcdserver@qwen3-reranker-0.6b-q4pcdServerGqwen3-reranker-0.6bfine-tunedq40.4029.228/6734/6710/2818/3924/39339
pcdserver@qwen2.5-0.5b-pcb-q2pcdServerGqwen2.5-0.5b-pcbfine-tunedq20.3418.127/6723/679/2818/3914/39334
pcdserver@zerocoder-0.8b-f16pcdServerGzerocoder-0.8bfine-tunedf161.6433.027/6723/6712/2815/3911/39288
pcdserver@zerocoder-0.8b-q4pcdServerGzerocoder-0.8bfine-tunedq40.5346.427/6724/6712/2815/3912/39270
pcdserver@qwen2.5-0.5b-f16pcdServerGqwen2.5-0.5bvanillaf161.2717.826/6725/679/2817/3916/39158
pcdserver@qwen1.5-0.5b-q4pcdServerGqwen1.5-0.5bvanillaq40.4121.626/6728/678/2818/3921/39217
pcdserver@qwen3.5-0.8b-unsloth-iq2pcdServerGqwen3.5-0.8b-unslothvanillaiq20.3428.226/6723/679/2817/3914/39312
pcdserver@qwen1.5-1.8b-q2pcdServerGqwen1.5-1.8bvanillaq20.8639.826/6736/6710/2816/3926/39392
systemone@systemone-lite-0.5b †SystemOne Lite (MPS)Gsystemone-lite-0.5bdedicatedbf16/fp320.9956.426/6712/2814/39
ollaya@gliclass-largeollayaCgliclass-largededicatedfp321.7759.826/679/2817/393,794
pcdserver@qwen2.5-0.5b-q8pcdServerGqwen2.5-0.5bvanillaq80.6817.825/6724/678/2817/3916/3940
pcdserver@zerocoder-0.8b-q8pcdServerGzerocoder-0.8bfine-tunedq80.8734.425/6721/6711/2814/3910/39184
pcdserver@zerocoder-0.8b-q6pcdServerGzerocoder-0.8bfine-tunedq60.6835.324/6721/6711/2813/3910/39182
pcdserver@qwen2.5-0.5b-q6pcdServerGqwen2.5-0.5bvanillaq60.6522.223/6724/678/2815/3916/39134
pcdserver@qwen2.5-0.5b-q5pcdServerGqwen2.5-0.5bvanillaq50.5218.922/6722/677/2815/3915/39219
pcdserver@qwen1.5-0.5b-q2pcdServerGqwen1.5-0.5bvanillaq20.3021.121/6722/678/2813/3914/39247
pcdserver@qwen3-reranker-0.6b-q3pcdServerGqwen3-reranker-0.6bfine-tunedq30.3524.221/6721/678/2813/3913/3989
systemone@raz-nli-base †Raz NLI (MPS)Graz-nli-basededicatedfp320.7438.521/6710/2811/39
dohnuts-metal@this-that-model-1.2-q2dohnuts (Metal)Gthis-that-model-1.2dedicatedq20.97112.221/6718/678/2813/3910/39
pcdserver@qwen2.5-0.5b-pcb-q3pcdServerGqwen2.5-0.5b-pcbfine-tunedq30.3617.720/6720/677/2813/3913/39185
ollaya@nli-modernbert-largeollayaGnli-modernbert-largededicatedfp320.8022.720/679/2811/391,201
pcdserver@euaiact-0.5b-q4pcdServerGeuaiact-0.5bfine-tunedq40.4023.820/6720/678/2812/3912/39697
systemone@system-one-270m †Transformers slot (MPS)Gsystem-one-270mdedicatedbf16/fp320.5432.820/678/2812/39
pcdserver@qwen2.5-0.5b-pcb-f16pcdServerGqwen2.5-0.5b-pcbfine-tunedf160.9917.219/6720/677/2812/3913/39144
pcdserver@qwen2.5-0.5b-pcb-q6pcdServerGqwen2.5-0.5b-pcbfine-tunedq60.5117.519/6720/677/2812/3913/3979
pcdserver@qwen2.5-0.5b-pcb-q8pcdServerGqwen2.5-0.5b-pcbfine-tunedq80.5317.519/6720/677/2812/3913/3940
pcdserver@qwen2.5-0.5b-pcb-q5pcdServerGqwen2.5-0.5b-pcbfine-tunedq50.4217.619/6719/677/2812/3912/39259
pcdserver@qwen2.5-0.5b-q3pcdServerGqwen2.5-0.5bvanillaq30.4318.418/6719/675/2813/3914/39316
pcdserver@zerocoder-0.8b-q5pcdServerGzerocoder-0.8bfine-tunedq50.6036.218/6717/678/2810/399/39216
dohnuts-metal@decider-0.8b-q2dohnuts (Metal)Gdecider-0.8bdedicatedq20.4255.218/6717/677/2811/3910/39
slot@decider-0.8b-q2slot (llama-server)Gdecider-0.8bdedicatedq20.4264.718/6717/677/2811/3910/39108
pcdserver@qwen2.5-0.5b-pcb-q4pcdServerGqwen2.5-0.5b-pcbfine-tunedq40.4022.917/6719/676/2811/3913/39361
pcdserver@decider-0.8b-q2pcdServerGdecider-0.8bdedicatedq20.4229.317/6717/674/2813/3913/39577
systemone@oomu-systemone-0.6b †llama-cpp-python (Metal)Goomu-systemone-0.6bdedicatedq8_00.6465.217/6717/677/2810/3910/39
pcdserver@qwen2.5-0.5b-q2pcdServerGqwen2.5-0.5bvanillaq20.4217.615/6716/674/2811/3912/39477
pcdserver@qwen3-4b-unsloth-iq1pcdServerGqwen3-4b-unslothvanillaiq11.0872.513/6714/676/287/398/39594
pcdserver@lq-decide-0.6b-q8pcdServerGlq-decide-0.6bfine-tunedq80.6423.111/6711/674/287/397/39135
pcdserver@qwen3-reranker-0.6b-q2pcdServerGqwen3-reranker-0.6bfine-tunedq20.3023.910/6714/675/285/399/39241
pcdserver@lq-decide-0.6b-q4pcdServerGlq-decide-0.6bfine-tunedq40.4029.510/6711/674/286/397/39183

CPU and mixed runs

The audit found 19 CPU-only runs and 19 runs with an explicit CPU/GPU split. This table identifies them all. It excludes CPU/Neural Engine and automatic-placement runs; those have their own codes in the full table. Historical labels describe recorded configurations, rather than a per-operation utilization trace.

CPU-only runs and pipelines with an explicit CPU model/head component.
MethodC/GExecutionEvidencems/queryTranslatedDirect
pcdserver@rune-mradermacher-26b-a4b-q5 †C+GCPU weights (--gpu-layers 0); Metal computation and KV offload enabledconfigured962.365/6765/67
pcdserver@rune-mradermacher-26b-a4b-q8 †CCPU only (--gpu-layers 0 --cpu); compute and KV offload disabledconfigured758.164/6764/67
gguf-head@jev-omni-q3C+GMetal encoder; CPU decision head (NumPy/PyTorch)configured1,222.364/6764/67
pcdserver@rune-mradermacher-26b-a4b-q4 †C+GCPU weights (--gpu-layers 0); Metal computation and KV offload enabledconfigured487.163/6764/67
gguf-head@jev-omni-q8C+GMetal encoder; CPU decision head (NumPy/PyTorch)configured1,124.163/6763/67
gguf-head@jev-omni-q5C+GMetal encoder; CPU decision head (NumPy/PyTorch)configured1,212.763/6763/67
dohnuts@decider-0.8b-dreamblooms-q8CCPU; recorded runtime/launch configurationconfigured275.262/6760/67
coreml@gliner25-decide-fp16-l256-cpu-gpu †C+GCore ML explicitly CPU_AND_GPU; Neural Engine disabled; exact operation placement unprofiledconfigured21.961/67
onnx@gliner25-decide-fp32CONNX Runtime CPUExecutionProviderconfigured67.361/67
ollaya@decider-0.8bCollaya ONNX Runtime CPU on this Mac; recorded runtime recipeconfigured284.861/6760/67
coreml@kev-0.8b-fp16-l512 †C+GCore ML CPU_AND_GPU; exact operation placement unprofiledconfigured121.360/6762/67
ollaya@kev-0.8bCollaya ONNX Runtime CPU on this Mac; recorded runtime recipeconfigured268.260/6762/67
ollaya@von-1.1Collaya ONNX Runtime CPU on this Mac; recorded runtime recipeconfigured62.857/67
onnx@von-1.2CONNX Runtime CPUExecutionProviderconfigured52.657/67
ollaya@decision-eosCollaya ONNX Runtime CPU on this Mac; recorded runtime recipeconfigured322.957/6756/67
gguf-head@laya-neutron-mlC+GMetal encoder; CPU decision head (NumPy/PyTorch)configured42.754/6751/67
systemone@system-one-scorer-onnx †CONNX Runtime CPUExecutionProvider; scalar Q4 graph, CPU tokenizationconfigured1,410.853/67
laya-ml-onnx-fp32CONNX Runtime CPUExecutionProviderconfigured23.452/6749/67
onnx@laya-ml-fp32CONNX Runtime CPUExecutionProviderconfigured24.752/6749/67
laya-ml-onnx-int8CONNX Runtime CPUExecutionProviderconfigured25.552/6749/67
laya-wd-gguf-q8C+GMetal encoder; CPU decision head (NumPy/PyTorch)configured44.452/6749/67
laya-wd-gguf-f16C+GMetal encoder; CPU decision head (NumPy/PyTorch)configured44.452/6749/67
laya-ml-ggufCCPU; recorded runtime/launch configurationconfigured57.652/6749/67
onnx@laya-ml-fp16CONNX Runtime CPUExecutionProviderconfigured57.652/6749/67
laya-ml-gguf-f16CCPU; recorded runtime/launch configurationconfigured61.452/6749/67
laya-wd-gguf-q6C+GMetal encoder; CPU decision head (NumPy/PyTorch)configured44.551/6747/67
laya-ml-gguf-q4CCPU; recorded runtime/launch configurationconfigured55.051/6747/67
laya-wd-gguf-q3C+GMetal encoder; CPU decision head (NumPy/PyTorch)configured45.349/6743/67
laya-wd-gguf-q5C+GMetal encoder; CPU decision head (NumPy/PyTorch)configured45.547/6743/67
gguf-head@laya-neutron-enC+GMetal encoder; CPU decision head (NumPy/PyTorch)configured62.446/67
systemone@clm-8b-q8C+GMetal/MLX backbone; CLM decision head configured --device cpuconfigured107.639/6728/67
systemone@clm-8b-mlx6 †C+GMetal/MLX backbone; CLM decision head configured --device cpuconfigured415.739/67
ollaya@clm-8bCollaya ONNX Runtime CPU on this Mac; recorded runtime recipeconfigured1,677.939/6730/67
systemone@clm-8b-mlx8C+GMetal/MLX backbone; CLM decision head configured --device cpuconfigured100.438/6731/67
systemone@clm-8b-q5C+GMetal/MLX backbone; CLM decision head configured --device cpuconfigured119.238/6727/67
systemone@clm-8b-q6-k †C+GMetal/MLX backbone; CLM decision head configured --device cpuconfigured490.738/67
systemone@clm-8b-q5-km †C+GMetal/MLX backbone; CLM decision head configured --device cpuconfigured509.938/67
onnx@decima-small-int8CONNX Runtime CPUExecutionProviderconfigured12.437/6738/67
laya-wd-gguf-q4C+GMetal encoder; CPU decision head (NumPy/PyTorch)configured43.336/6738/67
systemone@clm-8b-q4C+GMetal/MLX backbone; CLM decision head configured --device cpuconfigured113.836/6728/67
systemone@clm-8b-q4-km †C+GMetal/MLX backbone; CLM decision head configured --device cpuconfigured449.436/67
systemone@clm-8b-mlx4C+GMetal/MLX backbone; CLM decision head configured --device cpuconfigured89.633/6728/67
onnx@julia-1CONNX Runtime CPUExecutionProviderconfigured12.231/6731/67
ollaya@gliclass-largeCollaya ONNX Runtime CPU on this Mac; recorded runtime recipeconfigured59.826/67

One set of weights on several engines

An engine is not a neutral container. This table puts the same model file side by side on every engine that ran it. decider-0.8b scores 61 or 62 through its own trained readout on dohnuts and llama-server, but 55 on pcdServer, which reads it as a chat model. Chapter 6 explains the difference.

Click a column header to sort.
ModelQuantEngineC/Gms/queryTranslatedDirect
decider-0.8bf16pcdServerG30.755/6756/67
decider-0.8bf16dohnuts (Metal)G51.961/6760/67
decider-0.8bf16slot (llama-server)G63.661/6760/67
decider-0.8bfp16ExecuTorch (MLX)G89.761/6760/67
decider-0.8bfp32ollayaC284.861/6760/67
decider-0.8bq2pcdServerG29.317/6717/67
decider-0.8bq2dohnuts (Metal)G55.218/6717/67
decider-0.8bq2slot (llama-server)G64.718/6717/67
decider-0.8bq3pcdServerG29.352/6747/67
decider-0.8bq3dohnuts (Metal)G54.557/6755/67
decider-0.8bq3slot (llama-server)G69.757/6755/67
decider-0.8bq4pcdServerG40.257/6757/67
decider-0.8bq4dohnuts (Metal)G53.361/6759/67
decider-0.8bq4slot (llama-server)G67.461/6759/67
decider-0.8bq5pcdServerG29.957/6757/67
decider-0.8bq5dohnuts (Metal)G55.262/6760/67
decider-0.8bq5slot (llama-server)G67.962/6760/67
decider-0.8bq6pcdServerG29.855/6757/67
decider-0.8bq6dohnuts (Metal)G53.562/6762/67
decider-0.8bq6slot (llama-server)G68.162/6762/67
decider-0.8bq8pcdServerG30.255/6757/67
decider-0.8bq8dohnuts (Metal)G51.462/6761/67
decider-0.8bq8slot (llama-server)G64.062/6761/67
decider-0.8b-dreambloomsq8pcdServerG28.355/6757/67
decider-0.8b-dreambloomsq8dohnuts (Metal)G52.262/6761/67
decider-0.8b-dreambloomsq8slot (llama-server)G65.162/6761/67
decider-0.8b-dreambloomsq8dohnuts (CPU)C275.262/6760/67
decider-2bf16pcdServerG42.260/6760/67
decider-2bf16dohnuts (Metal)G100.359/6762/67
decider-2bf16slot (llama-server)G112.459/6762/67
decider-2bfp16ExecuTorch (MLX)G251.359/6762/67
decider-2biq4pcdServerG41.457/6757/67
decider-2biq4dohnuts (Metal)G106.756/6757/67
decider-2biq4slot (llama-server)G126.856/6757/67
decider-2b-dreambloomsq8pcdServerG40.159/6760/67
decider-2b-dreambloomsq8dohnuts (Metal)G103.759/6762/67
decider-2b-dreambloomsq8slot (llama-server)G114.459/6762/67
decider-35b-a3bq4dohnuts (Metal)G266.364/6764/67
decider-35b-a3bq4pcdServerG296.863/6763/67
decider-35b-a3bq4slot (llama-server)G299.164/6764/67
decider-4bbf16PyTorchG348.263/6765/67

Best model per family

The best row of every model, grouped by family. A dedicated model was trained for this kind of question and has its own readout; a fine-tuned model is a chat model tuned by someone for a task; a vanilla model is a stock chat model. Best method names the row the other columns come from. Chapter 4 describes the families.

Click a column header to sort.
FamilyModelEngineC/GGBms/queryTranslatedDirectBest method
dedicatedtypesafe/jev-1.13OpenRouter APIR364.465/6765/67openrouter@typesafe/jev-1.13
dedicatedrune-26b-a4bslot (llama-server)G13.52366.665/6757/67slot@rune-26b-a4b-q3
dedicatedrune-mradermacher-26b-a4bpcdServerC+G19.13962.365/6765/67pcdserver@rune-mradermacher-26b-a4b-q5
dedicatedjaredpalmer/kev-4bOpenRouter APIR1,653.065/6765/67openrouter@jaredpalmer/kev-4b
dedicateddecider-35b-a3bdohnuts (Metal)G21.17266.364/6764/67dohnuts-metal@decider-35b-a3b-q4
dedicatedneohorse-4bSystem One serverG5.17273.164/6764/67systemone@neohorse-4b-q8
dedicatedjpt-4bdohnuts (Metal)G4.48335.664/6764/67dohnuts-metal@jpt-4b-q8
dedicatedtogethercomputer/tev1-4b-experimentalOpenRouter APIR339.364/6765/67openrouter@togethercomputer/tev1-4b-experimental
dedicatedinception/mercury-decide:freeOpenRouter APIR408.364/6764/67openrouter@inception/mercury-decide:free
dedicatedliquid/d1OpenRouter APIR566.264/6764/67openrouter@liquid/d1
dedicatedwinnow-12bollayaG12.67891.164/6765/67ollaya@winnow-12b
dedicatedjev-omnillama.cpp + MLX headG6.09908.464/6764/67gguf-head@jev-omni-q3-last-mlx
dedicatedjpt-9bpcdServerG6.47155.263/6764/67pcdserver@jpt-9b-q5
dedicatedopenjev-35b-a3bMLXG20.99250.063/6762/67mlx@openjev-35b-a3b-4bit
dedicatedjeb-35b-a3bSystem One serverG17.17333.263/6763/67systemone@jeb-35b-a3b-q3
dedicateddecider-4bPyTorchG348.263/6765/67torch@decider-4b-bf16
dedicatedclef-flashClef joint head (MLX)G5.28554.863/6763/67systemone@clef-flash-mlx4
dedicatedlev-4bSystem One serverG9.492,203.863/6764/67systemone@lev-4b
dedicateddecider-0.8bdohnuts (Metal)G0.8151.462/6761/67dohnuts-metal@decider-0.8b-q8
dedicateddecider-0.8b-dreambloomsdohnuts (Metal)G0.8152.262/6761/67dohnuts-metal@decider-0.8b-dreamblooms-q8
dedicatedsystem-one-minicpm5-2b-q8System One (MLX)G2.67141.862/6759/67systemone@system-one-minicpm5-2b-q8
dedicatedjet-4bdohnuts (Metal)G4.48281.462/6763/67dohnuts-metal@jet-4b-q8
dedicatedjevk5-4bslot (llama-server)G2.71350.562/6763/67slot@jevk5-4b-q4
dedicatedimajev-4bSystem One serverG9.81363.162/6764/67systemone@imajev-4b
dedicatedkev-4blaya.cppG1,103.162/6755/67kev-4b-gguf
dedicatedupstage/solar-decideOpenRouter APIR1,253.362/6763/67openrouter@upstage/solar-decide
dedicatedgliner25-decideCore MLC+G0.9321.961/67coreml@gliner25-decide-fp16-l256-cpu-gpu
dedicateddecider-2bpcdServerG3.7842.260/6760/67pcdserver@decider-2b-f16
dedicatedtev1-0.8bMLXG0.4264.860/6759/67mlx@tev1-0.8b-4bit
dedicatedkev-0.8bCore MLC+G1.51121.360/6762/67coreml@kev-0.8b-fp16-l512
dedicateddecider-2b-dreambloomspcdServerG2.0140.159/6760/67pcdserver@decider-2b-dreamblooms-q8
dedicatedvon-1.2ONNX RuntimeC1.5852.657/67onnx@von-1.2
dedicatedvon-1.1ollayaC1.5962.857/67ollaya@von-1.1
dedicatedthis-that-model-1.2dohnuts (Metal)G1.10113.157/6756/67dohnuts-metal@this-that-model-1.2-q3
dedicatedleo-1.7bSystem One serverG3.52136.257/6762/67systemone@leo-1.7b
dedicateddecision-eosollayaC1.54322.957/6756/67ollaya@decision-eos
dedicatedrespan/span-01-lite:freeOpenRouter APIR436.056/6760/67openrouter@respan/span-01-lite:free
dedicatedrespan/span-01-liteOpenRouter APIR447.556/6760/67openrouter@respan/span-01-lite
dedicatedrespan/span-01OpenRouter APIR487.456/6760/67openrouter@respan/span-01
dedicatedbosun-1.7bllama-cpp-python (Metal)G1.26130.855/6755/67systemone@bosun-1.7b-q5
dedicatedlaya-multilingualllama.cpp embeddings + headC+G0.2742.754/6751/67gguf-head@laya-neutron-ml
dedicatedsystem-one-scorer-onnxONNX scalar (CPU)C2.641,410.853/67systemone@system-one-scorer-onnx
dedicatedsemif-4b-nliMLXG8.41120.651/6749/67mlx@semif-4b-nli
dedicatedsystem-one-qwen3-0.6bSystem One (MLX)G1.5247.749/6752/67systemone@system-one-qwen3-0.6b
dedicatedlaya-typed-decisionsPyTorchG17.748/6743/67laya-typed-decisions
dedicatedlaya-enllama.cpp embeddings + headC+G0.2962.446/67gguf-head@laya-neutron-en
dedicatedxDecisionxDecision (MLX)G0.4014.345/6743/67systemone@xdecision-q8-mlx
dedicatedsystem-one-distilledTransformers NLI (MPS)G0.2821.242/67systemone@system-one-distilled
dedicatedsystem-one-goldTransformers NLI (MPS)G0.2821.442/67systemone@system-one-gold
dedicatedraz-nli-xsmall-openjevRaz NLI (MPS)G0.2849.442/67systemone@raz-nli-xsmall-openjev
dedicatedclm-8bSystem One serverC+G8.25107.639/6728/67systemone@clm-8b-q8
dedicateddecima-smallONNX RuntimeC0.1312.437/6738/67onnx@decima-small-int8
dedicatedsystem-one-zeroshotTransformers NLI (MPS)G0.2821.237/67systemone@system-one-zeroshot
dedicatedpulse-decide-150mPyTorchG0.6017.934/67torch@pulse-decide-150m
dedicatedlumma-fev-0.6bPyTorchG1.3069.533/6728/67torch@lumma-fev-0.6b
dedicatedjulia-1MLXG0.586.831/6731/67mlx@julia-1
dedicatedsystemone-lite-0.5bSystemOne Lite (MPS)G0.9956.426/67systemone@systemone-lite-0.5b
dedicatedgliclass-largeollayaC1.7759.826/67ollaya@gliclass-large
dedicatedraz-nli-baseRaz NLI (MPS)G0.7438.521/67systemone@raz-nli-base
dedicatednli-modernbert-largeollayaG0.8022.720/67ollaya@nli-modernbert-large
dedicatedsystem-one-270mTransformers slot (MPS)G0.5432.820/67systemone@system-one-270m
dedicatedoomu-systemone-0.6bllama-cpp-python (Metal)G0.6465.217/6717/67systemone@oomu-systemone-0.6b
fine-tunedqwen3.5-4b-hmmpcdServerG4.4887.664/6765/67pcdserver@qwen3.5-4b-hmm-q8
fine-tunedqwen3.8-4b-distillpcdServerG2.7893.662/6763/67pcdserver@qwen3.8-4b-distill-q4
fine-tunedqwen3.8-9b-distillpcdServerG5.78137.862/6764/67pcdserver@qwen3.8-9b-distill-q4
fine-tunedornith-1.5-35b-a3b-splashSplash (Metal)G20.95322.462/6763/67splash@ornith-1.5-35b-a3b-splash
fine-tunedqwen3.5-4b-supercoderpcdServerG2.6184.261/6762/67pcdserver@qwen3.5-4b-supercoder-q4
fine-tunedqwen2.5-coder-7bpcdServerG4.68108.961/6762/67pcdserver@qwen2.5-coder-7b-q4
fine-tunedqwen2.5-coder-3bpcdServerG2.1060.260/6759/67pcdserver@qwen2.5-coder-3b-q4
fine-tunedpulse-0.6bpcdServerG0.4027.954/6752/67pcdserver@pulse-0.6b-q4
fine-tunedlq-decide-1.7bpcdServerG1.8335.150/6746/67pcdserver@lq-decide-1.7b-q8
fine-tunedqwen3.8-2b-distillpcdServerG1.3142.949/6751/67pcdserver@qwen3.8-2b-distill-q4
fine-tunedzerocoder-0.8bpcdServerG0.3834.742/6736/67pcdserver@zerocoder-0.8b-q2
fine-tunedqwen3-reranker-0.6bpcdServerG0.4424.740/6744/67pcdserver@qwen3-reranker-0.6b-q5
fine-tunedlyco-router-0.6bpcdServerG0.4826.435/6733/67pcdserver@lyco-router-0.6b-q4
fine-tunedqwen2.5-0.5b-pcbpcdServerG0.3418.127/6723/67pcdserver@qwen2.5-0.5b-pcb-q2
fine-tunedeuaiact-0.5bpcdServerG0.4023.820/6720/67pcdserver@euaiact-0.5b-q4
fine-tunedlq-decide-0.6bpcdServerG0.6423.111/6711/67pcdserver@lq-decide-0.6b-q8
vanillaqwen3.5-4b-unslothpcdServerG4.4885.063/6764/67pcdserver@qwen3.5-4b-unsloth-q8
vanillaqwen2.5-3bpcdServerG3.2953.362/6757/67pcdserver@qwen2.5-3b-q8
vanillaqwen3.5-4bpcdServerG3.0189.862/6763/67pcdserver@qwen3.5-4b-q4
vanillabonsai-27bpcdServerG3.80513.562/6763/67pcdserver@bonsai-27b-q1
vanillaqwen3.5-2bpcdServerG1.2242.761/6762/67pcdserver@qwen3.5-2b-q3
vanillaqwen3-4b-unslothpcdServerG8.0571.561/6762/67pcdserver@qwen3-4b-unsloth-bf16
vanillaqwen3-4bpcdServerG2.5075.561/6762/67pcdserver@qwen3-4b-q4
vanillaqwen3-8bpcdServerG5.03116.061/6762/67pcdserver@qwen3-8b-q4
vanillaqwen3.5-9b-unslothpcdServerG5.68136.961/6763/67pcdserver@qwen3.5-9b-unsloth-q4
vanillaqwen3.5-0.8bpcdServerG0.6532.157/6757/67pcdserver@qwen3.5-0.8b-q5
vanillaqwen3-1.7b-unslothpcdServerG1.8335.157/6759/67pcdserver@qwen3-1.7b-unsloth-q8
vanillaqwen3-1.7bpcdServerG2.1735.257/6759/67pcdserver@qwen3-1.7b-q8
vanillaqwen3.5-2b-unslothpcdServerG2.0139.557/6759/67pcdserver@qwen3.5-2b-unsloth-q8
vanillaqwen3.5-0.8b-unslothpcdServerG0.5929.656/6757/67pcdserver@qwen3.5-0.8b-unsloth-q5
vanillaqwen3.5-0.8b-ggmlpcdServerG1.5630.355/6757/67pcdserver@qwen3.5-0.8b-ggml-bf16
vanillabonsai-1.7bpcdServerG0.4934.954/6755/67pcdserver@bonsai-1.7b-q2
vanillaqwen3-0.6bpcdServerG0.6425.553/6754/67pcdserver@qwen3-0.6b-q8
vanillaqwen1.5-4bpcdServerG4.20247.842/6743/67pcdserver@qwen1.5-4b-q8
vanillaqwen1.5-0.5bpcdServerG0.3521.235/6732/67pcdserver@qwen1.5-0.5b-q3
vanillaqwen1.5-1.8bpcdServerG1.0241.834/6735/67pcdserver@qwen1.5-1.8b-q3
vanillaqwen2.5-0.5bpcdServerG0.4920.228/6726/67pcdserver@qwen2.5-0.5b-q4

Quantization sweeps

Each row is one model on one engine, measured at several quantizations. Each cell is the translated score out of 67, then the mean time per query; an empty cell was not measured. Between Q4 and Q8 the scores barely move, and below Q3 they collapse. Chapter 7 reads the grid.

Translated score out of 67 and mean ms per query; separated by CPU/GPU configuration
ModelEngineC/Giq1q2iq2q3iq3q4mlx4q5q6q8mlx8f16bf16
clef-flashllama-server (System One)G
clm-8bSystem One serverC+G36 · 114 ms33 · 90 ms38 · 119 ms39 · 108 ms38 · 100 ms
decider-0.8bdohnuts (Metal)G18 · 55 ms57 · 54 ms61 · 53 ms62 · 55 ms62 · 54 ms62 · 51 ms61 · 52 ms
decider-0.8bpcdServerG17 · 29 ms52 · 29 ms57 · 40 ms57 · 30 ms55 · 30 ms55 · 30 ms55 · 31 ms
decider-0.8bslot (llama-server)G18 · 65 ms57 · 70 ms61 · 67 ms62 · 68 ms62 · 68 ms62 · 64 ms61 · 64 ms
jpt-4bdohnuts (Metal)G61 · 315 ms62 · 291 ms63 · 367 ms64 · 336 ms
jpt-4bpcdServerG60 · 94 ms63 · 93 ms62 · 99 ms63 · 86 ms
jpt-9bdohnuts (Metal)G61 · 637 ms63 · 620 ms63 · 620 ms63 · 625 ms
jpt-9bpcdServerG60 · 139 ms61 · 137 ms63 · 155 ms61 · 142 ms
laya-multilingualllama.cpp embeddingsC+G49 · 45 ms36 · 43 ms47 · 45 ms51 · 45 ms52 · 44 ms52 · 44 ms
qwen1.5-0.5bpcdServerG21 · 21 ms35 · 21 ms26 · 22 ms28 · 22 ms30 · 23 ms28 · 21 ms
qwen1.5-1.8bpcdServerG26 · 40 ms34 · 42 ms33 · 40 ms32 · 41 ms31 · 40 ms32 · 36 ms
qwen1.5-4bpcdServerG41 · 233 ms39 · 237 ms41 · 232 ms39 · 284 ms40 · 276 ms42 · 248 ms
qwen2.5-0.5bpcdServerG15 · 18 ms18 · 18 ms28 · 20 ms22 · 19 ms23 · 22 ms25 · 18 ms26 · 18 ms
qwen2.5-0.5b-pcbpcdServerG27 · 18 ms20 · 18 ms17 · 23 ms19 · 18 ms19 · 17 ms19 · 17 ms19 · 17 ms
qwen2.5-3bpcdServerG53 · 58 ms49 · 56 ms57 · 61 ms62 · 59 ms61 · 62 ms62 · 62 ms62 · 53 ms62 · 54 ms
qwen3-1.7bpcdServerG54 · 38 ms50 · 38 ms54 · 41 ms57 · 39 ms57 · 39 ms57 · 35 ms57 · 36 ms
qwen3-1.7b-unslothpcdServerG44 · 37 ms53 · 38 ms56 · 40 ms57 · 39 ms57 · 39 ms57 · 35 ms
qwen3-4b-unslothpcdServerG13 · 72 ms59 · 73 ms60 · 78 ms61 · 76 ms60 · 79 ms60 · 79 ms60 · 70 ms61 · 72 ms
qwen3-reranker-0.6bpcdServerG10 · 24 ms21 · 24 ms28 · 29 ms40 · 25 ms33 · 25 ms30 · 23 ms30 · 24 ms
qwen3.5-0.8bpcdServerG40 · 32 ms46 · 31 ms52 · 30 ms57 · 32 ms55 · 29 ms55 · 31 ms55 · 30 ms
qwen3.5-0.8b-unslothpcdServerG43 · 28 ms26 · 28 ms48 · 30 ms34 · 28 ms55 · 50 ms56 · 30 ms55 · 29 ms55 · 29 ms55 · 31 ms
qwen3.5-2bpcdServerG49 · 41 ms61 · 43 ms57 · 47 ms57 · 42 ms57 · 43 ms57 · 39 ms
qwen3.5-2b-unslothpcdServerG53 · 41 ms44 · 41 ms57 · 42 ms57 · 40 ms57 · 45 ms57 · 43 ms56 · 44 ms57 · 40 ms57 · 42 ms
qwen3.5-4b-unslothpcdServerG61 · 94 ms52 · 87 ms60 · 93 ms61 · 90 ms61 · 94 ms63 · 94 ms62 · 94 ms63 · 85 ms
qwen3.8-2b-distillpcdServerG49 · 43 ms45 · 46 ms43 · 43 ms44 · 39 ms
this-that-model-1.2dohnuts (Metal)G21 · 112 ms57 · 113 ms51 · 110 ms54 · 117 ms53 · 108 ms
zerocoder-0.8bpcdServerG42 · 35 ms32 · 35 ms27 · 46 ms18 · 36 ms24 · 35 ms25 · 34 ms27 · 33 ms

Every query

Every one of the 67 queries, hardest first. Right, translated and right, direct are the share of all methods that answered it correctly in each mode. jev is jev's answer on the translated text, which you can compare with the labelled task. Chapter 9 discusses the three hardest.

Click a column header to sort.
QueryLangTaskRight, translatedRight, directjev
How do I write a liga feature?endocs12 %12 %fea
Which words should I look at to judge the spacing of round letters?ensample23 %23 %sample
Show me how a glyph is stored in a VFJ fileendocs26 %26 %vfj
Text with lots of kerning pairsensample38 %38 %sample
VFJ 형식으로 이름이 hyphen이고 너비가 300인 글리프 정의를 작성해 줘kovfj39 %90 %vfj
Loop through masters and print their namesenpython48 %48 %python
Rename all glyphs in every open font, I don't want to clickenpython53 %53 %python
Positioning rule that lifts the acute over capital lettersenfea57 %57 %fea
Can FontLab batch rename glyphs?endocs60 %60 %docs
Script the kerning for A Venfea60 %60 %fea
I need substitutions so that 1/2 becomes onehalfenfea61 %61 %fea
پائتھن اسکرپٹ لکھو جو تمام گلفس کی چوڑائی 600 کر دےurpython62 %69 %python
Can you automate adding sidebearings of 40 to every glyph in the selection?enpython64 %64 %python
Donne-moi un pangramme français pour tester mes accentsfrsample65 %67 %sample
תן לי טקסט לדוגמה בעברית כדי לבדוק את הריווחhesample67 %75 %sample
fi と fl の liga フィーチャーを書いてくださいjafea68 %80 %fea
Come faccio a esportare il font in OTF?itdocs69 %73 %docs
Ge mig ett pangram på svenska för att testa typsnittetsvsample71 %72 %sample
Napisz coś do okna Glyph, żeby sprawdzić ligatury fi flplsample71 %67 %sample
I want a pangram that uses all Cyrillic letters to test in the Glyph windowensample73 %73 %sample
چطور فونت را به صورت OTF خروجی بگیرم؟fadocs73 %65 %docs
Anna esimerkkiteksti, jossa on paljon ä- ja ö-kirjaimiafisample74 %72 %sample
Describe the letter O as JSON with nodes for the FontLab formatenvfj75 %75 %vfj
Як змінити сімейну назву шрифту?ukdocs76 %71 %docs
Escreva o código da feature smcp para versaletesptfea76 %79 %fea
Jak vyexportovat font jako OTF?csdocs76 %66 %docs
Hoe voeg ik een nieuwe as toe aan een variabel font?nldocs76 %70 %docs
Écris une fonctionnalité liga pour fi et flfrfea77 %78 %fea
Give me the code that makes ff become a ligatureenfea77 %77 %fea
Գրիր kern feature Ա և Վ տառերի համարhyfea78 %79 %fea
Something to type in the preview so I can see all my ligaturesensample78 %78 %sample
The font exports fine but the OTF name shows wrong in Word, why?endocs79 %79 %docs
What would the JSON for a glyph look like?envfj79 %79 %vfj
Which menu has the option to flip the selection?endocs80 %80 %docs
Írj egy kern feature-t az A és V betűk közéhufea80 %80 %fea
Write a liga feature that substitutes f i with fi and f l with flenfea80 %80 %fea
Schreib mir ein kern-Feature für die Buchstaben A V W Tdefea80 %74 %fea
Jak zmienić kąt pochylenia kursywy w FontLab?pldocs81 %72 %docs
اكتب كود feature للربط بين الحرفين f و iarfea81 %84 %fea
Πώς αλλάζω τη γωνία των πλάγιων γραμμάτων στο FontLab;eldocs81 %74 %docs
FontLabでイタリック角度を変更するには？jadocs82 %76 %docs
How do I change the italic angle of my font in FontLab?endocs82 %82 %docs
I need kern feature code with classes for round letters versus flat lettersenfea82 %82 %fea
Something to type in the Glyph window that shows off my ligaturesensample82 %82 %sample
Wo stelle ich die vertikalen Metriken ein?dedocs82 %78 %docs
დამიწერე Python სკრიპტი, რომელიც ყველა გლიფს გაასწორებსkapython83 %92 %python
FontLab में kerning पैनल कहाँ है?hidocs84 %83 %docs
A와 V 사이의 커닝 feature 코드를 작성해 줘kofea84 %81 %fea
What does the Match Masters command actually do?endocs84 %84 %docs
给我一段包含所有拼音字母的示例文本，用来检查字体zhsample85 %86 %sample
Viết định nghĩa VFJ cho glyph dấu gạch ngang rộng 300vivfj86 %90 %vfj
Give me a script to export every open font as OTF into ~/Desktop/outenpython87 %87 %python
Дай мне текст-образец с кириллицей для окна Glyphrusample89 %85 %sample
Dame un texto de muestra con muchas letras redondasessample89 %90 %sample
Give me a sample text with lots of round letters so I can check my o and censample90 %90 %sample
写一个 Python 脚本，把所有字形的宽度设为 600zhpython90 %92 %python
Escribe un script en Python que elimine los glifos sin unicodeespython90 %94 %python
Give me a JSON glyph entry in VFJ format for a composite eacute made of e and acutecombenvfj91 %91 %vfj
Напиши скрипт на Python, который удалит все направляющиеrupython91 %89 %python
Napisz skrypt Python, który zmieni nazwy wszystkich glifów z .sc na .smcpplpython92 %87 %python
Напиши VFJ-описание глифа «точка» с юникодом 002Ebgvfj92 %91 %vfj
เขียนสคริปต์ Python ที่เปลี่ยนชื่อ glyph ทั้งหมดเป็นตัวพิมพ์เล็กthpython93 %88 %python
Tüm glifleri 50 birim sağa kaydıran bir Python betiği yaztrpython93 %93 %python
Write a Python script that renames all glyphs ending in .sc to .smcpenpython93 %93 %python
একটি VFJ গ্লিফ সংজ্ঞা লিখুন যার নাম period এবং প্রস্থ 250bnvfj93 %88 %vfj
Write the VFJ definition of a glyph named A with a triangular outline, unicode 0041, width 600envfj95 %95 %vfj
Scrivimi la definizione VFJ di un glifo quadrato largo 500itvfj95 %94 %vfj

The translator

The translated mode runs Hy-MT2-1.8B at Q4_K_M on llama-server in front of the classifier, with a short glossary of font-editing terms. The table lists the 39 non-English queries, their English translations and the time each took. Each query was translated once, so every classifier saw the same English. Chapter 5 describes the prompt.

Click a column header to sort.
QueryLangEnglishms
Jak zmienić kąt pochylenia kursywy w FontLab?plHow can I change the italic angle in FontLab?149
Napisz skrypt Python, który zmieni nazwy wszystkich glifów z .sc na .smcpplWrite a Python script that changes the names of all glyphs from .sc to .smcp112
Schreib mir ein kern-Feature für die Buchstaben A V W TdeWrite me a kern feature for the letters A V W T90
Wo stelle ich die vertikalen Metriken ein?deWhere do I set the vertical metrics?61
Come faccio a esportare il font in OTF?itHow can I export the font to OTF?77
Scrivimi la definizione VFJ di un glifo quadrato largo 500itWrite me the VFJ definition for a square glyph with a width of 50099
Écris une fonctionnalité liga pour fi et flfrWrite a liga functionality for fi and fl70
Donne-moi un pangramme français pour tester mes accentsfrGive me a French pangram to test my accents71
Dame un texto de muestra con muchas letras redondasesGive me a sample text with many round letters74
Escribe un script en Python que elimine los glifos sin unicodeesWrite a Python script that removes glifos without Unicode.85
Jak vyexportovat font jako OTF?csHow to export a font as OTF?66
Escreva o código da feature smcp para versaletesptWrite the code for the smcp feature for Versaletes85
Hoe voeg ik een nieuwe as toe aan een variabel font?nlHow do I add a new font style to a variable font?91
Tüm glifleri 50 birim sağa kaydıran bir Python betiği yaztrA Python script that shifts all glyphs 50 units to the right95
Írj egy kern feature-t az A és V betűk közéhuWrite a kern feature between the letters A and V81
Anna esimerkkiteksti, jossa on paljon ä- ja ö-kirjaimiafiAn example text with many A and O letters72
Viết định nghĩa VFJ cho glyph dấu gạch ngang rộng 300viWrite the VFJ definition for the wide 300-width horizontal glyph90
Ge mig ett pangram på svenska för att testa typsnittetsvGive me a pangram in Swedish to test the font81
Дай мне текст-образец с кириллицей для окна GlyphruGive me a text sample with Cyrillic characters for the Glyph window98
Напиши скрипт на Python, который удалит все направляющиеruWrite a Python script that removes all leading spaces66
Як змінити сімейну назву шрифту?ukHow do I change the family name of a font?83
Напиши VFJ-описание глифа «точка» с юникодом 002EbgWrite the VFJ description for the glyph “dot” with Unicode 002E104
Πώς αλλάζω τη γωνία των πλάγιων γραμμάτων στο FontLab;elHow do I change the angle of the horizontal lines in FontLab?95
Գրիր kern feature Ա և Վ տառերի համարhyGriir kern feature A and V for spacing81
დამიწერე Python სკრიპტი, რომელიც ყველა გლიფს გაასწორებსkaWrite the Python script, which contains many glyphs76
اكتب كود feature للربط بين الحرفين f و iarWrite a feature code to connect the letters f and i81
چطور فونت را به صورت OTF خروجی بگیرم؟faHow can I export the font as an OTF file?85
תן לי טקסט לדוגמה בעברית כדי לבדוק את הריווחheGive me a Hebrew text example so I can check the kerning89
پائتھن اسکرپٹ لکھو جو تمام گلفس کی چوڑائی 600 کر دےurWrite the font script so that all glyphs have a width of 600.104
FontLab में kerning पैनल कहाँ है?hiWhere is the kerning panel in FontLab?77
একটি VFJ গ্লিফ সংজ্ঞা লিখুন যার নাম period এবং প্রস্থ 250bnWrite a VFJ glyph definition named period with a width of 250.99
เขียนสคริปต์ Python ที่เปลี่ยนชื่อ glyph ทั้งหมดเป็นตัวพิมพ์เล็กthWrite a Python script that changes the name of all glyphs to lowercase.99
FontLabでイタリック角度を変更するには？jaHow to change the italic angle in FontLab?77
fi と fl の liga フィーチャーを書いてくださいjaPlease write the features of fi and fl liga.77
给我一段包含所有拼音字母的示例文本，用来检查字体zhGive me an example text containing all phonetic letters to check the font.88
写一个 Python 脚本，把所有字形的宽度设为 600zhWrite a Python script to set the width of all glyphs to 600.97
A와 V 사이의 커닝 feature 코드를 작성해 줘koWrite the kerning feature code between A and V81
VFJ 형식으로 이름이 hyphen이고 너비가 300인 글리프 정의를 작성해 줘koThe FontLab file has a hyphen name and a glyph definition with a width of 300.113
Napisz coś do okna Glyph, żeby sprawdzić ligatury fi flplWrite something in the Glyph window to check the ligatures fi fl94

Language detection

Before it translates, the pipeline has to know whether a query is English. Two detectors, and a majority vote that breaks ties in lingua's favour, were measured over 100 passes of the 67 queries, single-threaded. Chapter 5 lists lingua's four misses.

DetectorCorrectMean µsCalls/s
lingua63/67481.12,079
papagan43/6711.190,437
majority63/67492.22,032

Calibration

These tables group the answers of decider-0.8b (DreamBlooms' Q8_0 file, the one ornotto registers) on dohnuts (Metal) by their top probability. Share right is the fraction of answers in that band that were correct; a calibrated model is right about as often as its probability says. Chapter 9 reads the bands.

With translation:

Top probabilityAnswersRightShare right
below 0.68788 %
0.6 to 0.81111100 %
0.8 to 0.95171482 %
0.95 and up313097 %

On the original text:

Top probabilityAnswersRightShare right
below 0.610880 %
0.6 to 0.89889 %
0.8 to 0.95161488 %
0.95 and up323197 %

A fallback gate

A gate asks decider-0.8b on dohnuts first and sends the query to Qwen3.5-4B-Hmm on pcdServer only when decider's top probability is below the threshold. The rows replay the benchmark's cached answers at each threshold. Fallbacks counts the queries sent on; mean ms is the cost per query averaged over all 67, where a query that falls back pays for both calls. The thresholds were chosen on the same queries they are scored on, so this is not a held-out test. Chapter 9 explains why the gate pays only on the original text.

With translation:

Gate: top probability belowFallbacksCorrectMean ms
0.507/6762/6761.2
0.608/6762/6762.5
0.7017/6762/6773.9
0.8019/6762/6776.4
0.8523/6762/6781.6
0.9029/6763/6789.2
0.9536/6763/6798.1

On the original text:

Gate: top probability belowFallbacksCorrectMean ms
0.508/6762/6762.7
0.6010/6762/6765.2
0.7015/6763/6771.5
0.8019/6763/6776.6
0.8522/6763/6780.6
0.9028/6763/6788.3
0.9535/6764/6797.2

Copyright © 2026 Fontlab Ltd. Text and data CC BY 4.0, code Apache-2.0.