Find the fastest useful answer¶
An extra correct answer can cost hundreds of milliseconds. Choose the score you need, then compare the runs that meet it. The default floor is 50 correct answers out of 67. Laya on MLX clears it at 52/67 in 6.9 ms on translated text; a model does not have to top the accuracy table to be useful.
These are 320 recorded configurations, not 320 different models. Each point names its engine, quantization and execution device. Set Direct to compare original multilingual queries; its chart uses the recorded direct-mode latency. The translated-mode latency covers classification alone and excludes the translation step.
Loading interactive charts…
C = CPU · G = GPU · C+G = mixed · C+N = CPU + Neural Engine · A = automatic CPU/GPU/Neural Engine placement · R = remote · ? = CPU/GPU unrecorded. Configuration describes the run; it is not a hardware utilization trace.
Hover a point for its exact run. Click a point or a table row for execution evidence.
| Run | C/G | Correct / 67 | ms/query | Weight GB | Pareto | Readout |
|---|
OpenRouter: nine remote models, ten runs¶
The earlier jev row and the later typesafe/jev-1.13 row measure the same TypeSafe Jev model over OpenRouter. They are separate runs, not different models. Both original timings and scores remain; the homepage counts Jev once.
All nine requested endpoints were measured on the same 67 queries, in translated and direct modes, on 3 October 2026. Translations and local results are retained from the earlier runs. Each endpoint received 106 distinct inputs, sequentially, with duplicate inputs reused across modes. Latency includes the network trip and provider processing; R means the provider did not disclose CPU/GPU placement or weight precision.
Six endpoints use a native five-way choice. Respan's three endpoints support noul only: each request contains five independent yes/no questions using the same task descriptions. We choose the largest P(true), then normalize the five values for the routing report. This noul OVR readout is labelled separately; its values are not calibrated exclusive-choice probabilities. Choose OpenRouter API in the explorer's engine filter to select ten recorded runs of nine endpoints. Its table shows the readout for each run.
| Model | Run | Resolved version | C/G | Readout | Translated | Direct | Translated ms | Direct ms |
|---|---|---|---|---|---|---|---|---|
| typesafe/jev-1.13 | 3 October 2026 | typesafe/jev-1.13-20260917 | R | native choice | 65/67 | 65/67 | 364.4 | 375.1 |
| typesafe/jev-1.13 | earlier run | not logged | R | native choice | 65/67 | 65/67 | 466.1 | 455.2 |
| jaredpalmer/kev-4b | 3 October 2026 | jaredpalmer/kev-4b-20260924 | R | native choice | 65/67 | 65/67 | 1,653.0 | 1,525.0 |
| togethercomputer/tev1-4b-experimental | 3 October 2026 | togethercomputer/tev1-4b-experimental-20260923 | R | native choice | 64/67 | 65/67 | 339.3 | 335.1 |
| inception/mercury-decide:free | 3 October 2026 | inception/mercury-decide-20260930 | R | native choice | 64/67 | 64/67 | 408.3 | 404.5 |
| liquid/d1 | 3 October 2026 | liquid/d1-20260930 | R | native choice | 64/67 | 64/67 | 566.2 | 583.6 |
| upstage/solar-decide | 3 October 2026 | upstage/solar-decide-20260928 | R | native choice | 62/67 | 63/67 | 1,253.3 | 1,292.3 |
| respan/span-01-lite:free | 3 October 2026 | respan/span-01-lite-20260925 | R | noul OVR | 56/67 | 60/67 | 436.0 | 474.6 |
| respan/span-01-lite | 3 October 2026 | respan/span-01-lite-20260925 | R | noul OVR | 56/67 | 60/67 | 447.5 | 462.6 |
| respan/span-01 | 3 October 2026 | respan/span-01-20260925 | R | noul OVR | 56/67 | 60/67 | 487.4 | 566.3 |
Read the frontier¶
A run is on the Pareto frontier when no other visible run is both at least as accurate and at least as fast (or small), with a strict improvement in one. Equal points remain tied. Change a filter and the frontier is recalculated within that subset. It is a description of these measurements, not statistical evidence that one model is better.
- Speed frontier compares accuracy with mean classification latency. A hosted point includes its network trip; local points include their runtime and loopback overhead. Device filters let you compare CPU runs with CPU runs, or GPU runs with GPU runs.
- Size frontier compares accuracy with recorded storage size in decimal GB (weight files, or the complete prepared Splash package). It excludes runs with no recorded size, including hosted Jev. It does not measure peak memory, KV cache or computation buffers.
- Fastest qualifying runs sorts the current selection by latency. With “Pareto frontier only” checked, that bar chart, the table and the CSV show the speed frontier; the size plot shows its own frontier.
Hover to inspect a point, click for execution evidence, drag to zoom, and double-click to reset the chart. The camera control exports an SVG. Filter settings are stored in the URL so you can share a view. A blank direct score means an English-only run and is excluded from Direct mode.
Rune Q4: the device changed the result¶
The fresh full-Metal Q4 run requested all layers with --gpu-layers -1; the server reported MTL0. It measured 124.8 ms/query, with 63/67 translated and 64/67 direct, against 487.1 ms/query for the earlier CPU-weight/Metal-compute configuration. That is 3.9× faster with the same answers. Both rows are retained: search rune q4 and lower the minimum score if necessary.
Q3 and the new Q4/Q5 configurations request GPU weights. The historical Q5 keeps CPU weights with Metal computation and KV offload (C+G). Q8 is CPU-only (C). Their timings must not be read as a pure quantization comparison. See the CPU audit for the other CPU and mixed pipelines.
Rune Q5 and Splash: two completed October runs¶
The separate Rune Q5 full-Metal run scores 64/67 in both modes, at 153.3 ms/query translated and 152.8 ms direct. It is 6.3× faster than the retained mixed CPU-weight/Metal-compute run at 962.3 ms, but changes one answer in each mode and loses one correct answer. The weights and frozen translations are the same; changed device arithmetic is not evidence of identical predictions. Search rune q5 to compare both configurations.
Ornith 1.5 35B-A3B on Splash 1.1.0 scores 62/67 translated at 322.4 ms and 63/67 direct at 328.5 ms. Select Splash (Metal) to see its single completed row. The native System One endpoint reads five option probabilities without generating text. These timings do not measure speculative decoding throughput. Calibration on this task is unverified. Its 20.95 GB is the prepared package's storage size, including target, draft and tokenizer, rather than runtime memory; the packed MoE format has mixed q4/q8 sections. It is a fine-tuned model, and package geometry limits compatibility. See the Splash source and runtime documentation.
Each new method has 106 successful distinct-input records, reused to evaluate all 67 queries in both modes. All 300 earlier public rows and their underlying query records and translations are retained unchanged.
Runs excluded from the charts¶
Qwen3.6 Splash was refused before loading: 33 GiB available versus the unchanged conservative 40 GiB requirement. Swift Splash loading coincided with a host kernel panic involving the storage stack; its root cause remains unresolved. Dense Qwen3.8 Splash and Rune Q8 full-Metal remain untried and excluded. Laya Q8 on ggmlc Metal loaded but aborted on its first question with unsupported operation MAP_CUSTOM2; no successful benchmark records were saved. The F16 and Q4 alternatives are deferred because they use the same runtime path. None is a zero-score chart point or a completed run.
The GPU audit retains existing faster native alternatives for CPU dohnuts and Laya. ONNX CPU rows remain labelled CPU: a Core ML provider setting would permit fallback and needs a separately verified run. Automatic Core ML/Core AI placement stays A. Jev-Omni and Neutron now have separately measured GPU-head alternatives with probability agreement checks; their historical CPU-head rows remain. Lower-scoring CLM pipelines are not priority retries. GPU preference follows measured score and latency, not a launch flag.
Three GPU provider and head comparisons¶
All three completed configurations preserve every earlier result. Timings below exclude translation; the same frozen texts were used.
| Configuration | Translated / direct correct | ms/query translated / direct | Retained comparison |
|---|---|---|---|
| GLiNER2.5-Decide, Core ML CPU/GPU | 61 / – | 21.9 / – | Automatic placement: 61 / – at 349.1 ms |
| Jev-Omni Q3, last-token pooling and MLX head | 64 / 64 | 908.4 / 904.4 | GPU encoder, CPU head: 64 / 64 at 1222.3 ms translated |
| Neutron multilingual Q4, MPS head | 54 / 51 | 50.3 / 49.4 | GPU encoder, CPU head: 54 / 51 at 42.7 ms translated |
GLiNER is 15.9 times faster on this run with no changed answers; its largest probability difference is 0.0038. The compiled package permits CPU and GPU and disables the Neural Engine; operations were not individually traced. Jev-Omni is 1.35 times faster with identical saved four-decimal probabilities. Pooling and head placement changed together, so the gain cannot be assigned to the head alone. Neutron also preserves those probabilities but is 18% slower; retain the mixed configuration when latency matters. MLX/MPS GPU labels refer to the configured encoder/head computation; HTTP and tokenization still use CPU. These benchmark head adapters are separate from the package's native encoder integrations.
Weidows Q8: another MPS counterexample¶
The same Weidows Q8 encoder and decision head also completed all 106 inputs with the head explicitly on MPS. It scored 52/49 at 45.9/75.9 ms translated/direct, versus 44.4/44.6 ms for the retained CPU head. Every saved four-decimal probability is identical. The direct mean includes a 2020 ms Bengali request; it remains in the result rather than being removed as an outlier. This single run does not establish a general MPS penalty. The native Laya MLX result remains faster at 6.9 ms, with a different probability profile. The new row records the pinned Weidows weights and 0.37 GB encoder-plus-head storage size; the original row's missing size/provenance fields remain unchanged.
Direct ONNX-to-CoreML attempts produced no completed runs. The von model failed to build a Core ML execution plan; enabling its compute-plan diagnostic first stalled until the guarded timeout. GLiNER and Laya exports failed with an axis error; disabling subgraph offload and graph optimization did not fix GLiNER. Their original CPU rows remain. These failures provide no GPU accuracy or speed measurement, and do not establish the cause of the earlier host kernel panic.
Devices and evidence¶
C means CPU inference; G means GPU inference on Metal, MLX or PyTorch MPS. C+G identifies an explicit CPU model/head component plus GPU work, and C+N permits CPU and Neural Engine. Routine CPU tokenization and orchestration do not turn every GPU run into C+G. A means automatic CPU/GPU/Neural Engine placement whose exact assignment was not recorded. R means a remote service whose hardware is undisclosed; ? is reserved for missing CPU/GPU evidence.
Most historical labels are reconstructed from recorded launch settings and runtime recipes and marked configured. The fresh Rune Q4/Q5 labels are observed from its Metal server log. These labels do not claim that every operation was profiled. Four automatic-placement runs cannot honestly be labelled CPU-only or GPU-only.
Download or regenerate¶
Download the self-contained explorer. Save the HTML file and open it locally: its results, chart library and controls are embedded, so it works offline. The CSV button exports the currently selected runs with their device descriptions and evidence.
The tool is src_docs/gen_diagrams.py. Run python3 src_docs/gen_diagrams.py --data src_docs/data/classifiers.json --out explorer.html from the repository. ./docs.sh regenerates the book's explorer and offline download automatically. Plotly.js 3.1.0 is bundled under its MIT licence.
How this compares with the Decision Index¶
The Jev Decision Index uses a broad, chance-corrected benchmark panel. This report measures one five-way FontLab routing task, with 67 queries in 30 languages. Its 52/67 is a count of correct routing answers, not 52 Decision Index points. The suites, metrics and latency protocols differ, so their scores and timings cannot be ranked together.
Our contribution is a detailed comparison of local runtimes, quantizations and devices, with the hosted Jev reference measured on the same routing inputs. For the prompts, timing boundaries and limitations, read How we measured. A difference of one or two answers on this small set needs a larger, held-out test before it guides a close choice.