Results: the short version

We asked 320 methods one router question: which of five tasks does a FontLab user want? There were 67 queries in 30 languages, and each method was scored twice, on English translations and on the original text. The hosted reference, jev, answered 65 of 67 either way. Rune Q5 on pcdServer matches it in both modes at 962 ms/query with CPU weights. An earlier local method matched it on translated text: rune-26b-a4b version 1 at Q3_K_M, which scored 65/67 but fell to 57/67 on the original text. Among the 4B models, Qwen3.5-4B-Hmm on pcdServer scored 64/67 and 65/67 in 88 ms, and the best small local dedicated model, NeoHorse-Jev-4B, scored 64/67 in both modes in 273 ms. The fastest method at 60/67 or better is GLiNER2.5-Decide on Core ML CPU/GPU: 61/67 on translated text in 21.9 ms.

Rune Q5 full Metal adds a separate 64/64 at 153.3 ms run: 6.3× faster, with one fewer correct answer in each mode than its retained mixed run. Ornith Splash adds 62/63 at 322.4/328.5 ms translated/direct. Only these completed new runs appear; the explorer explains exclusions and device evidence.

The separate Rune Q4 Metal rerun scores 63/67 translated and 64/67 direct at 124.8 ms, against 487.1 ms with CPU weights. Every row now carries a C/G execution label. Explore the Pareto diagrams with your own accuracy floor: laya on MLX reaches 52/67 translated in 6.9 ms.

Fresh OpenRouter Jev scores 65/67 in both modes at 364.4 ms translated and 375.1 ms direct. Tev1 reaches 65/67 direct at 335.1 ms; Kev reaches 65/67 in both modes but takes over 1.5 seconds per query. All nine endpoints appear under OpenRouter API in the explorer.

The top fifteen

The 15 best methods by translated score, then direct score, then speed. Click to sort.

Engine	C/G	Model	Family	Quant	GB	ms/query	Translated	Direct
OpenRouter API	R	typesafe/jev-1.13	dedicated			364.4	65/67	65/67
pcdServer	C+G	rune-mradermacher-26b-a4b	dedicated	q5	19.13	962.3	65/67	65/67
OpenRouter API	R	jaredpalmer/kev-4b	dedicated			1,653.0	65/67	65/67
slot (llama-server)	G	rune-26b-a4b	dedicated	q3	13.52	366.6	65/67	57/67
pcdServer	G	qwen3.5-4b-hmm	fine-tuned	q8	4.48	87.6	64/67	65/67
pcdServer	G	qwen3.5-4b-hmm	fine-tuned	q4	2.71	92.6	64/67	65/67
pcdServer	G	qwen3.5-4b-hmm	fine-tuned	bf16	8.42	94.8	64/67	65/67
OpenRouter API	R	togethercomputer/tev1-4b-experimental	dedicated			339.3	64/67	65/67
ollaya	G	winnow-12b	dedicated	q8_0	12.67	891.1	64/67	65/67
pcdServer	G	rune-mradermacher-26b-a4b	dedicated	q3	13.29	146.1	64/67	64/67
pcdServer	G	rune-mradermacher-26b-a4b	dedicated	q5	19.13	153.3	64/67	64/67
dohnuts (Metal)	G	decider-35b-a3b	dedicated	q4	21.17	266.3	64/67	64/67
System One server	G	neohorse-4b	dedicated	q8	5.17	273.1	64/67	64/67
slot (llama-server)	G	decider-35b-a3b	dedicated	q4	21.17	299.1	64/67	64/67
dohnuts (Metal)	G	jpt-4b	dedicated	q8	4.48	335.6	64/67	64/67

Best per engine

Scores read translated / direct, out of 67. A dash means the model reads English only.

• dohnuts (Metal): decider-35b-a3b at Q4, 64 / 64 in 266 ms from 21 GB. The small decider-0.8b that ornotto registers scores 62 / 61 in 52 ms from 0.81 GB.
• pcdServer: Rune Q5 scores 65 / 65 in 962 ms with CPU weights and Metal computation (C+G). Qwen3.5-4B-Hmm at Q8 scores 64 / 65 in 88 ms on GPU; its 2.7 GB Q4_K_M file scores the same in 93 ms.
• slot (llama-server): rune-26b-a4b at Q3, 65 / 57 in 367 ms from 13.5 GB. pcdServer rejects its Gemma 4 files, so it runs only here.
• ollaya: winnow-12b, 64 / 65 in 891 ms from 12.7 GB.
• Encoders: GLiNER2.5-Decide on Core ML CPU/GPU, 61 / – in 21.9 ms. laya is the only method under 10 ms with a useful score: 52/67 in 7 ms on MLX.

The full list, one row per engine and runtime, is in chapter 6.

What to pick

• A good default: decider-0.8b on dohnuts. It scores 62/67 in 52 ms from 0.81 GB, and its probabilities are calibrated.
• Accurate local routing around 90 ms: Qwen3.5-4B-Hmm on pcdServer, 64/67 translated and 65/67 on the original text, from 2.7 GB at Q4_K_M. It needs no translator.
• Remote decisions: ornotto now supports all nine measured OpenRouter endpoints through its native API. Compare their scores and network-inclusive latency in the remote results table, or choose Engine: OpenRouter API in the explorer. Respan uses a separately labelled noul OVR readout.

Chapter 12 covers the other cases, from graded answers to several questions about one message.

Where to read more

• Results: every table has all 320 rows, filterable and sortable, and every other table the benchmark produced.
• Chapter 5 explains how we measured and what each column means.
• Chapter 6 reads the results model by model.
• Chapter 7 covers quantization and file size.
• Chapter 8 explains where the milliseconds go.
• Chapter 9 tests whether the probabilities can be trusted.

Two methods within two or three queries of each other are not measurably different here. To rank models on your own decision, run them on your own labelled examples.