6. Results

jev, the hosted reference, answered 65 of the 67 router queries correctly in both modes, at 466 ms per query. The best local model came within one query of it: Qwen3.5-4B-Hmm on pcdServer scored 64/67 on translated text and 65/67 on direct text, at 88 ms per query from a 4.5 GB file, or 93 ms from the 2.7 GB Q4_K_M file. The dedicated decider-0.8b on dohnuts (Metal) scored 62/67 and 61/67 at 52 ms from the 0.81 GB DreamBlooms Q8_0 file that ornotto registers, and the laya encoder scored 52/67 in 7 ms. 208 methods finished; two failed and are left out.

The top of the table

The fifteen best methods, ordered by translated accuracy, then direct accuracy, then speed:

The 15 best methods by translated score, then direct score, then speed. Click to sort.

Engine	Model	Family	Quant	GB	ms/query	Translated	Direct
jev (hosted)	jev	dedicated			466.1	65/67	65/67
pcdServer	qwen3.5-4b-hmm	fine-tuned	q8	4.48	87.6	64/67	65/67
pcdServer	qwen3.5-4b-hmm	fine-tuned	q4	2.71	92.6	64/67	65/67
pcdServer	qwen3.5-4b-hmm	fine-tuned	bf16	8.42	94.8	64/67	65/67
dohnuts (Metal)	decider-35b-a3b	dedicated	q4	21.17	266.3	64/67	64/67
slot (llama-server)	decider-35b-a3b	dedicated	q4	21.17	299.1	64/67	64/67
PyTorch	decider-4b	dedicated	bf16		348.2	63/67	65/67
pcdServer	qwen3.5-4b-unsloth	vanilla	q8	4.48	85.0	63/67	64/67
pcdServer	qwen3.5-4b-unsloth	vanilla	q5	3.14	94.5	63/67	64/67
pcdServer	decider-35b-a3b	dedicated	q4	21.17	296.8	63/67	63/67
pcdServer	qwen3.8-9b-distill	fine-tuned	q4	5.78	137.8	62/67	64/67
pcdServer	qwen3.5-4b	vanilla	q4	3.01	89.8	62/67	63/67
pcdServer	qwen3.5-4b-unsloth	vanilla	q6	3.53	93.6	62/67	63/67
pcdServer	qwen3.8-4b-distill	fine-tuned	q4	2.78	93.6	62/67	63/67
slot (llama-server)	qwen3.5-4b-hmm	fine-tuned	bf16	8.42	284.2	62/67	63/67

Five findings stand out.

• A small, dedicated model gets within three queries of jev. decider-0.8b answers 62 of 67 through its own readout, from a file a fifth the size of the Hmm model's Q8 build. decider-0.8b's Q8_0 conversions are the fastest methods to score 62 or more (51 ms for mradermacher's, 52 ms for DreamBlooms'), and its probabilities are calibrated.
• A tuned 4B chat model on pcdServer matches the 35B decider. Qwen3.5-4B-Hmm ties decider-35b-a3b on dohnuts on translated text (64 each) and beats it by one answer on direct text (65 against 64). Its 2.7 GB Q4 build is an eighth the size of the 35B file and runs three times faster. Its accuracy does not move from Q4 to BF16.
• Vanilla chat models are close behind. unsloth's Qwen3.5-4B at Q8 scored 63/64 at 85 ms, one query behind the Hmm tune, with no task-specific training. Qwen3.5-2B at Q3_K_M scored 61/62 at 43 ms from 1.2 GB.
• Size does not rescue an old model. The best Qwen1.5 method, at 4B, scored 42/67 in 248 ms. Qwen3.5 at 0.8B scored 57.
• Speed and accuracy separate at the extremes. laya runs in 4 to 7 ms on Apple silicon but tops out at 52. kev-4b reaches 62 on translated text but takes 1.1 seconds per query.

Every method

All 208 methods are in the table below. Click a column header to sort by it, and click again to reverse the order. Type in the box to keep only the rows that contain every word you type: pcdServer hmm keeps the Hmm model on pcdServer, and dedicated Metal keeps the dedicated models on dohnuts (Metal). Chapter 5 defines every column.

All 208 methods, best first. Click a header to sort; empty cells were not measured.

Method	Engine	Model	Family	Quant	GB	ms/query	Translated	Direct	English	Other, translated	Other, direct	Load ms
jev	jev (hosted)	jev	dedicated			466.1	65/67	65/67	26/28	39/39	39/39
pcdserver@qwen3.5-4b-hmm-q8	pcdServer	qwen3.5-4b-hmm	fine-tuned	q8	4.48	87.6	64/67	65/67	26/28	38/39	39/39	462
pcdserver@qwen3.5-4b-hmm-q4	pcdServer	qwen3.5-4b-hmm	fine-tuned	q4	2.71	92.6	64/67	65/67	26/28	38/39	39/39	296
pcdserver@qwen3.5-4b-hmm-bf16	pcdServer	qwen3.5-4b-hmm	fine-tuned	bf16	8.42	94.8	64/67	65/67	26/28	38/39	39/39	341
dohnuts-metal@decider-35b-a3b-q4	dohnuts (Metal)	decider-35b-a3b	dedicated	q4	21.17	266.3	64/67	64/67	26/28	38/39	38/39
slot@decider-35b-a3b-q4	slot (llama-server)	decider-35b-a3b	dedicated	q4	21.17	299.1	64/67	64/67	26/28	38/39	38/39	375
pcdserver@qwen3.5-4b-unsloth-q8	pcdServer	qwen3.5-4b-unsloth	vanilla	q8	4.48	85.0	63/67	64/67	25/28	38/39	39/39	346
pcdserver@qwen3.5-4b-unsloth-q5	pcdServer	qwen3.5-4b-unsloth	vanilla	q5	3.14	94.5	63/67	64/67	25/28	38/39	39/39	354
pcdserver@decider-35b-a3b-q4	pcdServer	decider-35b-a3b	dedicated	q4	21.17	296.8	63/67	63/67	24/28	39/39	39/39	799
torch@decider-4b-bf16	PyTorch	decider-4b	dedicated	bf16		348.2	63/67	65/67	26/28	37/39	39/39	16,802
dohnuts-metal@decider-0.8b-q8	dohnuts (Metal)	decider-0.8b	dedicated	q8	0.81	51.4	62/67	61/67	24/28	38/39	37/39
dohnuts-metal@decider-0.8b-dreamblooms-q8	dohnuts (Metal)	decider-0.8b-dreamblooms	dedicated	q8	0.81	52.2	62/67	61/67	24/28	38/39	37/39
pcdserver@qwen2.5-3b-q8	pcdServer	qwen2.5-3b	vanilla	q8	3.29	53.3	62/67	57/67	23/28	39/39	35/39	178
pcdserver@qwen2.5-3b-f16	pcdServer	qwen2.5-3b	vanilla	f16	6.18	53.5	62/67	60/67	23/28	39/39	37/39	252
dohnuts-metal@decider-0.8b-q6	dohnuts (Metal)	decider-0.8b	dedicated	q6	0.63	53.5	62/67	62/67	24/28	38/39	38/39
dohnuts-metal@decider-0.8b-q5	dohnuts (Metal)	decider-0.8b	dedicated	q5	0.58	55.2	62/67	60/67	24/28	38/39	36/39
pcdserver@qwen2.5-3b-q4	pcdServer	qwen2.5-3b	vanilla	q4	1.93	59.2	62/67	60/67	23/28	39/39	37/39	339
pcdserver@qwen2.5-3b-q6	pcdServer	qwen2.5-3b	vanilla	q6	2.54	62.0	62/67	60/67	23/28	39/39	37/39	213
slot@decider-0.8b-q8	slot (llama-server)	decider-0.8b	dedicated	q8	0.81	64.0	62/67	61/67	24/28	38/39	37/39	428
slot@decider-0.8b-dreamblooms-q8	slot (llama-server)	decider-0.8b-dreamblooms	dedicated	q8	0.81	65.1	62/67	61/67	24/28	38/39	37/39	689
slot@decider-0.8b-q5	slot (llama-server)	decider-0.8b	dedicated	q5	0.58	67.9	62/67	60/67	24/28	38/39	36/39	142
slot@decider-0.8b-q6	slot (llama-server)	decider-0.8b	dedicated	q6	0.63	68.1	62/67	62/67	24/28	38/39	38/39	216
pcdserver@qwen3.5-4b-q4	pcdServer	qwen3.5-4b	vanilla	q4	3.01	89.8	62/67	63/67	24/28	38/39	39/39	311
pcdserver@qwen3.5-4b-unsloth-q6	pcdServer	qwen3.5-4b-unsloth	vanilla	q6	3.53	93.6	62/67	63/67	24/28	38/39	39/39	291
pcdserver@qwen3.8-4b-distill-q4	pcdServer	qwen3.8-4b-distill	fine-tuned	q4	2.78	93.6	62/67	63/67	24/28	38/39	39/39	284
pcdserver@qwen3.8-9b-distill-q4	pcdServer	qwen3.8-9b-distill	fine-tuned	q4	5.78	137.8	62/67	64/67	25/28	37/39	39/39	449
dohnuts@decider-0.8b-dreamblooms-q8	dohnuts (CPU)	decider-0.8b-dreamblooms	dedicated	q8	0.81	275.2	62/67	60/67	24/28	38/39	36/39
slot@qwen3.5-4b-hmm-bf16	slot (llama-server)	qwen3.5-4b-hmm	fine-tuned	bf16	8.42	284.2	62/67	63/67	24/28	38/39	39/39	334
slot@qwen3.5-4b-hmm-q8	slot (llama-server)	qwen3.5-4b-hmm	fine-tuned	q8	4.48	287.3	62/67	63/67	24/28	38/39	39/39	285
slot@qwen3.5-4b-hmm-q4	slot (llama-server)	qwen3.5-4b-hmm	fine-tuned	q4	2.71	303.1	62/67	63/67	25/28	37/39	38/39	293
pcdserver@bonsai-27b-q1	pcdServer	bonsai-27b	vanilla	q1	3.80	513.5	62/67	63/67	24/28	38/39	39/39	1,827
kev-4b-gguf	laya.cpp	kev-4b	dedicated			1,103.1	62/67	55/67	24/28	38/39	31/39
pcdserver@qwen3.5-2b-q3	pcdServer	qwen3.5-2b	vanilla	q3	1.22	42.7	61/67	62/67	23/28	38/39	39/39	203
dohnuts-metal@decider-0.8b-f16	dohnuts (Metal)	decider-0.8b	dedicated	f16	1.52	51.9	61/67	60/67	23/28	38/39	37/39
dohnuts-metal@decider-0.8b-q4	dohnuts (Metal)	decider-0.8b	dedicated	q4	0.53	53.3	61/67	59/67	23/28	38/39	36/39
pcdserver@qwen2.5-3b-q5	pcdServer	qwen2.5-3b	vanilla	q5	2.22	61.7	61/67	57/67	22/28	39/39	36/39	300
slot@decider-0.8b-f16	slot (llama-server)	decider-0.8b	dedicated	f16	1.52	63.6	61/67	60/67	23/28	38/39	37/39	670
slot@decider-0.8b-q4	slot (llama-server)	decider-0.8b	dedicated	q4	0.53	67.4	61/67	59/67	23/28	38/39	36/39	231
pcdserver@qwen3-4b-unsloth-bf16	pcdServer	qwen3-4b-unsloth	vanilla	bf16	8.05	71.5	61/67	62/67	23/28	38/39	39/39	259
pcdserver@qwen3-4b-q4	pcdServer	qwen3-4b	vanilla	q4	2.50	75.5	61/67	62/67	23/28	38/39	39/39	230
pcdserver@qwen3-4b-unsloth-q4	pcdServer	qwen3-4b-unsloth	vanilla	q4	2.50	76.4	61/67	61/67	23/28	38/39	38/39	247
pcdserver@qwen3.5-4b-supercoder-q4	pcdServer	qwen3.5-4b-supercoder	fine-tuned	q4	2.61	84.2	61/67	62/67	23/28	38/39	39/39	351
executorch@decider-0.8b-fp16	ExecuTorch (MLX)	decider-0.8b	dedicated	fp16		89.7	61/67	60/67	23/28	38/39	37/39
pcdserver@qwen3.5-4b-unsloth-iq3	pcdServer	qwen3.5-4b-unsloth	vanilla	iq3	1.95	90.2	61/67	62/67	23/28	38/39	39/39	800
pcdserver@qwen3.5-4b-unsloth-q2	pcdServer	qwen3.5-4b-unsloth	vanilla	q2	1.94	93.5	61/67	61/67	22/28	39/39	39/39	829
pcdserver@qwen3.5-4b-unsloth-q4	pcdServer	qwen3.5-4b-unsloth	vanilla	q4	2.74	93.6	61/67	62/67	23/28	38/39	39/39	297
pcdserver@qwen2.5-coder-7b-q4	pcdServer	qwen2.5-coder-7b	fine-tuned	q4	4.68	108.9	61/67	62/67	23/28	38/39	39/39	386
pcdserver@qwen3-8b-q4	pcdServer	qwen3-8b	vanilla	q4	5.03	116.0	61/67	62/67	23/28	38/39	39/39	367
pcdserver@qwen3.5-9b-unsloth-q4	pcdServer	qwen3.5-9b-unsloth	vanilla	q4	5.68	136.9	61/67	63/67	24/28	37/39	39/39	452
pcdserver@decider-2b-f16	pcdServer	decider-2b	dedicated	f16	3.78	42.2	60/67	60/67	22/28	38/39	38/39	166
pcdserver@qwen2.5-coder-3b-q4	pcdServer	qwen2.5-coder-3b	fine-tuned	q4	2.10	60.2	60/67	59/67	24/28	36/39	35/39	295
pcdserver@qwen3-4b-unsloth-q8	pcdServer	qwen3-4b-unsloth	vanilla	q8	4.28	70.3	60/67	62/67	23/28	37/39	39/39	306
pcdserver@qwen3-4b-unsloth-q3	pcdServer	qwen3-4b-unsloth	vanilla	q3	2.08	78.3	60/67	61/67	22/28	38/39	39/39	706
pcdserver@qwen3-4b-unsloth-q6	pcdServer	qwen3-4b-unsloth	vanilla	q6	3.31	79.3	60/67	61/67	22/28	38/39	39/39	276
pcdserver@qwen3-4b-unsloth-q5	pcdServer	qwen3-4b-unsloth	vanilla	q5	2.89	79.4	60/67	62/67	23/28	37/39	39/39	321
pcdserver@qwen3.5-4b-unsloth-q3	pcdServer	qwen3.5-4b-unsloth	vanilla	q3	2.29	92.7	60/67	61/67	22/28	38/39	39/39	358
pcdserver@decider-2b-dreamblooms-q8	pcdServer	decider-2b-dreamblooms	dedicated	q8	2.01	40.1	59/67	60/67	22/28	37/39	38/39	198
pcdserver@qwen3-4b-unsloth-iq2	pcdServer	qwen3-4b-unsloth	vanilla	iq2	1.53	72.7	59/67	57/67	21/28	38/39	36/39	605
dohnuts-metal@decider-2b-f16	dohnuts (Metal)	decider-2b	dedicated	f16	3.78	100.3	59/67	62/67	24/28	35/39	38/39
dohnuts-metal@decider-2b-dreamblooms-q8	dohnuts (Metal)	decider-2b-dreamblooms	dedicated	q8	2.01	103.7	59/67	62/67	24/28	35/39	38/39
slot@decider-2b-f16	slot (llama-server)	decider-2b	dedicated	f16	3.78	112.4	59/67	62/67	24/28	35/39	38/39	153
slot@decider-2b-dreamblooms-q8	slot (llama-server)	decider-2b-dreamblooms	dedicated	q8	2.01	114.4	59/67	62/67	24/28	35/39	38/39	199
executorch@decider-2b-fp16	ExecuTorch (MLX)	decider-2b	dedicated	fp16		251.3	59/67	62/67	24/28	35/39	38/39
pcdserver@decider-0.8b-q5	pcdServer	decider-0.8b	dedicated	q5	0.58	29.9	57/67	57/67	22/28	36/39	35/39	180
pcdserver@qwen3.5-2b-q8	pcdServer	qwen3.5-2b	vanilla	q8	2.08	39.5	57/67	59/67	20/28	38/39	39/39	189
pcdserver@qwen3.5-2b-unsloth-q8	pcdServer	qwen3.5-2b-unsloth	vanilla	q8	2.01	39.5	57/67	59/67	20/28	38/39	39/39	126
pcdserver@decider-0.8b-q4	pcdServer	decider-0.8b	dedicated	q4	0.53	40.2	57/67	57/67	22/28	36/39	35/39	583
pcdserver@decider-2b-iq4	pcdServer	decider-2b	dedicated	iq4	1.24	41.4	57/67	57/67	21/28	37/39	37/39	167
pcdserver@qwen3.5-2b-unsloth-bf16	pcdServer	qwen3.5-2b-unsloth	vanilla	bf16	3.78	42.2	57/67	59/67	20/28	38/39	39/39	328
pcdserver@qwen3.5-2b-unsloth-q4	pcdServer	qwen3.5-2b-unsloth	vanilla	q4	1.28	44.8	57/67	57/67	19/28	39/39	38/39	139
dohnuts-metal@decider-0.8b-q3	dohnuts (Metal)	decider-0.8b	dedicated	q3	0.47	54.5	57/67	55/67	24/28	34/39	31/39
pcdserver@qwen2.5-3b-q3	pcdServer	qwen2.5-3b	vanilla	q3	1.59	60.6	57/67	51/67	20/28	38/39	31/39	672
slot@decider-0.8b-q3	slot (llama-server)	decider-0.8b	dedicated	q3	0.47	69.7	57/67	55/67	24/28	34/39	31/39	101
kev-0.8b-gguf	laya.cpp	kev-0.8b	dedicated			231.5	57/67	45/67	24/28	34/39	21/39
pcdserver@qwen3.5-0.8b-q5	pcdServer	qwen3.5-0.8b	vanilla	q5	0.65	32.1	57/67	57/67	21/28	36/39	37/39	284
pcdserver@qwen3-1.7b-unsloth-q8	pcdServer	qwen3-1.7b-unsloth	vanilla	q8	1.83	35.1	57/67	59/67	21/28	36/39	38/39	95
pcdserver@qwen3-1.7b-q8	pcdServer	qwen3-1.7b	vanilla	q8	2.17	35.2	57/67	59/67	21/28	36/39	38/39	94
pcdserver@qwen3-1.7b-bf16	pcdServer	qwen3-1.7b	vanilla	bf16	4.07	36.2	57/67	59/67	21/28	36/39	38/39	206
pcdserver@qwen3-1.7b-q5	pcdServer	qwen3-1.7b	vanilla	q5	1.47	38.7	57/67	59/67	21/28	36/39	38/39	148
pcdserver@qwen3-1.7b-unsloth-q5	pcdServer	qwen3-1.7b-unsloth	vanilla	q5	1.26	38.7	57/67	59/67	21/28	36/39	38/39	170
pcdserver@qwen3-1.7b-unsloth-q6	pcdServer	qwen3-1.7b-unsloth	vanilla	q6	1.42	39.2	57/67	59/67	21/28	36/39	38/39	158
pcdserver@qwen3-1.7b-q6	pcdServer	qwen3-1.7b	vanilla	q6	1.67	39.3	57/67	59/67	21/28	36/39	38/39	150
pcdserver@qwen3.5-2b-unsloth-iq3	pcdServer	qwen3.5-2b-unsloth	vanilla	iq3	0.93	40.2	57/67	57/67	21/28	36/39	37/39	435
pcdserver@qwen3.5-2b-unsloth-q3	pcdServer	qwen3.5-2b-unsloth	vanilla	q3	1.11	41.9	57/67	59/67	20/28	37/39	39/39	165
pcdserver@qwen3.5-2b-q5	pcdServer	qwen3.5-2b	vanilla	q5	1.57	42.3	57/67	57/67	19/28	38/39	39/39	196
pcdserver@qwen3.5-2b-q6	pcdServer	qwen3.5-2b	vanilla	q6	1.70	42.6	57/67	57/67	19/28	38/39	39/39	134
pcdserver@qwen3.5-2b-unsloth-q5	pcdServer	qwen3.5-2b-unsloth	vanilla	q5	1.44	43.5	57/67	57/67	19/28	38/39	38/39	142
pcdserver@qwen3.5-2b-q4	pcdServer	qwen3.5-2b	vanilla	q4	1.40	47.2	57/67	57/67	19/28	38/39	39/39	217
pcdserver@qwen3.5-0.8b-unsloth-q5	pcdServer	qwen3.5-0.8b-unsloth	vanilla	q5	0.59	29.6	56/67	57/67	21/28	35/39	36/39	184
pcdserver@qwen3-1.7b-unsloth-q4	pcdServer	qwen3-1.7b-unsloth	vanilla	q4	1.11	40.2	56/67	57/67	20/28	36/39	38/39	194
pcdserver@qwen3.5-2b-unsloth-q6	pcdServer	qwen3.5-2b-unsloth	vanilla	q6	1.57	43.8	56/67	57/67	19/28	37/39	39/39	132
dohnuts-metal@decider-2b-iq4	dohnuts (Metal)	decider-2b	dedicated	iq4	1.24	106.7	56/67	57/67	23/28	33/39	35/39
slot@decider-2b-iq4	slot (llama-server)	decider-2b	dedicated	iq4	1.24	126.8	56/67	57/67	23/28	33/39	35/39	719
pcdserver@decider-0.8b-dreamblooms-q8	pcdServer	decider-0.8b-dreamblooms	dedicated	q8	0.81	28.3	55/67	57/67	20/28	35/39	37/39	78
pcdserver@qwen3.5-0.8b-unsloth-q8	pcdServer	qwen3.5-0.8b-unsloth	vanilla	q8	0.81	28.7	55/67	57/67	20/28	35/39	37/39	177
pcdserver@qwen3.5-0.8b-q6	pcdServer	qwen3.5-0.8b	vanilla	q6	0.69	29.0	55/67	57/67	20/28	35/39	37/39	450
pcdserver@qwen3.5-0.8b-unsloth-q6	pcdServer	qwen3.5-0.8b-unsloth	vanilla	q6	0.64	29.2	55/67	57/67	20/28	35/39	37/39	186
pcdserver@decider-0.8b-q6	pcdServer	decider-0.8b	dedicated	q6	0.63	29.8	55/67	57/67	20/28	35/39	37/39	280
pcdserver@qwen3.5-0.8b-bf16	pcdServer	qwen3.5-0.8b	vanilla	bf16	1.56	30.0	55/67	57/67	20/28	35/39	37/39	407
pcdserver@decider-0.8b-q8	pcdServer	decider-0.8b	dedicated	q8	0.81	30.2	55/67	57/67	20/28	35/39	37/39	208
pcdserver@qwen3.5-0.8b-ggml-bf16	pcdServer	qwen3.5-0.8b-ggml	vanilla	bf16	1.56	30.3	55/67	57/67	20/28	35/39	37/39	337
pcdserver@decider-0.8b-f16	pcdServer	decider-0.8b	dedicated	f16	1.52	30.7	55/67	56/67	19/28	36/39	37/39	438
pcdserver@qwen3.5-0.8b-unsloth-bf16	pcdServer	qwen3.5-0.8b-unsloth	vanilla	bf16	1.52	31.0	55/67	57/67	20/28	35/39	37/39	208
pcdserver@qwen3.5-0.8b-q8	pcdServer	qwen3.5-0.8b	vanilla	q8	0.83	31.2	55/67	57/67	20/28	35/39	37/39	252
pcdserver@qwen3.5-0.8b-unsloth-q4	pcdServer	qwen3.5-0.8b-unsloth	vanilla	q4	0.53	50.1	55/67	54/67	20/28	35/39	34/39	255
pcdserver@pulse-0.6b-q4	pcdServer	pulse-0.6b	fine-tuned	q4	0.40	27.9	54/67	52/67	21/28	33/39	31/39	196
pcdserver@bonsai-1.7b-q2	pcdServer	bonsai-1.7b	vanilla	q2	0.49	34.9	54/67	55/67	19/28	35/39	36/39	259
pcdserver@qwen3-1.7b-q2	pcdServer	qwen3-1.7b	vanilla	q2	0.88	37.5	54/67	50/67	21/28	33/39	29/39	104
pcdserver@qwen3-1.7b-q4	pcdServer	qwen3-1.7b	vanilla	q4	1.28	40.7	54/67	57/67	20/28	34/39	38/39	343
pcdserver@qwen3-0.6b-q8	pcdServer	qwen3-0.6b	vanilla	q8	0.64	25.5	53/67	54/67	18/28	35/39	36/39	191
pcdserver@qwen3.5-0.8b-ggml-q4	pcdServer	qwen3.5-0.8b-ggml	vanilla	q4	0.56	28.8	53/67	55/67	19/28	34/39	36/39	801
pcdserver@qwen3-1.7b-unsloth-q3	pcdServer	qwen3-1.7b-unsloth	vanilla	q3	0.94	37.6	53/67	56/67	18/28	35/39	38/39	216
pcdserver@qwen3.5-2b-unsloth-q2	pcdServer	qwen3.5-2b-unsloth	vanilla	q2	0.97	41.3	53/67	48/67	19/28	34/39	29/39	255
pcdserver@qwen2.5-3b-q2	pcdServer	qwen2.5-3b	vanilla	q2	1.27	58.3	53/67	46/67	17/28	36/39	29/39	271
laya-mlx	MLX	laya-multilingual	dedicated			6.9	52/67	49/67	21/28	31/39	28/39	1,797
laya-coreai	Core AI	laya-multilingual	dedicated			10.0	52/67	49/67	21/28	31/39	28/39	3,164
laya-coreai-mlb	Core AI	laya-multilingual	dedicated			19.4	52/67	49/67	21/28	31/39	28/39	2,404
laya-ml-onnx-fp32	ONNX Runtime	laya-multilingual	dedicated	fp32		23.4	52/67	49/67	21/28	31/39	28/39	825
laya-ml-onnx-int8	ONNX Runtime	laya-multilingual	dedicated	int8		25.5	52/67	49/67	21/28	31/39	28/39	741
pcdserver@decider-0.8b-q3	pcdServer	decider-0.8b	dedicated	q3	0.47	29.3	52/67	47/67	18/28	34/39	29/39	161
pcdserver@qwen3.5-0.8b-q4	pcdServer	qwen3.5-0.8b	vanilla	q4	0.58	30.2	52/67	51/67	17/28	35/39	34/39	258
laya-wd-gguf-q8	llama.cpp embeddings	laya-multilingual	dedicated	q8		44.4	52/67	49/67	21/28	31/39	28/39	2,337
laya-wd-gguf-f16	llama.cpp embeddings	laya-multilingual	dedicated	f16		44.4	52/67	49/67	21/28	31/39	28/39	2,437
laya-ml-gguf	laya.cpp	laya-multilingual	dedicated			57.6	52/67	49/67	21/28	31/39	28/39
laya-ml-gguf-f16	laya.cpp	laya-multilingual	dedicated	f16		61.4	52/67	49/67	21/28	31/39	28/39
pcdserver@qwen3.5-4b-unsloth-iq2	pcdServer	qwen3.5-4b-unsloth	vanilla	iq2	1.52	87.0	52/67	54/67	16/28	36/39	38/39	659
laya-wd-gguf-q6	llama.cpp embeddings	laya-multilingual	dedicated	q6		44.5	51/67	47/67	20/28	31/39	27/39	2,276
laya-ml-gguf-q4	laya.cpp	laya-multilingual	dedicated	q4		55.0	51/67	47/67	18/28	33/39	29/39
pcdserver@pulse-0.6b-q8	pcdServer	pulse-0.6b	fine-tuned	q8	0.64	23.2	50/67	47/67	20/28	30/39	27/39	56
pcdserver@lq-decide-1.7b-q8	pcdServer	lq-decide-1.7b	fine-tuned	q8	1.83	35.1	50/67	46/67	20/28	30/39	26/39	116
pcdserver@qwen3-1.7b-q3	pcdServer	qwen3-1.7b	vanilla	q3	1.07	37.7	50/67	53/67	16/28	34/39	37/39	109
pcdserver@qwen3.5-2b-q2	pcdServer	qwen3.5-2b	vanilla	q2	1.07	41.5	49/67	49/67	19/28	30/39	30/39	127
pcdserver@qwen3.8-2b-distill-q4	pcdServer	qwen3.8-2b-distill	fine-tuned	q4	1.31	42.9	49/67	51/67	19/28	30/39	32/39	154
laya-wd-gguf-q3	llama.cpp embeddings	laya-multilingual	dedicated	q3		45.3	49/67	43/67	17/28	32/39	26/39	3,284
pcdserver@qwen2.5-3b-iq2	pcdServer	qwen2.5-3b	vanilla	iq2	1.14	55.5	49/67	49/67	16/28	33/39	33/39	427
laya-coreml-ane-w8-compact	Core ML, Neural Engine	laya-multilingual	dedicated	w8		4.5	48/67	49/67	17/28	31/39	32/39	21,052
laya-typed-decisions	PyTorch	laya-typed-decisions	dedicated			17.7	48/67	43/67	18/28	30/39	25/39	4,793
pcdserver@qwen3.5-0.8b-unsloth-q3	pcdServer	qwen3.5-0.8b-unsloth	vanilla	q3	0.47	30.1	48/67	48/67	17/28	31/39	31/39	186
laya-coreml-ane-compact	Core ML, Neural Engine	laya-multilingual	dedicated			4.2	47/67	49/67	17/28	30/39	32/39	20,130
laya-mlx-compact	MLX	laya-multilingual	dedicated			7.9	47/67	49/67	17/28	30/39	32/39	11,010
pcdserver@bonsai-1.7b-q1	pcdServer	bonsai-1.7b	vanilla	q1	0.25	34.9	47/67	41/67	14/28	33/39	27/39	243
laya-wd-gguf-q5	llama.cpp embeddings	laya-multilingual	dedicated	q5		45.5	47/67	43/67	17/28	30/39	26/39	2,350
pcdserver@qwen3.5-0.8b-q3	pcdServer	qwen3.5-0.8b	vanilla	q3	0.52	31.3	46/67	40/67	15/28	31/39	25/39	218
pcdserver@qwen3.8-2b-distill-q5	pcdServer	qwen3.8-2b-distill	fine-tuned	q5	1.45	46.1	45/67	45/67	17/28	28/39	28/39	140
pcdserver@qwen3-1.7b-unsloth-q2	pcdServer	qwen3-1.7b-unsloth	vanilla	q2	0.78	37.4	44/67	46/67	18/28	26/39	28/39	276
pcdserver@qwen3.8-2b-distill-q8	pcdServer	qwen3.8-2b-distill	fine-tuned	q8	2.08	39.5	44/67	45/67	17/28	27/39	28/39	129
pcdserver@qwen3.5-2b-unsloth-iq2	pcdServer	qwen3.5-2b-unsloth	vanilla	iq2	0.77	40.8	44/67	39/67	15/28	29/39	24/39	129
pcdserver@qwen3.5-0.8b-unsloth-q2	pcdServer	qwen3.5-0.8b-unsloth	vanilla	q2	0.42	28.5	43/67	42/67	14/28	29/39	28/39	764
pcdserver@qwen3.8-2b-distill-q6	pcdServer	qwen3.8-2b-distill	fine-tuned	q6	1.61	43.5	43/67	43/67	16/28	27/39	27/39	122
pcdserver@zerocoder-0.8b-q2	pcdServer	zerocoder-0.8b	fine-tuned	q2	0.38	34.7	42/67	36/67	15/28	27/39	21/39	339
pcdserver@qwen1.5-4b-q8	pcdServer	qwen1.5-4b	vanilla	q8	4.20	247.8	42/67	43/67	16/28	26/39	27/39	322
pcdserver@lq-decide-1.7b-q4	pcdServer	lq-decide-1.7b	fine-tuned	q4	1.11	43.8	41/67	42/67	15/28	26/39	27/39	108
pcdserver@qwen1.5-4b-q4	pcdServer	qwen1.5-4b	vanilla	q4	2.46	232.1	41/67	43/67	16/28	25/39	27/39	391
pcdserver@qwen1.5-4b-q2	pcdServer	qwen1.5-4b	vanilla	q2	1.62	233.0	41/67	42/67	16/28	25/39	26/39	444
pcdserver@qwen3-reranker-0.6b-q5	pcdServer	qwen3-reranker-0.6b	fine-tuned	q5	0.44	24.7	40/67	44/67	15/28	25/39	29/39	58
pcdserver@qwen3.5-0.8b-q2	pcdServer	qwen3.5-0.8b	vanilla	q2	0.46	32.2	40/67	42/67	13/28	27/39	29/39	541
pcdserver@qwen1.5-4b-q6	pcdServer	qwen1.5-4b	vanilla	q6	3.25	275.7	40/67	43/67	16/28	24/39	27/39	309
pcdserver@qwen1.5-4b-q3	pcdServer	qwen1.5-4b	vanilla	q3	2.03	237.2	39/67	43/67	16/28	23/39	27/39	441
pcdserver@qwen1.5-4b-q5	pcdServer	qwen1.5-4b	vanilla	q5	2.84	283.6	39/67	42/67	16/28	23/39	26/39	424
laya-wd-gguf-q4	llama.cpp embeddings	laya-multilingual	dedicated	q4		43.3	36/67	38/67	14/28	22/39	24/39	2,243
pcdserver@qwen1.5-0.5b-q3	pcdServer	qwen1.5-0.5b	vanilla	q3	0.35	21.2	35/67	32/67	11/28	24/39	21/39	248
pcdserver@lyco-router-0.6b-q4	pcdServer	lyco-router-0.6b	fine-tuned	q4	0.48	26.4	35/67	33/67	12/28	23/39	21/39	514
pcdserver@qwen3.5-0.8b-unsloth-iq3	pcdServer	qwen3.5-0.8b-unsloth	vanilla	iq3	0.40	28.2	34/67	31/67	12/28	22/39	19/39	246
pcdserver@qwen1.5-1.8b-q3	pcdServer	qwen1.5-1.8b	vanilla	q3	1.02	41.8	34/67	35/67	15/28	19/39	20/39	510
pcdserver@qwen3-reranker-0.6b-q6	pcdServer	qwen3-reranker-0.6b	fine-tuned	q6	0.49	24.5	33/67	37/67	13/28	20/39	24/39	61
pcdserver@qwen1.5-1.8b-q4	pcdServer	qwen1.5-1.8b	vanilla	q4	1.22	40.0	33/67	39/67	15/28	18/39	24/39	343
pcdserver@zerocoder-0.8b-q3	pcdServer	zerocoder-0.8b	fine-tuned	q3	0.45	34.6	32/67	30/67	11/28	21/39	19/39	291
pcdserver@qwen1.5-1.8b-q8	pcdServer	qwen1.5-1.8b	vanilla	q8	1.96	36.4	32/67	36/67	13/28	19/39	23/39	106
pcdserver@qwen1.5-1.8b-q5	pcdServer	qwen1.5-1.8b	vanilla	q5	1.38	41.3	32/67	35/67	13/28	19/39	22/39	327
pcdserver@qwen1.5-1.8b-q6	pcdServer	qwen1.5-1.8b	vanilla	q6	1.58	39.8	31/67	37/67	14/28	17/39	23/39	334
pcdserver@qwen1.5-0.5b-q6	pcdServer	qwen1.5-0.5b	vanilla	q6	0.51	22.6	30/67	28/67	8/28	22/39	21/39	197
pcdserver@qwen3-reranker-0.6b-q8	pcdServer	qwen3-reranker-0.6b	fine-tuned	q8	0.64	23.3	30/67	33/67	12/28	18/39	21/39	57
pcdserver@qwen3-reranker-0.6b-f16	pcdServer	qwen3-reranker-0.6b	fine-tuned	f16	1.20	23.8	30/67	32/67	12/28	18/39	20/39	53
pcdserver@qwen1.5-0.5b-q8	pcdServer	qwen1.5-0.5b	vanilla	q8	0.66	21.1	28/67	28/67	7/28	22/39	21/39	177
pcdserver@qwen1.5-0.5b-q5	pcdServer	qwen1.5-0.5b	vanilla	q5	0.46	21.7	28/67	28/67	7/28	22/39	22/39	238
pcdserver@qwen2.5-0.5b-q4	pcdServer	qwen2.5-0.5b	vanilla	q4	0.49	20.2	28/67	26/67	10/28	18/39	16/39	605
pcdserver@qwen3-reranker-0.6b-q4	pcdServer	qwen3-reranker-0.6b	fine-tuned	q4	0.40	29.2	28/67	34/67	10/28	18/39	24/39	339
pcdserver@qwen2.5-0.5b-pcb-q2	pcdServer	qwen2.5-0.5b-pcb	fine-tuned	q2	0.34	18.1	27/67	23/67	9/28	18/39	14/39	334
pcdserver@zerocoder-0.8b-f16	pcdServer	zerocoder-0.8b	fine-tuned	f16	1.64	33.0	27/67	23/67	12/28	15/39	11/39	288
pcdserver@zerocoder-0.8b-q4	pcdServer	zerocoder-0.8b	fine-tuned	q4	0.53	46.4	27/67	24/67	12/28	15/39	12/39	270
pcdserver@qwen2.5-0.5b-f16	pcdServer	qwen2.5-0.5b	vanilla	f16	1.27	17.8	26/67	25/67	9/28	17/39	16/39	158
pcdserver@qwen1.5-0.5b-q4	pcdServer	qwen1.5-0.5b	vanilla	q4	0.41	21.6	26/67	28/67	8/28	18/39	21/39	217
pcdserver@qwen3.5-0.8b-unsloth-iq2	pcdServer	qwen3.5-0.8b-unsloth	vanilla	iq2	0.34	28.2	26/67	23/67	9/28	17/39	14/39	312
pcdserver@qwen1.5-1.8b-q2	pcdServer	qwen1.5-1.8b	vanilla	q2	0.86	39.8	26/67	36/67	10/28	16/39	26/39	392
pcdserver@qwen2.5-0.5b-q8	pcdServer	qwen2.5-0.5b	vanilla	q8	0.68	17.8	25/67	24/67	8/28	17/39	16/39	40
pcdserver@zerocoder-0.8b-q8	pcdServer	zerocoder-0.8b	fine-tuned	q8	0.87	34.4	25/67	21/67	11/28	14/39	10/39	184
pcdserver@zerocoder-0.8b-q6	pcdServer	zerocoder-0.8b	fine-tuned	q6	0.68	35.3	24/67	21/67	11/28	13/39	10/39	182
pcdserver@qwen2.5-0.5b-q6	pcdServer	qwen2.5-0.5b	vanilla	q6	0.65	22.2	23/67	24/67	8/28	15/39	16/39	134
pcdserver@qwen2.5-0.5b-q5	pcdServer	qwen2.5-0.5b	vanilla	q5	0.52	18.9	22/67	22/67	7/28	15/39	15/39	219
pcdserver@qwen1.5-0.5b-q2	pcdServer	qwen1.5-0.5b	vanilla	q2	0.30	21.1	21/67	22/67	8/28	13/39	14/39	247
pcdserver@qwen3-reranker-0.6b-q3	pcdServer	qwen3-reranker-0.6b	fine-tuned	q3	0.35	24.2	21/67	21/67	8/28	13/39	13/39	89
pcdserver@qwen2.5-0.5b-pcb-q3	pcdServer	qwen2.5-0.5b-pcb	fine-tuned	q3	0.36	17.7	20/67	20/67	7/28	13/39	13/39	185
pcdserver@euaiact-0.5b-q4	pcdServer	euaiact-0.5b	fine-tuned	q4	0.40	23.8	20/67	20/67	8/28	12/39	12/39	697
pcdserver@qwen2.5-0.5b-pcb-f16	pcdServer	qwen2.5-0.5b-pcb	fine-tuned	f16	0.99	17.2	19/67	20/67	7/28	12/39	13/39	144
pcdserver@qwen2.5-0.5b-pcb-q6	pcdServer	qwen2.5-0.5b-pcb	fine-tuned	q6	0.51	17.5	19/67	20/67	7/28	12/39	13/39	79
pcdserver@qwen2.5-0.5b-pcb-q8	pcdServer	qwen2.5-0.5b-pcb	fine-tuned	q8	0.53	17.5	19/67	20/67	7/28	12/39	13/39	40
pcdserver@qwen2.5-0.5b-pcb-q5	pcdServer	qwen2.5-0.5b-pcb	fine-tuned	q5	0.42	17.6	19/67	19/67	7/28	12/39	12/39	259
pcdserver@qwen2.5-0.5b-q3	pcdServer	qwen2.5-0.5b	vanilla	q3	0.43	18.4	18/67	19/67	5/28	13/39	14/39	316
pcdserver@zerocoder-0.8b-q5	pcdServer	zerocoder-0.8b	fine-tuned	q5	0.60	36.2	18/67	17/67	8/28	10/39	9/39	216
dohnuts-metal@decider-0.8b-q2	dohnuts (Metal)	decider-0.8b	dedicated	q2	0.42	55.2	18/67	17/67	7/28	11/39	10/39
slot@decider-0.8b-q2	slot (llama-server)	decider-0.8b	dedicated	q2	0.42	64.7	18/67	17/67	7/28	11/39	10/39	108
pcdserver@qwen2.5-0.5b-pcb-q4	pcdServer	qwen2.5-0.5b-pcb	fine-tuned	q4	0.40	22.9	17/67	19/67	6/28	11/39	13/39	361
pcdserver@decider-0.8b-q2	pcdServer	decider-0.8b	dedicated	q2	0.42	29.3	17/67	17/67	4/28	13/39	13/39	577
pcdserver@qwen2.5-0.5b-q2	pcdServer	qwen2.5-0.5b	vanilla	q2	0.42	17.6	15/67	16/67	4/28	11/39	12/39	477
pcdserver@qwen3-4b-unsloth-iq1	pcdServer	qwen3-4b-unsloth	vanilla	iq1	1.08	72.5	13/67	14/67	6/28	7/39	8/39	594
pcdserver@lq-decide-0.6b-q8	pcdServer	lq-decide-0.6b	fine-tuned	q8	0.64	23.1	11/67	11/67	4/28	7/39	7/39	135
pcdserver@qwen3-reranker-0.6b-q2	pcdServer	qwen3-reranker-0.6b	fine-tuned	q2	0.30	23.9	10/67	14/67	5/28	5/39	9/39	241
pcdserver@lq-decide-0.6b-q4	pcdServer	lq-decide-0.6b	fine-tuned	q4	0.40	29.5	10/67	11/67	4/28	6/39	7/39	183

Some reading notes:

• English is the hard part. The ten deliberately ambiguous queries are nine English and one Polish, so most models lose more of the 28 English queries than of the 39 others. jev got 26 of 28 English and 39 of 39 others.
• Translation helps weak multilingual models and costs strong ones nothing. kev-0.8b scored 57 translated and 45 direct, Qwen2.5-3B at Q8 62 and 57, laya 52 and 49. The Hmm model scored one answer better without translation, 65 against 64, and so did the Qwen3.5-4B builds.
• Load times differ by three orders of magnitude. A GGUF on pcdServer or llama-server loaded in 40 ms to 1.8 seconds. Core ML models compiled for the Neural Engine took 20 seconds, and decider-4b in PyTorch 17 seconds.
• Quantization is flat until it falls off a cliff. Q4 to Q8 barely changes a score. Below Q3 scores drop fast: decider-0.8b at Q2 answered 17 or 18 of 67 on every engine, and Qwen3-4B at IQ1_S answered 13. Chapter 7 has the full sweeps.
• Two methods failed. A Core ML laya export fixed at 64 tokens and four options could not take a five-way question. The BF16 build of unsloth's Qwen3.5-4B was not on disk when its turn came. Neither appears in the tables.

Best method per engine and runtime

Most methods run on pcdServer, because it takes any chat GGUF: 156 of the 208. The other engines and runtimes ran only the models they were built for. The best method on each:

engine or runtime	methods	best method	translated / direct	ms/query
jev (hosted)	1	jev	65 / 65	466
pcdServer	156	pcdserver@qwen3.5-4b-hmm-q8	64 / 65	88
dohnuts (Metal)	12	dohnuts-metal@decider-35b-a3b-q4	64 / 64	266
slot (llama-server)	15	slot@decider-35b-a3b-q4	64 / 64	299
PyTorch	2	torch@decider-4b-bf16	63 / 65	348
dohnuts (CPU)	1	dohnuts@decider-0.8b-dreamblooms-q8	62 / 60	275
laya.cpp	5	kev-4b-gguf	62 / 55	1,103
ExecuTorch (MLX)	2	executorch@decider-0.8b-fp16	61 / 60	90
MLX	2	laya-mlx	52 / 49	7
Core AI	2	laya-coreai	52 / 49	10
ONNX Runtime	2	laya-ml-onnx-fp32	52 / 49	23
llama.cpp embeddings	6	laya-wd-gguf-q8	52 / 49	44
Core ML, Neural Engine	2	laya-coreml-ane-w8-compact	48 / 49	5

If you need an answer in under 10 ms on a Mac, only laya delivers it, at 52/67. If you can spend 40 to 55 ms, pcdServer with a 2B chat model or dohnuts with decider-0.8b reaches 61 or 62. Every method above 62 took at least 85 ms.

One set of weights, three scores

An engine is not a neutral container. The same GGUF file, read two ways, gives different answers:

Click a column header to sort.

Model	Quant	Engine	ms/query	Translated	Direct
decider-0.8b	f16	pcdServer	30.7	55/67	56/67
decider-0.8b	f16	dohnuts (Metal)	51.9	61/67	60/67
decider-0.8b	f16	slot (llama-server)	63.6	61/67	60/67
decider-0.8b	fp16	ExecuTorch (MLX)	89.7	61/67	60/67
decider-0.8b	q2	pcdServer	29.3	17/67	17/67
decider-0.8b	q2	dohnuts (Metal)	55.2	18/67	17/67
decider-0.8b	q2	slot (llama-server)	64.7	18/67	17/67
decider-0.8b	q3	pcdServer	29.3	52/67	47/67
decider-0.8b	q3	dohnuts (Metal)	54.5	57/67	55/67
decider-0.8b	q3	slot (llama-server)	69.7	57/67	55/67
decider-0.8b	q4	pcdServer	40.2	57/67	57/67
decider-0.8b	q4	dohnuts (Metal)	53.3	61/67	59/67
decider-0.8b	q4	slot (llama-server)	67.4	61/67	59/67
decider-0.8b	q5	pcdServer	29.9	57/67	57/67
decider-0.8b	q5	dohnuts (Metal)	55.2	62/67	60/67
decider-0.8b	q5	slot (llama-server)	67.9	62/67	60/67
decider-0.8b	q6	pcdServer	29.8	55/67	57/67
decider-0.8b	q6	dohnuts (Metal)	53.5	62/67	62/67
decider-0.8b	q6	slot (llama-server)	68.1	62/67	62/67
decider-0.8b	q8	pcdServer	30.2	55/67	57/67
decider-0.8b	q8	dohnuts (Metal)	51.4	62/67	61/67
decider-0.8b	q8	slot (llama-server)	64.0	62/67	61/67
decider-0.8b-dreamblooms	q8	pcdServer	28.3	55/67	57/67
decider-0.8b-dreamblooms	q8	dohnuts (Metal)	52.2	62/67	61/67
decider-0.8b-dreamblooms	q8	slot (llama-server)	65.1	62/67	61/67
decider-0.8b-dreamblooms	q8	dohnuts (CPU)	275.2	62/67	60/67
decider-2b	f16	pcdServer	42.2	60/67	60/67
decider-2b	f16	dohnuts (Metal)	100.3	59/67	62/67
decider-2b	f16	slot (llama-server)	112.4	59/67	62/67
decider-2b	fp16	ExecuTorch (MLX)	251.3	59/67	62/67
decider-2b	iq4	pcdServer	41.4	57/67	57/67
decider-2b	iq4	dohnuts (Metal)	106.7	56/67	57/67
decider-2b	iq4	slot (llama-server)	126.8	56/67	57/67
decider-2b-dreamblooms	q8	pcdServer	40.1	59/67	60/67
decider-2b-dreamblooms	q8	dohnuts (Metal)	103.7	59/67	62/67
decider-2b-dreamblooms	q8	slot (llama-server)	114.4	59/67	62/67
decider-35b-a3b	q4	dohnuts (Metal)	266.3	64/67	64/67
decider-35b-a3b	q4	pcdServer	296.8	63/67	63/67
decider-35b-a3b	q4	slot (llama-server)	299.1	64/67	64/67
decider-4b	bf16	PyTorch	348.2	63/67	65/67

decider-0.8b scores 62/67 on dohnuts and 62/67 on slot. Both read the model where it was trained to answer: the logits of the option letters after Answer: (, divided by the checkpoint's temperature. The agreement is exact: on all 106 benchmark texts the two picked the same task, and their probabilities differed by at most 0.0004. They differ only in speed. dohnuts (Metal) took 52 ms per query, slot on llama-server 65 ms, and dohnuts on the CPU with eight threads 275 ms.

pcdServer scored the same file 55/67 translated and 57/67 direct, at 28 ms. pcdServer does not know the decider layout. It wraps the query in the model's chat template, lists the fields in a system prompt, and scores the first token of each allowed value. decider was never trained to answer that way, so seven answers go missing, although pcdServer is the fastest engine on this model.

The larger deciders narrow the gap. decider-2b's Q8_0 and F16 files scored 59 or 60 translated on every engine that ran them (the IQ4_NL file 56 or 57). On direct text they scored 62 through the readout and 60 through pcdServer. The 35B model scored 64/64 through its readout and 63/63 through pcdServer.

The same weights also ran in Mapika's own runtimes: ExecuTorch on MLX scored decider-0.8b at 61/60 in 90 ms, and decider-2b at 59/62 in 251 ms.

Reading a dedicated model through its own readout is worth several answers on small models. For a fine-tuned or vanilla chat model, pcdServer's chat-template readout is the one that works: the Hmm model scored 64/65 there and 62/63 through its letter readout on llama-server, and in a third of the time.

What the results do not say

A five-way router over 67 queries decides nothing about your task. A model that is one query behind another here is not measurably worse, and the ranking between models within two or three queries of each other would change with a different set of 67. The results are clear about the large gaps: dedicated or modern models against old ones, sensible quantizations against 1- and 2-bit ones, and a trained readout against a foreign one. Chapter 12 turns those gaps into recommendations. Chapter 8 explains where the milliseconds go.