A quantization study of QVAC's VisionPsy-Nano-460M
VisionPsy-Nano, Re-Quantized.
A 460M-parameter on-device vision-language model, with SigLIP2 eyes, a SmolLM2-360M brain, and one small projector between them, re-quantized with calibration data that actually looks like its job. Same weights, same speed, better answers, measured on QVAC's full 17-benchmark suite with two labeled rulers, one of them QVAC's own.
// bits are cheap. well-placed bits are everything.
faster than every tested QVAC configuration DomCal-EmbGuard · 8 wins / 5 losses / 4 ties vs q4_k_m-imat · 786.7 ms/itemsmallest package measured DomCal-Slim · 325.9 MiB · below every QVAC packagePareto flagship TriStack · 9 wins / 7 losses / 1 tie vs q4_k_m-imat · 1.4% smaller
01 · The base model
A tiny VLM with a very important bridge
QVAC's VisionPsy-Nano-460M is built to run where the cloud isn't: an image is cut into patches, encoded by a SigLIP2 vision encoder, translated into language-model territory by a small multimodal projector (the mmproj), and read by a SmolLM2-360M backbone that writes the answer. About 460 million parameters, meant for a laptop. Watch the violet box in the middle. It is small, it is shipped at Q8 precision, and it becomes the subject of the projector-compression technique below.
The reference bar: QVAC q4_k_m-imat (Q4_K_M 4-bit + imatrix, 392.8 MiB with the Q8 mmproj). QVAC ships 11 GGUF configurations; the one used as the comparison reference throughout this site is their q4_k_m-imat, and the choice is QVAC's own words before it is this study's: their GGUF card calls it the "best size/quality trade-off (most users)" and the sweet spot for mobile/laptop. On their published numbers it is the pareto knee of the lineup: Normalized 61.51 vs 61.75 for both the FP32 and Q8_0 references, i.e. within 0.24 pts (−0.39%) of lossless, while the LM file is 69% the size of Q8_0 and 18.5% of FP32. Below it, quality falls faster than size (IQ4_XS-imat gives back −0.68 pts for another −16% weights); above it, Q5_K_M through FP32 buy back at most 0.24 pts for up to 5.4× the bytes; and the legacy Q4_0 (−2.85 pts, kept by QVAC only for latency cross-checks) shows what 4-bit costs without importance-matrix calibration. Setting the bar at the strongest, most deployment-versatile point of their quantization range (not the weakest) is what keeps the comparisons below conservative: clearing q4_k_m-imat means clearing the build QVAC itself tells users to install.
Diagram not to scale; the projector is drawn large on purpose. It is the smallest piece of the stack and, for a quantization study, the most interesting: it is the only component that ships at Q8 while everything around it goes 4-bit.
02 · The problem
The default download is the weakest quant
QVAC publishes several quants of this model, but the one quoted for on-device latency is q4_0, built with no importance matrix at all. No calibration means every weight is treated as equally disposable, and the quality cost is measurable. Domain calibration recovers most of that MMStar gap in the same q4_0 format; protecting the sensitive embeddings then carries this work's two flagship packages beyond QVAC's generic-corpus q4_k_m-imat, without giving up deployment-class speed.
MMStar · strict accuracy (%) · same harness
QVAC q4_0no imatrix · the latency-quoted build
+3.5 pt recovered at the same format →
42.3
DomCalVisionPsy-Nano DomCal · same q4_0 format, domain-calibrated imatrix
Canonical same-harness MMStar records, strict and judge-free. QVAC q4_0 trails q4_k_m-imat by 4.3 points; DomCal recovers 3.5 points without changing the q4_0 format. The stacked variants go further: TriStack scores 46.9 and DomCal-EmbGuard 47.1, beating q4_k_m-imat's 46.6 by 0.3 and 0.5 points. The fp32 value is the same-harness tier-A reference.
pooled strict vs q4_0 +2.2 pt · p = 4.7×10⁻⁵ (paired McNemar)pooled judged +2.5 pt · p = 0.0088tier-A paired MMStar screen +3.9 vs q4_0 · p = 0.0004OCRBench strict +2.4 · p = 0.007ScienceQA +0.9 · p = 0.31latency 802 vs 804 ms/item; speed-neutral
03 · The three techniques
Three ways to spend 4 bits more wisely
Everything on this page comes from three composable ideas. One changes what the quantizer looks at, one changes what it's allowed to touch, and one changes how many bits the projector carries.
calibration sweep builds the importance mapsalient weights, protectedeveryone else, 4-bit
Domain calibration
Domain-calibrated imatrix quantization
An importance matrix tells the quantizer which weights the model actually uses, measured by running calibration data through it. QVAC's calibrated builds use a generic corpus. The calibration here used the model's actual job instead: VQA-domain data: VQAv2-train + TextVQA-train, ChatML-formatted, train splits only, never benchmark test items. The sweep above is that importance map being built; the survivors keep their precision, the rest go 4-bit.
pooled strict +2.2 pt · p = 4.7×10⁻⁵ties QVAC's best 4-bit, judged 761 vs 777 · p = 0.12 nsat ~12% smaller LM size
Leakage control: verbatim question overlap with the calibration corpus is 0% (ScienceQA), 0.6% (MMStar), 8.3% (OCRBench; template questions, different images). Re-scored without the 83 overlapping OCRBench items: 614 → 637 (+23), matching the full-set +24. The improvement is genuine, not memorized.
Embedding protection
Embedding-protected quantization
Naive quantization treats every tensor the same, including the input embedding table, the place where the model keeps its vocabulary. Embedding protection guards that single tensor at Q8_0 and lets the rest of the network go 4-bit. Cost: +22 MB. A model that misremembers its dictionary misspells its thoughts; for +22 MB, it keeps the dictionary.
ScienceQA +1.59 pt · p = 0.00097in the DomCal-EmbGuard stack: tier-A MMStar 47.5best MMStar measured anywhere in the study; above fp32 (45.9)
Projector compression
mmproj compression, q8 → q6_K
As flagged in §01, the projector ships at Q8, accounting for over a quarter of the small packages' total size, spent on a component that turns out not to need it. Compressing it to q6_K saves 21.5% of the projector, and the tier-A evidence says the answers don't move: parity, p = 0.49. That is 23 MB reclaimed at no measured quality cost.
projector −21.5% · 108,782,144 → 85,355,072 Bquality parity p = 0.49 (tier-A)enables the smallest package of the study: 325.9 MiB
04 · The four models
Four release variants, pick your size
The three techniques stack. Each variant below is a hash-pinned GGUF artifact (never overwritten), paired with either QVAC's original Q8 projector or this work's q6_K one. These are this work's builds; QVAC's artifacts keep their plain names. On speed: the fastest of the four (and of all fifteen configurations measured, every QVAC quantization included) is DomCal-EmbGuard at 786.7 ms/item; this work's builds hold the three fastest slots (786.7 / 795.4 / 802.2), ahead of QVAC's best (q4_0, 803.5).
VisionPsy-Nano DomCal
DomCal
domain calibration
348.2 MiBLM 256,379,776 B (q4_0 domcal) + mmproj 108,782,144 B (QVAC q8)
Complete 17-row record · MMStar 45.8 strict · 12 wins / 4 losses / 1 tie descriptively versus QVAC q4_0, with 9 wins / 4 losses / 1 tie on the non-selection rows · normalized score 60.31 · speed-neutral (802 ms/item; 3rd fastest of the 15 measured)
409.7 MiBLM 320,862,016 B (q4_k_m domcal+embguard) + mmproj 108,782,144 B (QVAC q8)
Tier-B: MMStar 47.1 strict; highest measured quality · same-harness displayed-row count 8 wins / 5 losses / 4 ties vs QVAC q4_k_m-imat · tier-A MMStar 47.5, best measured (above fp32 45.9) · fastest configuration of the study; 786.7 ms/item ⚡, ahead of all 11 QVAC quantizations
full-17 scored · strict + both judged rulers
smallest
VisionPsy-Nano DomCal-Slim
DomCal-Slim
domain calibration×Q6 projector
325.9 MiBLM 256,379,776 B (q4_0 domcal) + mmproj 85,355,072 B (q6_K)
Smallest package of the study: 325.9 MiB · 2.5% smaller than QVAC's smallest package, iq3_xxs-imat (334.2 MiB) · q6_K projector at parity with QVAC's q8 original (p = 0.49, tier-A) · speed-neutral vs the contender (817.5 vs 813.2 ms/item, +0.5%)
387.4 MiBLM 320,862,016 B (q4_k_m domcal+embguard) + mmproj 85,355,072 B (q6_K)
Tier-B: MMStar 46.9 strict · same-harness displayed-row count 9 wins / 7 losses / 1 tie vs the contender q4_k_m-imat at −1.4% package; wins land on reasoning & knowledge (SciQA 85.1, highest measured · MathVista 47.8 vs 47.1 · MMVet 33.5 vs 32.8 · MMMU 31.3 vs 29.3) · MM-IFEval @2048 41.5 vs 41.1 · full-suite normalized score 61.64 vs DomCal-EmbGuard's 61.97 at −5.4% package · 795.4 ms/item; 2nd fastest measured, faster than every QVAC configuration
full-17 scored · strict + both judged rulers · on the Pareto frontier
Sizes are stat-verified bytes; totals in MiB (1 MiB = 1,048,576 B). Model names used from here on: DomCal = VisionPsy-Nano DomCal · DomCal-EmbGuard = VisionPsy-Nano DomCal-EmbGuard · DomCal-Slim = VisionPsy-Nano DomCal-Slim · TriStack = VisionPsy-Nano TriStack · imat = QVAC q4_k_m-imat.
The full lineup, by package size
QVAC ships eleven GGUFs of this model; four more were built in this work. Every package below is ordered by total size (LM + mmproj), with this work's same-harness scores where they exist and N/A where they don't yet.
package
LM MiB
mmproj MiB
total MiB
MMStar
SciQA
OCRBench
notes
DomCal-Slim
244.5
81.4
325.9
45.4
81.9
ˢ718· 746 ʲ
smallest package of the study; undercuts every QVAC configuration
QVAC iq3_xxs-imat
230.4
103.7
334.2
—
—
—
size reference only
QVAC iq3_m-imat
239.9
103.7
343.6
46.0ᵃ
82.1ᵃ
ˢ703 · 755 ʲᵃ
tier-A trio scored
QVAC iq4_xs-imat
241.6
103.7
345.3
46.7ᵃ
83.3ᵃ
ˢ700 · 750 ʲᵃ
tier-A trio scored
QVAC q4_0
243.9
103.7
347.7
42.3
81.0
ˢ694 · 743 ʲ
the latency-quoted build; no calibration at all · full-17 scored here
QVAC iq4_nl-imat
243.9
103.7
347.7
—
—
—
size reference only
DomCal
244.5
103.7
348.2
45.8
81.9
ˢ716 · 753 ʲ
same size class as q4_0 (+0.6 MiB; the imatrix's cost)
TriStack
306.0
81.4
387.4
46.9
85.1
ˢ728 · 764 ʲ
flagship · full-17 scored · undercuts q4_k_m-imat by 5.4 MiB
QVAC q4_k_m-imat
289.1
103.7
392.8
46.6
84.7
ˢ739 · 777 ʲ
QVAC's calibrated 4-bit reference · full-17 scored here
DomCal-EmbGuard
306.0
103.7
409.7
47.1
84.7
ˢ728 · 766 ʲ
full-17 scored · top MMStar (47.1)
QVAC q5_k_m
310.4
103.7
414.1
46.2ᵃ
84.7ᵃ
ˢ728 · 774 ʲᵃ
tier-A trio scored
QVAC q5_k_m-imat
310.4
103.7
414.1
—
—
—
size reference only
QVAC q8_0
416.4
103.7
520.2
—
—
—
size reference only
QVAC bf16
782.2
103.7
885.9
—
—
—
speed reference · 849 ms/item solo (§7 shootout)
ᵃ tier-A trio campaign (MMStar / ScienceQA / OCRBench) · ᵇ tier-B full-17 campaign · ˢ strict judge-free · ʲ judged, default ruler. Unmarked full-17 rows (DomCal-Slim, QVAC q4_0, DomCal, TriStack, q4_k_m-imat, DomCal-EmbGuard) are tier-B. Tier-A cards for the two rows that joined tier-B today: QVAC q4_0 42.2 / 80.8 / 736 ʲᵃ · DomCal 46.1 / 81.7 / 761 ʲᵃ. Sizes are stat-verified bytes; totals = LM + mmproj (QVAC q8 = 103.7 MiB; this work's q6_K = 81.4 MiB).
fp32 footnote (reference only; now on the Pareto chart thanks to its log axis): LM 1,562.4 + 103.7 = 1,666.2 MiB; MMStar 45.9ᵃ · SciQA 84.3ᵃ · OCR ˢ721 · 783 ʲᵃ · 920 ms/item solo.
Speed audit: the full lineup, fastest first
Every quality claim on this page is speed-audited: same harness, same 200-item MMStar protocol, GPU strictly solo; a contention-flagged first pass was deleted and re-measured, and no shared-GPU number is ever cited. QVAC publishes no speed telemetry for these artifacts, so all fifteen rows below are this harness's measurements. In this solo sweep, the three fastest observed configurations are this work's builds: DomCal-EmbGuard 786.7 ms/item, TriStack 795.4, DomCal 802.2, ahead of QVAC's fastest observed configuration (q4_0, 803.5) and the contender q4_k_m-imat (813.2). The same-harness normalized-score leader is also the observed speed leader (−3.3% latency vs the contender); even the smallest package, DomCal-Slim (817.5), matches the contender within 0.5%; the quality gains are speed-neutral.
#
package
total MiB
ms/item
p95
peak VRAM MiB
notes
1
VisionPsy-Nano DomCal-EmbGuard
409.7
786.7 ⚡
1039.7
1349
same-harness normalized-score leader (8 wins / 5 losses / 4 ties vs the contender) and fastest observed
2
VisionPsy-Nano TriStack
387.4
795.4
1050.7
1327
flagship stack; faster than every QVAC configuration
3
VisionPsy-Nano DomCal
348.2
802.2
1068.9
1307
speed-neutral vs its q4_0 base (−0.2%)
4
QVAC q4_0
347.7
803.5
1066.2
1307
QVAC's latency-quoted build
5
QVAC iq3_m-imat
343.6
804.6
1070.6
1305
6
QVAC iq4_nl-imat
347.7
805.7
1066.1
1307
7
QVAC q8_0
520.2
808.7
1064.7
1457
8-bit, +49% size over 4-bit, no speed gain
8
QVAC q4_k_m-imat
392.8
813.2
1080.1
1349
the contender
9
QVAC iq3_xxs-imat
334.2
814.2
1084.0
1293
10
QVAC q5_k_m
414.1
815.7
1078.4
1367
11
VisionPsy-Nano DomCal-Slim
325.9
817.5
1083.9
1285
smallest package; +1.7% vs q4_0 is the q6_K projector's encode cost
12
QVAC q5_k_m-imat
414.1
823.3
1087.6
1367
13
QVAC iq4_xs-imat
345.3
835.4
1107.1
1305
slowest 4-bit
14
QVAC bf16
885.9
849.2
1127.8
1791
15
QVAC fp32
1666.2
919.8
1207.1
2475
+17% vs the 4-bit class
Protocol: MMStar first 200 items, max_new 32, end-to-end wall-clock per item (image encode + prefill + decode), q8 mmproj (this work's q6_K where noted), RTX 4060 Laptop 8 GB, seed 17, one job at a time. Decode tok/s and TTFT are not separable at this generation length (decode ≈ 2% of item time); direct telemetry from the tier-B full-17 records (53,584 items, max_new 128): QVAC q4_0 decode 373 tok/s median (10th percentile 334), TTFT 763 ms median (90th percentile 984). Spread across all 4-bit builds is ≤ 4% (786.7–835.4); format barely moves this prefill-dominated workload.
05 · Quality per mebibyte
Size against score, honestly plotted
Every package, every benchmark, one chart: total package size against score, for all 17 benchmarks; pick one and the points re-position. Marker shape and color say where a number comes from: amber discs are this work's builds scored in this harness, green squares are QVAC's same-harness comparisons, blue diamonds are QVAC's bf16/fp32 references scored in this harness (tier-A subset), violet diamonds are QVAC's published card (QVAC harness and judge configuration). Judged benchmarks plot the judged value (this work's default ruler where applicable); multiple-choice plots accuracy. Hover or focus any point for its provenance.
Choose a benchmarkSelect one to update the chart.
This work: this harnessQVAC: same-harness comparisonQVAC 16/32-bit refs: this harness (tier-A)QVAC: published card (all 11 GGUF configs)
ALL 11 QVAC GGUF CONFIGS · total package MiB (LM + mmproj), size order · ◆ violet = published card, all 17 benchmarks · ■ green = same-harness comparison · ◆ blue = also scored in this harness (tier-A)IQ3_XXS-i · 334.2IQ3_M-i · 343.6IQ4_XS-i · 345.3IQ4_NL-i · 347.7Q4_0 · 347.7Q4_K_M-i · 392.8Q5_K_M · 414.1Q5_K_M-i · 414.1Q8_0 · 520.2BF16 · 885.9FP32 · 1666.2
The honest read: DomCal versus QVAC q4_0 has 12 wins / 4 losses / 1 tie over all 17 descriptive rows and 9 wins / 4 losses / 1 tie over the 14 non-selection rows, at +0.6 MiB in the same q4_0 size class. QVAC q4_0 keeps POPE, RealWorldQA and MME; MMMU is tied. TriStack beats the q4_k_m-imat contender at a smaller package size on MMStar (46.9 at 387.4 MiB versus 46.6 at 392.8 MiB), while q4_k_m-imat keeps judged OCRBench (777). DomCal-EmbGuard leads same-harness MMStar at 47.1. Compare only within a marker class: published-card diamonds come from the QVAC harness and remain context, never same-harness medals. Every release-ledger package now has a complete 17-row record; MathVista uses the corrected aggregation and every MM-IFEval value uses the final 2048-token record.
All benchmark scores shown rounded to one decimal (round-half-up); where rounding produces a tie, models share 🥇. Full-precision values are preserved in the release evidence ledger (final_results.json and result CSVs).
06 · The measuring machine
The benchmark harness, rebuilt from scratch
QVAC's numbers come from VLMEvalKit + PR #1613: an adapter that loads the Hub checkpoint through transformers trust_remote_code (or an in-process vLLM plugin) and judges with a local FP8 model. That harness cannot load the artifact this project ships: a GGUF + mmproj pair for llama.cpp. So one was built here that can, from their own pinned components.
Their path · VLMEvalKit + PR #1613
HF checkpoint: safetensors via trust_remote_code, or visionpsy-vllm-plugin
✓ has it · ~ partial · N/A: absent. Rows where the QVAC harness scores higher stay in, on purpose: QVAC's model zoo and batched engine were traded for artifact fidelity, telemetry, and auditability on one 8 GB card.
Why build from scratch
The artifact is the GGUF. PR #1613 benchmarks the Hub checkpoint; it physically cannot score the file users download and run. Scoring this work's quants through the HF path would grade different weights than the ones it ships.
Their fork's server binary doesn't build (dead web-UI asset pin). So mtmd-batch → mtmd-metrics was written: a C++ batch driver on their own fork; load model + mmproj once, stream items, reset the KV cache between them.
Half the study is efficiency. VLMEvalKit records predictions only; no TTFT, no decode tok/s, no peak VRAM. Those axes are the point of a requantization study.
One 8 GB card, 14 GB RAM. Stock TSV loading peaked at 8.36 GB RSS on the 631 MB DocVQA file; 53,584 items × 6 packages on one GPU demands chunking, crash-resume, and an exclusive-GPU guard.
No judge fits in 8 GB. The judge moved to an API; so two labeled rulers are shipped instead of one (this work's + QVAC's exact serving config), validated against their published card in §07.
53,584items per model through the harness; 17 benchmarks, two judged rulers, ~10 h GPU each, on a single RTX 4060 laptop card. Same TSVs, same prompt builders, same scorers, same judge model as theirs: only the inference runtime and the judge transport were replaced; and the swap was then proven on QVAC's own published numbers.
07 · Why you can trust the numbers
The ruler was validated against theirs
Any benchmark win is only as good as the ruler it was measured with. So before anything was claimed, the measurement stack was checked against the only ground truth available: QVAC's own published model card.
STEP 01
Borrow their ruler
Run QVAC's own released models through this harness, judged with QVAC's exact serving config: thinking disabled, max_tokens = 16, temp 0.3, parse-fail = wrong.
STEP 02
Check it against their card
Eight judged numbers, published vs re-measured. 6 of 8 land within ±1 pt. The misses are small and expected: their judge ran locally FP8, this work's runs via OpenRouter Venice; different stack, no shared seed.
STEP 03
Then measure everything twice
Every judged number on this site ships with both rulers labeled: the QVAC ruler (validated against their card) and this work's default ruler (reasoning on, no token cap). Never mixed in one cell.
6/8benchmarks within ±1 pt of QVAC's published card; same artifacts, this harness, their judge config. Validation target was ±2 on OCRBench/MathVista: MathVista PASS (−0.5), OCRBench within −3.
benchmark · QVAC q4_k_m-imatpublished vs this work's qvac-rulerΔ
Worst |Δ| = 3 (OCRBench q4_k_m-imat) and 2.7 (MMVet). Residual noise is expected; different serving stack, no seed control. The bottom row repeats the check on QVAC q4_0's published OCRBench (725).
08 · Live scoreboard
Six packages, seventeen rows, one ruler
The full comparison on one board, columns ordered by package size. 🥇/🥈 mark the top two completed same-harness cells per row; medals only ever compare runs measured with the same ruler. All six packages now have complete 17-row records in the canonical ledger; MathVista uses the corrected aggregation and every MM-IFEval value uses the final 2048-token record.
This harness · judged = default ruler · medals among completed runs only
benchmark
DomCal-Slim325.9 MiB
QVAC q4_0347.7 MiB
DomCal348.2 MiB
TriStack387.4 MiB
QVAC q4_k_m-imat392.8 MiB
DomCal-EmbGuard409.7 MiB
MMStar
45.4
42.341.7ᵖ
45.8
🥈46.9
46.6
🥇47.1
MMBench
57.7
55.056.1ᵖ
58.0
🥈61.4
60.7
🥇61.5
ScienceQA
81.9
81.080.8ᵖ
81.9
🥇85.1
🥈84.7tie
🥈84.7tie
MMMU
🥇32.7tie
🥇32.7tie · 31.3ᵖ
🥇32.7tie
31.3
29.3
🥈32.0
POPE
87.3
87.787.4ᵖ
87.4
🥇87.9
87.6
🥈87.8
SEEDBench
68.4
67.767.7ᵖ
🥇68.6
🥈68.5
🥈68.5
🥈68.5
RealWorldQA
🥇60.7
60.460.3ᵖ
60.1
59.6
🥈60.5
59.6
AI2D
64.8
63.964.2ᵖ
64.9
🥈65.9
65.7
🥇66.0
MME (total)
1527.1
🥇1562.51553.3ᵖ
1512.9
1528.3
🥈1532.3
1530.3
OCRBench (judged /1000)
746(ˢ718)
743(ˢ694) · 725ᵖ
753(ˢ716)
764(ˢ728)
🥇777(ˢ739)
🥈766(ˢ728)
DocVQA (judged)
81.8(ˢ76.3)
81.4(ˢ75.4) · 80.8ᵖ
81.9(ˢ76.4)
83.2(ˢ78.0)
🥇83.6(ˢ78.3)
🥇83.6(ˢ78.2)
ChartQA (judged)
77.1(ˢ71.9)
76.1(ˢ73.2) · 75.4ᵖ
76.8(ˢ71.9)
77.2(ˢ74.0)
🥇77.6(ˢ74.6)
🥈77.4(ˢ74.6)
InfoVQA (judged)
45.4(ˢ38.4)
45.1(ˢ37.9) · 44.2ᵖ
45.7(ˢ38.8)
47.3
🥇47.7
🥈47.6
TextVQA (judged)
78.0(ˢ67.7)
77.6(ˢ67.1) · 76.7ᵖ
78.5(ˢ68.2)
79.6
🥇80.0
🥇80.0
MathVista (corrected)
44.3†
41.741.8ᵖ
43.9
🥈47.8
47.1
🥇48.6
MMVet
31.7
🥈33.830.6ᵖ
31.1
33.5
32.8
🥇35.6
MM-IFEval
38.8@2048
34.8@2048 · 34.3ᵖ
38.8@2048
🥈41.5@2048
41.1@2048
🥇42.2@2048
ᵃ tier-A subset run (MMStar / ScienceQA / OCRBench only) · ᵖ QVAC published card; QVAC harness, shown for context only, never medaled · ˢ strict judge-free value. Judged cells on this board are the default ruler; their QVAC-ruler twins live in the benchmark wall below.
Medals compare completed same-harness runs only. Every MM-IFEval cell on this board ran at QVAC's full 2048-token protocol: DomCal 38.8, DomCal-EmbGuard 42.2, TriStack 41.5, q4_k_m-imat 41.1, DomCal-Slim 38.8, and QVAC q4_0 34.8. The earlier max_new = 128 artifacts are superseded. MathVista uses the corrected aggregation. A full-17 run takes ~9–10 h per model on a single GPU, serialized through a queue so latency stays uncontended.
09 · The benchmark wall
All 17 benchmarks, both rulers, no cherry-picking
QVAC's full published suite, this harness, seed 17. This wall keeps the three-package dual-ruler comparison: DomCal-EmbGuard, TriStack, and QVAC q4_k_m-imat. Their available strict, default-ruler, and QVAC-ruler records are complete; MM-IFEval on the QVAC ruler remains N/A by construction. The best value in each comparable ruler group is shown in green, without a dot marker. All values below are tier-B full-17 unless labeled otherwise; judged columns never mix rulers.
The four capability areas: QVAC's grouping, this harness's numbers
QVAC's blog groups the 17 benchmarks into four capability areas, normalized 0–100 (MME*/2800, OCRBench*/1000; percentages otherwise), each area scored as the plain mean of its benchmarks. Same recipe, applied here: this work's builds plus the QVAC q4_0 and q4_k_m-imat same-harness comparisons use this harness, default ruler; ᵖ columns are QVAC's published card or blog (QVAC harness; context, never medaled); DomCal-Slim's judged cells all landed on the default ruler (2026-08-05); its MM-IFEval@2048 scored 38.8. Medals compare the completed same-harness set only.
Instruction Following & ReliabilityMM-IFEval · POPE
63.0
61.3
63.1
🥈 64.7
64.4
🥇 65.0
65.1
64.3
46.9
51.4
Area scores are means of normalized per-benchmark values (recipe verified against QVAC's published 73.9/61.5/52.2/65.1). The complete DomCal row is 71.6 / 60.2 / 50.0 / 63.1; its component records include corrected MathVista 43.9 and MM-IFEval@2048 38.8. QVAC q4_0's same-harness comparison is 70.9 / 59.7 / 49.2 / 61.3, so every build from this work is higher on all four descriptive area aggregates. Cohort columns (LFM2.5-VL / SmolVLM2 / nanoVLM) and the remaining ᵖ columns are QVAC's published figures from the QVAC harness and remain context only.
Capability areas at a glance: the four area scores, grouped by area, one bar per model
Filter the capability viewSelect one to update the comparison.
One point per config (17 plotted; switch the four capability areas with the buttons): ● this work's builds, ■ QVAC same-harness comparisons, ◆ QVAC published cards. DomCal is included with its complete 71.6 / 60.2 / 50.0 / 63.1 area row. The dashed line is contextual because it spans different rulers; same-harness claims remain separated in the tables. x-axis logarithmic; MME is normalized /2800 and OCRBench /1000.
The honest read by area: Reasoning & Knowledge is this work's: DomCal-EmbGuard 52.3 tops every completed model, including QVAC's fp32 card (52.2) from a 4-bit package 4× smaller. Document Understanding and Visual Perception are q4_k_m-imat's by 0.3 and 0.1, margins inside the documented judge-leniency band (OCRBench) and noise floor; the top three sit within 0.5 of each other on both. Instruction Following is settled on this harness: the same-harness @2048 results are now complete: DomCal-EmbGuard 65.0 🥇 (MM-IFEval 42.2) and TriStack 64.7 🥈 (41.5) both clear q4_k_m-imat's 64.4 (41.1) under the identical 2048-token protocol; DomCal-Slim reads 63.0 (38.8, P 21.0 / C 44.7). Against QVAC q4_0, every build from this work wins all four areas on both rulers: the card (69.9/59.9/48.4/60.9) and the same-harness comparison (70.9/59.7/49.2/61.3), including DomCal-Slim, 6.3% smaller than q4_0, at 71.4/60.3/50.1/63.0. Against the QVAC-harness card (q4_k_m-imat ᵖ, 72.5/61.1/51.2/66.0, same artifact, QVAC's published numbers), DomCal-EmbGuard still takes Document Understanding (73.0 vs 72.5) and Reasoning (52.3 vs 51.2), Visual Perception is a 0.1-margin coin flip, and Instruction Following stays with the card (66.0 vs 65.0, different harness, context only). Per-benchmark values with provenance live in the 17-row wall table below and the interactive scatter above; the panel condenses them to the four area numbers, grouped by area, one bar per model (hover any bar). MME* /2800, OCRBench* /1000.
Strict · judge-free
Judged · default ruler
Judged · QVAC ruler
benchmark
DomCal-EmbGuard409.7 MiB
TriStack387.4 MiB
QVAC q4_k_m-imat392.8 MiB
DomCal-EmbGuard409.7 MiB
TriStack387.4 MiB
QVAC q4_k_m-imat392.8 MiB
DomCal-EmbGuard409.7 MiB
TriStack387.4 MiB
QVAC q4_k_m-imat392.8 MiB
MMStar
47.1
46.9
46.6
—
—
—
—
—
—
MMBench
61.5
61.4
60.7
—
—
—
—
—
—
ScienceQA
84.7
85.1
84.7
—
—
—
—
—
—
MMMU
32.0
31.3
29.3
—
—
—
—
—
—
POPE
87.8
87.9
87.6
—
—
—
—
—
—
SEEDBench
68.5
68.5
68.5
—
—
—
—
—
—
RealWorldQA
59.6
59.6
60.5
—
—
—
—
—
—
AI2D
66.0
65.9
65.7
—
—
—
—
—
—
MME
1530.3
1528.3
1532.3
—
—
—
—
—
—
OCRBench (/1000)
728
728
739
766
764
777
754
752
762
DocVQA
78.2
78.0
78.3
83.6
83.2
83.6
83.1
83.0
83.2
ChartQA
74.6
74.0
74.6
77.4
77.2
77.6
77.4
77.0
77.3
InfoVQA
40.7
40.5
40.8
47.6
47.3
47.7
47.3
47.1
47.5
TextVQA
70.2
69.7
70.1
80.0
79.6
80.0
79.1
78.7
79.1
MathVista
—
—
—
48.6
47.8
47.1
48.7
47.9
47.1
MMVet
—
—
—
35.6
33.5
32.8
31.9
29.9
29.1
MM-IFEval
—
—
—
42.2†
41.5†
41.1‡
N/A ‡
N/A ‡
N/A ‡
Tier-B full-17, this harness (llama.cpp mtmd fork, seed 17, max_new 128). Strict = judge-free scoring; judged columns are labeled per ruler and never mixed. OCRBench shows both strict and judged variants.
* QVAC q4_k_m-imat strict MME = 1532.3 (perception 1244.8 + reasoning 287.5), re-verified from the archived full-17 results JSON; this cell was unverified in the 20:30 extraction.
† MM-IFEval: DomCal-EmbGuard and TriStack were re-generated at QVAC's full 2048-token protocol; 42.2 (P 19.0 / C 50.0, 12/400 judge_failed) and 41.5 (P 17.0 / C 49.7) under the default ruler, both beating imat's same-protocol 41.1; their earlier 37.1 / 37.0 at max_new = 128 were truncation-limited (52% truncation) and are superseded.
‡ The QVAC q4_k_m-imat same-harness comparison used the full 2048-token budget: 41.1 under the default ruler (P 16.0 / C 49.5, 13/400 judge_failed); the valid same-harness reference. The QVAC ruler is N/A to MM-IFEval by construction: its 16-token judge cap truncates the structured rubric verdicts (297/400 calls failed on q4_k_m-imat itself), so QVAC's published 44.2 necessarily used a larger judge budget.
MathVista uses the corrected aggregation; an earlier prefetch double-count produced spurious 68.x values that are never cited (root cause: projects/tier-b/report/JUDGE_PORTABILITY.md). MMMU note: DomCal-EmbGuard 32.0 and TriStack 31.3 both clear q4_k_m-imat's 29.3 (this harness), and DomCal-EmbGuard's 32.0 also tops q4_k_m-imat's declared 31.8 (QVAC harness).
All benchmark scores shown rounded to one decimal (round-half-up); where rounding produces a tie, models share 🥇. Full-precision values are preserved in the release evidence ledger (final_results.json and result CSVs).
DomCal-EmbGuard vs QVAC q4_k_m-imat: 8 wins / 5 losses / 4 tiesDomCal-EmbGuard vs QVAC q4_0: 14 wins / 3 lossesTriStack vs QVAC q4_0: 13 wins / 4 lossesTriStack vs QVAC q4_k_m-imat: 9 wins / 7 losses / 1 tie · MathVista +0.7TriStack vs DomCal-EmbGuard: 3 wins / 13 losses / 1 tie · normalized 61.64 vs 61.97 at −5.4% packageSolo 200-item speed sweep (15 configurations): the 3 fastest observed are this work's; 786.7 / 795.4 / 802.2 ms/item vs QVAC's best observed 803.5
Pairwise badges above compare only the canonical same-harness records: strict rows use the judge-free scorer, judged rows use the default ruler, and MM-IFEval uses the shared 2048-token protocol. The DomCal-EmbGuard and TriStack counts against q4_k_m-imat use the one-decimal values displayed in this table, so small full-precision margins become visible ties after rounding; the other pairwise counts retain their full-precision source values because those comparison rows are not all displayed here. They are descriptive row counts, not significance claims; exact values remain preserved in the final score sources. All six release-ledger packages have complete 17-row records.
10 · Negative results
Tested, measured, rejected
Four techniques were evaluated and rejected with evidence. Negative results carry the same measurement rigor as the wins; they are half the value of this study, and the full trail is preserved.
REJECTED
Reasoning effort
k-sample self-consistency voting
No accuracy change at 1×, 4.1× and 8.1× sampling cost. Additional votes do not shift the answer distribution at 460M scale.
flat · rejected
REJECTED
Adaptive routing
Nano↔Flash adaptive-resolution routing
+0.17 pt at +1.4% time; within noise, and no token-cost axis to justify the added complexity.
no net benefit
REJECTED PRE-RUN
Speculative decoding
draft-model acceleration
Rejected pre-run, with evidence: vocab mismatch 49,218 vs 49,152, no --model-draft in the mtmd fork, 32-token cap.
rejected pre-run
REJECTED
Adaptive token budget
resolution-aware output limits
Quality loss at every reduced resolution: 47.7 → 45.7 → 44.4. The token savings do not pay for the accuracy lost.
quality loss
Also on record: domain calibration on already-calibrated quants adds nothing (iq4_xs −1.2 pt, p = 0.14; q4_k_m −1.7 pt, p = 0.019; a significant regression), and domain calibration at 5-bit hurts (−2.2 pt strict OCR, p = 0.003). Domain calibration is a corrective for uncalibrated quants, not a general improvement.
11 · Conclusion
What this study leaves behind
The useful result is not that one quantization won everywhere. It is that the same ~460M-parameter model, with the same trained weights, becomes four meaningfully different deployment tools once calibration and precision are assigned deliberately. VisionPsy-Nano DomCal, DomCal-Slim, DomCal-EmbGuard and TriStack are quantized derivatives — not newly trained checkpoints — and each one fills an operating point that QVAC's published lineup does not offer.
Why the study was worth doing
A label like “4-bit” says how many bits are available, not whether those bits protect the tensors that matter for the model's actual job. The experiments turned that broad question into a measured map of what helps, what does not, and where the quality cliffs begin.
Domain calibration is a targeted repair, not a universal upgrade. It materially improves the uncalibrated q4_0 build (+2.2 pooled points, p = 4.7×10⁻⁵), but the same idea applied to QVAC's already-calibrated q4_k_m did not stack, and at 5-bit it actively hurt (strict OCR −2.2, p = 0.003). Calibration matters most where importance information is missing.
The projector has a practical compression floor. Moving the mmproj from Q8 to q6_K removes 21.5% of the projector and 6.4% of the complete package with no statistically detectable quality change (pooled p = 0.49); both q5_K implementations failed their quality or embedding-equivalence gates. The floor is q6_K, not “as few bits as possible”.
Rejected ideas narrowed the design space. Self-consistency voting spent 4.1× and 8.1× the sampling cost without moving accuracy; adaptive routing gained 0.17 point at 1.4% more time; speculative decoding was rejected on a hard vocabulary mismatch; adaptive token budgets, the multimodal calibration mixture and per-tensor mixed precision all returned null or negative — across four independent implementation arms. The four surviving models are what remains after those ideas were measured and discarded.
It produced the only same-protocol speed table for these artifacts. QVAC publishes no latency telemetry; the 15-row solo sweep (FP32 down to IQ3_XXS, one GPU, no contention) is the first like-for-like speed comparison of the whole Nano quantization family.
Four artifacts, four jobs
VisionPsy-Nano DomCal
Best fit · replacing an uncalibrated q4_0
domain calibrationsame-size repair
Normalized score 60.31 vs 59.48 for QVAC q4_0, with 12 wins / 4 losses / 1 tie across the 17 rows and a pooled +2.2 points (p = 4.7×10⁻⁵). Stands out on MMStar 45.8 vs 42.3, MMBench 58.1 vs 55.0 and MathVista 43.9 vs 41.7.
348.2 vs 347.7 MiB · 802.2 vs 803.5 ms/item — the q4_0 envelope, repaired at zero measured cost
storage first
VisionPsy-Nano DomCal-Slim
Best fit · minimum download and disk footprint
domain calibration×Q6 projector
At 325.9 MiB it is the smallest package measured anywhere in this study: 6.3% below QVAC q4_0 and 2.5% below QVAC's own smallest, iq3_xxs-imat (334.2 MiB). Its normalized score is still 60.30 — +0.82 over q4_0.
q6_K projector floor backed by pooled parity (p = 0.49) · the storage-first option, not a maximum-quality claim
quality + speed
VisionPsy-Nano DomCal-EmbGuard
Best fit · maximum measured quality and reasoning
domain calibration×embedding protection
The normalized leader at 61.97 (imat 61.54, q4_0 59.48), with 8 wins / 5 losses / 4 ties against imat. Leads all six packages on MMStar 47.13, MMBench 61.46, MathVista 48.6, MMVet 35.6 and MM-IFEval 42.24 — and it is the fastest configuration observed: 786.7 ms/item, quicker than every tested QVAC build.
409.7 MiB (+4.3% vs imat) · reasoning area 52.3 — above even QVAC's fp32 card (52.2) from a ~4× smaller package
61.64 vs 61.54 for q4_k_m-imat with 9 wins / 7 losses / 1 tie, while 1.4% smaller (387.4 MiB) and 2.2% faster (795.4 vs 813.2 ms/item). Best ScienceQA of the study at 85.13; leads imat on MMMU 31.3 vs 29.3, MathVista 47.8 vs 47.1, MMVet 33.5 vs 32.8 and MM-IFEval 41.5 vs 41.1.
387.4 MiB · 795.4 ms/item observed — the cleanest measured balance of size, quality and speed
Mechanism note: DomCal-EmbGuard is a strong shipped configuration, but the incremental EmbGuard step at Q4_K_M precision was not separately significant against its domain-calibrated parent (+0.27 pooled point, p = 0.38). Its complete result should not be attributed to embedding protection alone.
The head-to-head, on one wall
Six packages, one harness, seed 17: the four builds from this work and the two QVAC controls — size, observed latency, normalized quality and QVAC's own four capability areas, side by side.
package
MiBlower is better
ms/item200-item solo sweep
normalized17-benchmark
doc & OCRarea /100
perceptionarea /100
reasoningarea /100
instructionarea /100
best for
DomCal-SlimP1 × P10 · this work
🥇 325.9
817.5
60.30
71.4
60.3
50.1
63.0
tightest download & disk
QVAC q4_0control · same harness
🥈 347.7
803.5
59.48
70.9
59.7
49.2
61.3
MME row lead · QVAC's latency reference
DomCalP1 · this work
348.2
802.2
60.31
71.6
60.2
50.0
63.1
drop-in q4_0 repair
TriStackP1 × P4 × P10 · this work
387.4
🥈 795.4
🥈 61.64
72.8
🥈 61.0 tie
🥈 51.8
🥈 64.7
balanced default
QVAC q4_k_m-imatcontrol · same harness
392.8
813.2
61.54
🥇 73.3
🥇 61.1
51.1
64.4
OCR & document extraction
DomCal-EmbGuardP1 × P4 · this work
409.7
🥇 786.7
🥇 61.97
🥈 73.0
🥈 61.0 tie
🥇 52.3
🥇 65.0
max measured quality + speed
Same harness, default ruler, seed 17 throughout; latency is the 200-item MMStar solo sweep with no GPU contention. Medals compare all six rows per column (MiB: lower wins). Win/loss detail: DomCal vs q4_0 12W/4L/1T (+2.2 pooled, p = 4.7×10⁻⁵) · EmbGuard vs imat 8W/5L/4T · TriStack vs imat 9W/7L/1T. QVAC's smallest artifact anywhere, iq3_xxs-imat (334.2 MiB), is not a row because it has no completed full-17 run here; QVAC ships no latency telemetry, so the sweep above is the only same-protocol speed table for this model family.
What the wall says
This work holds the three fastest rows — 786.7 / 795.4 / 802.2 ms against QVAC's best 803.5 — and the smallest row, 325.9 MiB against QVAC's 334.2 anywhere.
The normalized leader is DomCal-EmbGuard (61.97), whose reasoning area (52.3) also clears QVAC's fp32 card (52.2) from a package roughly 4× smaller.
Every build from this work beats QVAC q4_0 on all four areas at once — including DomCal-Slim, which does it while being 6.3% smaller.
They occupy operating points QVAC does not ship: a same-envelope q4_0 repair, a sub-330 MiB package, a quality-and-speed flagship, and a balanced build that is smaller and faster than the imat flagship while out-scoring it.
Where the wall says no
Documents and OCR stay with imat — area 73.3, OCRBench 777 vs 766 / 764 / 753 / 746 — the one capability family QVAC clearly keeps.
Perception is a 0.1-margin coin flip (61.1 vs 61.0 tie), and the MME row stays with q4_0 (1562.5; this work 1512.9–1530.3). RealWorldQA is also narrowly imat's, 60.5 vs 59.6 for the guarded builds.
Instruction following splits the family. The guarded builds lead (areas 65.0 / 64.7; MM-IFEval 42.2 / 41.5 vs 41.1), but the cheap builds trail imat (areas 63.1 / 63.0; MM-IFEval 38.8 / 38.8) — still ahead of q4_0's 34.8. When instructions dominate, pick EmbGuard or TriStack.
Which one should you use?
Prefer these builds when…
Reasoning matters more than exact transcription. DomCal-EmbGuard's reasoning area is 52.3 and TriStack's 51.8, versus 51.1 for q4_k_m-imat — and MathVista reads 48.6 vs 47.1.
Deployment efficiency has a defined target. DomCal holds the q4_0 envelope, DomCal-Slim minimizes storage, TriStack stays below imat in both size and observed latency, and EmbGuard maximizes quality and speed together.
Instruction following matters and the choice is within the family. EmbGuard (area 65.0) and TriStack (64.7) post the two best measured results — pick them over the cheaper DomCal / DomCal-Slim (63.1 / 63.0).
The workload resembles short, single-image VQA. That is the regime actually measured by this study and the one in which its quality and speed evidence is most relevant.
Prefer QVAC—or a larger model—when…
OCR and document fidelity dominate. q4_k_m-imat keeps OCRBench 777 vs 766 / 764 for EmbGuard / TriStack and the top document aggregate, 73.3.
MME is the deciding benchmark. QVAC q4_0 retains the observed row lead at 1562.5; none of the new variants wins every capability family.
The workload is instruction-dense but the budget only fits the cheapest builds. There, imat's 41.1 on MM-IFEval beats DomCal / DomCal-Slim's 38.8.
The task is long, agentic, multilingual, multi-image or safety-critical. Those requirements are outside this English-focused ~460M model's evidence and should move the decision to a larger, explicitly trained system.
Bottom line
There is no universal replacement — and that is exactly what makes the family useful. DomCal is the evidence-backed q4_0 repair; DomCal-Slim is the storage-first build; DomCal-EmbGuard is the maximum measured quality-and-speed choice when another 16.9 MiB is acceptable; and TriStack is the most compelling balanced alternative to q4_k_m-imat. For OCR and document extraction, imat remains the safer choice. But for reasoning-heavy, instruction-sensitive, mixed on-device work, these builds are more useful than anything in QVAC's original lineup at their size — because they occupy operating points that lineup does not offer, with the costs and limits measured instead of assumed.