2-bit GGUF leaderboard: Qwen3.8-27B and Qwen3.5-2B (measured, paired)
Published 2-bit GGUF files, scored by us with the same text, context and stride, and compared window by window. Lowest perplexity: DASH-Q on both models. Pingala-8 is our own file. On Qwen3.8-27B, Pingala-8 v2 is the smallest file within 10% of the best (8.31 GB, runs on a 12 GB GPU; ranks 2 of 5). On Qwen3.5-2B, v2 ranks 3 of 6 and beats both unsloth files.
#1 on Qwen3.8-27B
DASH-Q IQ2_XS (jkim96): ppl 6.5431, 8.551 GB
Our Pingala-8 v2 is 5.29% higher (paired dNLL +0.0516, wins 0/20 windows) at 8.314 GB.
#1 on Qwen3.5-2B
DASH-Q IQ2_M (jkim96): ppl 13.3331, 0.786 GB
DASH-Q IQ2_XS is also the smallest file here (0.757 GB) and within 4.27%. Our v2 is 5.43% higher than #1.
Smallest file within X% of the best perplexity
| Perplexity within | Qwen3.8-27B | Qwen3.5-2B |
|---|---|---|
| 1% of the best | DASH-Q IQ2_XS 8.551 GB, ppl 6.5431 (best) | DASH-Q IQ2_M 0.786 GB, ppl 13.3331 (best) |
| 5% of the best | DASH-Q IQ2_XS 8.551 GB, ppl 6.5431 (best) | DASH-Q IQ2_XS 0.757 GB, ppl 13.9028 (+4.27%) |
| 10% of the best | Pingala-8 v2 8.314 GB, ppl 6.8894 (+5.29%) | DASH-Q IQ2_XS 0.757 GB, ppl 13.9028 (+4.27%) |
| 20% of the best | Pingala-8 v2 8.314 GB, ppl 6.8894 (+5.29%) | DASH-Q IQ2_XS 0.757 GB, ppl 13.9028 (+4.27%) |
Qwen3.8-27B, 20 windows
Qwen3.5-2B, 72 windows
DASH-Q (jkim96)unslothPingala-8 (Genrius AI)stock llama.cpp IQ2_XXS (local)ringed = rank 1
Qwen3.8-27B
| # | File | Repo | Size GB | Bytes | PPL | vs best | Paired dNLL vs best | Windows won | Runtime / placement |
|---|---|---|---|---|---|---|---|---|---|
| 1 | DASH-Q IQ2_XS Qwen3.8-27B-DASHQ-IQ2_XS.gguf | jkim96/Qwen3.8-27B-DASHQ-Q2-GGUF | 8.551 | 8,551,168,768 | 6.5431 | best | reference | - | llama.cpp upstream 889f2ec GPU; IQ1_S/IQ2_S/IQ3_S tensors on CPU |
| 2 | Pingala-8 v2 Qwen3.8-27B-Pingala8-2bit-0000{1..9}-of-00009.gguf (9 parts) | Genrius-AI/Qwen3.8-27B-Pingala8-2bit | 8.314 | 8,314,241,600 | 6.8894 | +5.29% | +0.0516 ± 0.0117 | 0/20 | Pingala-8 runtime (patched llama.cpp) GPU |
| 3 | unsloth UD-IQ2_S Qwen3.8-27B-UD-IQ2_S.gguf | unsloth/Qwen3.8-27B-GGUF | 8.372 | 8,371,970,048 | 7.3447 | +12.25% | +0.1156 ± 0.0702 | 2/20 | llama.cpp upstream 889f2ec GPU; IQ1_S/IQ2_S/IQ3_S tensors on CPU |
| 4 | Pingala-8 v1 Qwen3.8-27B-Pingala8-2bit-0000{1..9}-of-00009.gguf (9 parts, earlier revision) | Genrius-AI/Qwen3.8-27B-Pingala8-2bit | 8.314 | 8,314,489,600 | 7.3449 | +12.25% | +0.1156 ± 0.0132 | 0/20 | Pingala-8 runtime (patched llama.cpp) GPU |
| 5 | llama.cpp IQ2_XXS q38-27b-IQ2_XXS.gguf (not published) | not published | 8.433 | 8,433,183,520 | 7.8765 | +20.38% | +0.1855 ± 0.0303 | 0/20 | llama.cpp upstream 889f2ec GPU |
Qwen3.5-2B
| # | File | Repo | Size GB | Bytes | PPL | vs best | Paired dNLL vs best | Windows won | Runtime / placement |
|---|---|---|---|---|---|---|---|---|---|
| 1 | DASH-Q IQ2_M Qwen3.5-2B-DASHQ-IQ2_M.gguf | jkim96/Qwen3.5-2B-DASHQ-Q2-GGUF | 0.786 | 786,259,840 | 13.3331 | best | reference | - | llama.cpp upstream 889f2ec CPU only (CUDA hidden), 12 threads |
| 2 | DASH-Q IQ2_XS Qwen3.5-2B-DASHQ-IQ2_XS.gguf | jkim96/Qwen3.5-2B-DASHQ-Q2-GGUF | 0.757 | 757,096,320 | 13.9028 | +4.27% | +0.0418 ± 0.0060 | 5/72 | llama.cpp upstream 889f2ec GPU; IQ1_S/IQ2_S/IQ3_S tensors on CPU |
| 3 | Pingala-8 v2 Qwen3.5-2B-Pingala8-2bit-v2.gguf | Genrius-AI/Qwen3.5-2B-Pingala8-2bit | 0.778 | 778,189,408 | 14.0573 | +5.43% | +0.0529 ± 0.0081 | 3/72 | Pingala-8 runtime (patched llama.cpp) GPU |
| 4 | unsloth UD-IQ2_M Qwen3.5-2B-UD-IQ2_M.gguf | unsloth/Qwen3.5-2B-GGUF | 0.860 | 859,857,152 | 14.9555 | +12.17% | +0.1148 ± 0.0087 | 0/72 | llama.cpp upstream 889f2ec CPU only (CUDA hidden), 12 threads |
| 5 | Pingala-8 v1 Qwen3.5-2B-Pingala8-2bit.gguf | Genrius-AI/Qwen3.5-2B-Pingala8-2bit | 0.779 | 778,744,192 | 15.6193 | +17.15% | +0.1583 ± 0.0107 | 0/72 | Pingala-8 runtime (patched llama.cpp) GPU |
| 6 | unsloth UD-IQ2_XXS Qwen3.5-2B-UD-IQ2_XXS.gguf | unsloth/Qwen3.5-2B-GGUF | 0.768 | 768,270,592 | 23.4124 | +75.60% | +0.5630 ± 0.0177 | 0/72 | llama.cpp upstream 889f2ec GPU; IQ1_S/IQ2_S/IQ3_S tensors on CPU |
Method
- Text: WikiText-2 raw test split (Hugging Face rows joined), 297,193 tokens with the Qwen tokenizer.
llama-perplexity -c 2049 --ppl-stride 4095 -b 512 -ub 512 -ngl 99: strided windows of 4096 tokens. Qwen3.8-27B: first 20 windows (time limit); Qwen3.5-2B: all 72. - Rival files ran on llama.cpp upstream 889f2ec. On our RTX 5070 Ti Laptop GPU (sm_120, WSL2), the IQ1_S/IQ2_S/IQ3_S CUDA kernels failed test-backend-ops, so those tensors ran on the CPU (
-ot ...=CPU --no-op-offload). The two Qwen3.5-2B IQ2_M runs ran fully on the CPU. - A CPU/GPU cross-check agreed within 0.2%.
- Pingala files ran on the patched Pingala-8 runtime (a llama.cpp fork); those numbers are for that runtime.
- The paired statistic is the mean per-window difference in NLL against the best file, with a 95% interval (1.96 sd / sqrt(n)) and the count of windows won. Per-window NLL is recovered from the running perplexity: NLL_k = k ln P_k - (k-1) ln P_(k-1).
Caveats
- Perplexity on one English text only; no downstream task scores here. Qwen3.8-27B uses 20 windows (about 82k tokens), so its intervals are wider.
- The log prints running perplexity to 4 decimals, so derived per-window NLL carries a small rounding error that grows with the window index (at most about 3e-4 nats at window 72).
- Rival files and our files ran on different builds and, for some tensors, different devices; see the placement column.
- The Qwen3.8-27B IQ2_XXS row is our own stock llama-quantize file with the imatrix used for Pingala-8; it is not published. Pingala-8 v1 for Qwen3.8-27B is an earlier revision of the same repo, replaced by v2.
- Genrius AI makes Pingala-8. We publish every window so anyone can check the ranking.
Data and links
- Raw per-window measurements (CC BY 4.0): Genrius-AI/2bit-gguf-wikitext2-eval
- Method page: Pingala-8 paper (HTML) and project site
- Paper DOI: 10.5281/zenodo.23481757
- Model repos: jkim96/Qwen3.8-27B-DASHQ-Q2-GGUF, jkim96/Qwen3.5-2B-DASHQ-Q2-GGUF, unsloth/Qwen3.8-27B-GGUF, unsloth/Qwen3.5-2B-GGUF, Genrius-AI/Qwen3.8-27B-Pingala8-2bit, Genrius-AI/Qwen3.5-2B-Pingala8-2bit
- Collection: Pingala-8: 2-bit LLMs for llama.cpp