Making Large Models Fit Smaller Hardware
One 27B checkpoint, measured on a 2019 Turing workstation card, an Ada 48 GB card, and a DGX Spark. Same model, same benchmark flags — decode ranges from 12.38 to 75.58 tok/s depending on card, quant, and serving stack.
The problem
The usual question — "will this model run on our hardware?" — has no useful yes/no answer. Qwen3.8-27B fits on a 2019 Turing Quadro RTX 6000 (24 GB) and on an Ada RTX 6000 (48 GB) and on a DGX Spark (GB10, ~128 GB unified), and behaves completely differently on each. The Q8_0 GGUF is 26.62 GiB and does not fit the 24 GB card at all; Q6_K fits at 20.56 GiB but costs decode throughput; Q4_K_M fits at 15.40 GiB and leaves KV headroom. Meanwhile the choice of serving stack moves the number further than the choice of card: on the same GB10 node, llama.cpp Q4_K_M measures 12.38 decode tok/s while vLLM with an NVFP4 checkpoint and speculative decoding measures 50.13 tok/s on a new write. Procurement decisions get made on vendor slides that collapse all of this into one figure. The work below is the opposite: fixed hardware, fixed checkpoint, published flags, published logs, and the numbers that came out.
How it was measured
Fix the hardware envelope first, then pick the quantization. Each card gets its own recipe: Turing (SM 75, 24 GB) and Ada (SM 89, 48 GB) are documented as separate packs precisely because the Turing recipe is not the Ada recipe and neither is the Blackwell/NVFP4/vLLM path.
Benchmark the same GGUF checkpoint on every card with identical llama-bench flags (pp512 / tg128, ngl 99, flash-attn on), pinned driver and llama.cpp commit, and the raw log published alongside the table. Turing: driver 582.08, llama.cpp dd1ea52. Ada: driver 595.84, CUDA 13.0, llama.cpp 885c5bb. Spark: llama.cpp b610 / 70dfba5.
Establish run-to-run variance before quoting a delta. The Turing Q6_K configuration was re-run in a separate morning session: 650.09 ± 13.64 vs 653.58 ± 8.99 prefill, 17.88 ± 0.03 vs 18.00 ± 0.04 decode. Differences smaller than that spread are not findings.
Test the assumptions that sound obviously true. Adding a second Ada card via layer split raised prefill from 2090 ± 73 to 2460 ± 89 tok/s and moved decode from 46.0 ± 0.08 to 46.5 ± 0.05 — inside noise. Dual-GPU layer split does not buy decode for this model.
Cross the engine boundary only deliberately, and never inside one average. On the GB10 node the fastest GGUF path is stock Q4_K_M at 12.38 ± 0.03 tg128; the vLLM NVFP4 + DSpark depth-14 path at 256k context measures 50.13 tok/s on a new write and 75.58 tok/s on a cached repeat file. Those are llama-bench tg128 and vLLM wall-clock respectively — reported side by side, never averaged.
Keep a quality number attached to every throughput number. On the Spark, sixcat-eval scored 82.3 overall for the stock Q4 build (b610) against 80.0 for the AEON bake Q4 + MTP build (9f73) — a throughput change that also moved the quality score. Where no stock quant lands inside the envelope, the mixed-precision path (TensorQuant) plans an explicit tier — 2.47, 2.97, or 3.52 effective BPW — with an inspectable per-tensor policy, a strict llama.cpp dry run, and a SHA-256-verified quantizer binary rather than a guessed flag.
Benchmarks
Every value below is a published measurement. Dashes are configurations that were not measured — not zeros, and not estimates.
| Card / node | Stack + quant | Size on disk | Prefill tok/s (pp512) | Decode tok/s | Measurement path |
|---|---|---|---|---|---|
| Turing Quadro RTX 6000 (SM 75, 24 GB, 2019) | llama.cpp Q4_K_M | 15.40 GiB | 720.40 ± 13.13 | 24.16 ± 0.07 | llama-bench pp512/tg128, 2026-08-14 |
| Turing Quadro RTX 6000 (24 GB) | llama.cpp Q6_K | 20.56 GiB | 650.09 ± 13.64 | 17.88 ± 0.03 | llama-bench pp512/tg128, 2026-08-14 |
| Turing Quadro RTX 6000 (24 GB) | llama.cpp Q6_K, repeat run 09:34 PDT | 20.56 GiB | 653.58 ± 8.99 | 18.00 ± 0.04 | llama-bench pp512/tg128, 2026-08-14 |
| Turing Quadro RTX 6000 (24 GB) | llama.cpp Q8_0 | 26.62 GiB | — | — | Does not fit on the 24 GB card |
| Ada RTX 6000 (SM 89, 48 GB) | llama.cpp Q4_K_M | 15.40 GiB | 2090 ± 73 | 46.0 ± 0.08 | llama-bench pp512/tg128, 2026-08-14 |
| Ada RTX 6000 ×2 (layer split) | llama.cpp Q4_K_M | 15.40 GiB | 2460 ± 89 | 46.5 ± 0.05 | llama-bench pp512/tg128, 2026-08-14 |
| DGX Spark (GB10 / SM121, ~128 GB unified) | llama.cpp Q4_K_M stock (b610) | 15.40 GiB | 844.11 ± 8.48 | 12.38 ± 0.03 | llama-bench pp512/tg128 |
| DGX Spark (GB10 / SM121) | llama.cpp Q5_K_M stock | 17.90 GiB | — | 10.82 ± 0.01 | llama-bench tg128 |
| DGX Spark (GB10 / SM121) | llama.cpp Q6_K stock | 20.56 GiB | — | 9.42 ± 0.01 | llama-bench tg128 |
| DGX Spark (GB10 / SM121) | vLLM NVFP4 + DSpark depth 14, 256k ctx, warm | — | — | 50.13 new write / 75.58 repeat file | vLLM wall-clock — not comparable to tg128 |
| DGX Spark (GB10 / SM121) | SGLang NVFP4 RadixArk, mem=0.80 | — | — | 11.94 new write / 11.97 repeat file | vLLM-style wall-clock |
| DGX Spark (GB10 / SM121) | vLLM NVFP4 + in-file MTP n=3 | — | — | 29.61 new write / 32.34 repeat file | wall-clock |
What the numbers cost
Quantization buys headroom and costs decode, in that order. On the 24 GB Turing card, moving from Q6_K to Q4_K_M raised decode from 17.88 to 24.16 tok/s and dropped the file from 20.56 to 15.40 GiB — but Q6_K plus q4_0 KV cache quantization is what reaches the model's native 262144 context, where the Q4 default sits at a safe 32768. There is no setting that is best at both.
Throughput numbers from different engines are not one scale. llama.cpp tg128 and vLLM wall-clock tok/s measure different things, and the DGX Spark data spans both. Reporting 12.38 and 50.13 in the same column without saying which is which produces a confidently wrong comparison — the exact failure mode catalogued across the 122 entries in our model-serving minefield registry.
The speculative-decoding numbers are workload-shaped, not universal. The 75.58 tok/s figure is a cached repeat file; the honest planning number for new generation on that config is 50.13. Draft acceptance was 98.8% (3134 / 3171) on the cold DSpark depth-7 workload — high acceptance is why the gap is large here, and it will not hold for every prompt distribution.
What this does not show
- No VRAM or KV-cache occupancy was measured on Turing or Ada — only model size on disk.
- No concurrency or multi-user benchmarks exist for this model; every row is single-stream.
- Quality was evaluated only on the DGX Spark. Quantization-vs-quality on Turing and Ada is untested.
- llama.cpp
tg128and vLLM wall-clock are different measurement paths and are kept visually separate above. They are not one scale. - No cost, power draw, or $/token figures exist, so no price-performance claim is made.
Run it yourself
Every row is reproducible from published scripts. Turing: llama.cpp built -DGGML_CUDA=ON for sm_75, commit dd1ea52 (519), driver 582.08, flash-attn on; raw log published at logs/llama-bench-turing6000-20260814_123315.log; re-run with ./bench.sh. Ada: log at logs/llama-bench-ada6000-20260814_123834.log (2026-08-14 12:38–12:40 PDT), driver 595.84, CUDA 13.0, llama.cpp 885c5bb, ngl 99, flash-attn on; download.sh / start.sh / stop.sh / bench.sh with documented env-var overrides. DGX Spark: vLLM 0.1.dev1+g75231eff2.d20260809 via vllm-start.sh (DEPTH=14 default) with python3 ./vllm-smoke.py for timing, SGLang 0.5.17 with QUANT and MEM overrides, llama.cpp binary 70dfba5. Quality scores come from sixcat-eval (default --limit 20, --max-minutes 30, about 120 scored rows across knowledge, math, truth, instruction, code, tools). Mixed-precision plans reference llama.cpp commit 70dfba5aee36793fb51ae649723b3c30ed9e99d3 with validated llama-quantize SHA-256 57a7b0ddb8117ba9ec745db6f46687302c679b175812edaaf5c71c36d5c076c9.