Our 2-bit MoE has no GPU kernel — and the GPU still makes it 4× faster
We measured our 35B 2-bit mixture-of-experts model (TQ2Q) on an NVIDIA A10G expecting to
confirm it "runs on the GPU." It doesn't: mainline llama.cpp has no CUDA kernel
for the 2-bit TQ2_0 format, so the expert weights compute on the CPU no matter
what flags you pass. And yet moving the model onto the GPU still makes decode
4.1× faster — 14 → 59 tokens/sec on the 35B. The reason is the useful part:
the 2-bit experts are only ~13% of the bytes you read per token, so everything else offloads,
and that's where the time was.
Correction (2026-07-07). An earlier version of this post claimed the 2-bit kernel runs on CUDA. It doesn't — we mis-read a bit-identical perplexity as "the kernel works" when it was actually the signature that the experts never left the CPU. The corrected story below is, if anything, the more useful one.
What we expected — and why it was wrong
TQ2Q packs Mixture-of-Experts weights into llama.cpp's 2-bit TQ2_0 container,
but uses a fourth code word (q=3 → +2×scale) that plain "ternary" leaves empty.
We wanted to know whether a GPU would read that fourth level correctly or strip it. So we ran
each GGUF CPU-only (-ngl 0) versus fully offloaded (-ngl 99) and
compared perplexity on WikiText-2, reasoning: if q=3 is stripped on the GPU,
perplexity collapses; if it matches, q=3 survived.
| Model | CPU perplexity | Full-GPU perplexity |
|---|---|---|
| OLMoE-1B-7B TQ2Q | 11.2148 | 11.2148 |
| Qwen3.6-35B-A3B TQ2Q | 6.7337 | 6.7409 |
The perplexity matched — OLMoE bit-identical, the 35B within 0.1%. We read
that as "the fourth level survives on CUDA." It was the wrong conclusion: a match is
also exactly what you see if the experts never ran on the GPU at all. And that is what
happens. Mainline llama.cpp has no CUDA kernel for TQ2_0 —
grep ggml-cuda/ for it and you get nothing — so the ggml scheduler keeps every
2-bit expert matmul on the CPU backend, even at -ngl 99. The bit-identical
OLMoE number is the proof the experts stayed on the CPU (the same CPU kernel ran
both times); the 35B's 0.1% wobble is a small activation-rounding difference on the
other tensors — the ones that did move to the GPU, where the quantized shared-expert
and head weights use the CUDA path's finer Q8_1 activation blocks instead of the
CPU's Q8_K. Our test couldn't tell "q=3 survives on CUDA" from "q=3 never
touched CUDA."
No CUDA kernel; the 2-bit experts run on the CPU. So
q=3-on-CUDA is still untested — verifying it needs an explicit build
of compilade's in-progress TQ2_0 CUDA branch, which we haven't done. That's
the real next experiment.
So why is "the GPU" 4× faster?
Because -ngl 99 doesn't move the experts — it moves everything else. Per token,
our 35B reads ~1.97 GB of weights, and only ~260 MB of that — about 13% — is
the TQ2_0 routed experts. The other ~1.7 GB is not 2-bit: F16 attention,
the Q5_K always-on shared expert, the Q6_K vocab head, the router. Full offload puts that
1.7 GB on the GPU's ~600 GB/s while the small 2-bit slice rides on the CPU. Decode is
memory-bound, so throughput follows the bytes:
Decode: hybrid CPU+GPU
| Model | CPU (t/s) | Full-GPU (t/s) | Speedup |
|---|---|---|---|
| OLMoE-1B-7B TQ2Q | 65.2 | 118.8 | 1.8× |
| Qwen3.6-35B-A3B TQ2Q | 14.4 | 59.4 | 4.1× |
The 35B jumps from 14 to ~59 tokens/sec — not because the 2-bit weights are on the GPU (they aren't), but because the 87% of per-token bytes that aren't 2-bit are. The A10G shares the RTX 3070 Ti's memory bandwidth, so a consumer card with room for the non-expert tensors would do much the same.
Prefill barely moves
Prefill — digesting the prompt before the first token — is a different story, and now a predictable one:
| Test | CPU (t/s) | Full-GPU (t/s) | Speedup |
|---|---|---|---|
| prompt 512 | 143.2 | 153.5 | 1.07× |
| prompt 2048 | — | 154.6 | flat |
The GPU is barely ahead of the CPU here — 1.07×, and flat from 512 to
2048 tokens. Which is exactly what you'd expect now: prefill is compute-bound, and the
expert matmuls — the bulk of that compute — run on the CPU regardless of
-ngl. Offloading everything else can't move a number the CPU still gates.
Why it matters
The useful takeaway is the one we didn't expect: you don't need your 2-bit quant to have a GPU kernel to get a GPU speedup. A 2-bit MoE puts almost all of its parameters into experts that are individually tiny and only sparsely activated — so the parts actually worth accelerating (attention, the always-on shared expert, the head) are the unquantized parts, and those already have GPU kernels. Offload them, leave the 2-bit experts on the CPU, and get ~4× decode on a single 24 GB card. It's hybrid, not "on the GPU" — but it's a real lever for anyone with a mid-range GPU and a CPU with memory bandwidth to spare.
Two honest limits. This is not a correctness result for our fourth level on
CUDA — testing q=3 on a real TQ2_0 GPU kernel needs a build of
compilade's in-progress branch, and that's our next experiment. And prefill stays CPU-bound
until such a kernel exists.
Measured with llama-bench, single stream, on mainline llama.cpp master
(build 9892, July 2026 — the TQ2_0 CUDA kernel, PR #11183, was still unmerged,
so ggml-cuda/ is grep-clean for it) on an AWS A10G. Corrected 2026-07-07 after a
fleet cross-check flagged the buffer placement — the 2-bit experts sit in host RAM, not VRAM.