← ollama-tune

OLLAMA_KV_CACHE_TYPE: when q8_0 helps — and when it makes things slower

The KV cache defaults to f16. Setting OLLAMA_KV_CACHE_TYPE=q8_0 halves its VRAM footprint, which on paper means roughly double the context before you hit your VRAM ceiling. Sometimes that's exactly what happens. Sometimes the "optimization" makes everything slower. The difference is your GPU/backend combination — which is why this setting should be measured, not copied from a forum post.

Two things people miss

It's server-wide and startup-only. The variable is read when ollama serve starts and applies to every model. Changing it means restarting the server — an export in your shell does nothing to the already-running service.

Flash attention interacts. Quantized KV generally wants OLLAMA_FLASH_ATTENTION=1; without it, some backends fall back to slower paths or ignore the cache setting entirely.

The backfire case (measured, not hypothetical)

On an RDNA2 card running the Vulkan backend, we measured q8_0 forcing a CPU spill that stock f16 didn't have — the dequantization path allocated differently and pushed the model over its placement budget at the same context. Result: the "memory-saving" setting cut generation speed by more than half. On other hardware (notably CUDA cards), q8_0 at high context is a genuine free win. Same flag, opposite outcomes.

How to decide: A/B it on your own card

1. Pick your real target context (the A/B must run where the cache pressure actually is — a 2k test hides everything).
2. Restart Ollama with f16, load, record placement (/api/ps) and tok/s.
3. Restart with OLLAMA_KV_CACHE_TYPE=q8_0, repeat.
4. Winner = fastest type that stays 100% on GPU. Persist it in your service environment; restore stock if neither improves.