← ollama-tune

Ollama using CPU instead of GPU: spill mechanics and your VRAM ceiling

Two different problems get reported as "Ollama isn't using my GPU": the backend never initialized (everything runs on CPU), or — far more common — the model partially spills: most layers on GPU, a few on CPU, and generation speed falls off a cliff anyway. The second one is sneaky because the GPU shows activity, so everything looks fine.

Check what's actually happening

curl -s http://localhost:11434/api/ps

Compare size_vram with size while a model is loaded. Equal → fully on GPU. size_vram smaller → spilled; the ratio is your placement. For the backend question, read Ollama's own server log: the backend it says it selected (CUDA / ROCm / Vulkan / Metal / CPU) is the truth, regardless of what drivers you have installed.

Why it spills: the KV cache is part of the footprint

VRAM must hold weights + KV cache + compute buffers. The KV cache scales linearly with your requested num_ctx — and it's reserved up front, whether or not you use the tokens. A model that fits comfortably at 8k can be over budget at 64k. That's why "it worked yesterday" breaks after you raised the context: the weights didn't change, the reservation did.

Why partial spill hurts so much

Generation is sequential: every token needs every layer. If even a few layers sit in system RAM, each token round-trips over PCIe, and the whole pipeline runs at the speed of its slowest hop. We've measured a model going from ~48 tok/s (100% GPU) to ~9 tok/s at 81% GPU. There is no "mostly fast" — placement is a cliff, not a slope.

The fix: find your ceiling and stay under it

Sweep context sizes (2k → 8k → 16k → 32k → …), check placement at each, and cap num_ctx at the largest value that stays 100% resident. Bake that into a tuned model so no client accidentally requests more. If the ceiling is lower than the context you genuinely need, the levers are a smaller quant or a quantized KV cache — measured, because the KV option can backfire.