← ollama-tune

num_ctx in Ollama: picking the right context size and baking it in

num_ctx is the most consequential setting in a local Ollama install, and the default is wrong for almost everyone: too small for real work (silent truncation), while the model's advertised maximum is usually too big for your VRAM (CPU spill). The right value is a property of your machine, not the model.

What it actually controls

The size of the KV cache reservation. Cost scales linearly with the value and is paid up front, in VRAM, whether or not your prompts use it. Rules of thumb:

• Too low → conversations clip from the top; the model "forgets".
• Too high → the reservation pushes model + cache past VRAM; layers spill to CPU; generation drops 5×.
• Sweet spot → the largest value that stays 100% GPU-resident. Measure it.

Find your sweet spot (manual method)

# at each candidate size, load and check placement:
curl -s localhost:11434/api/chat -d '{
  "model":"qwen3.5:9b","stream":false,
  "messages":[{"role":"user","content":"ok"}],
  "options":{"num_ctx":16384,"num_predict":8}}'
curl -s localhost:11434/api/ps     # size_vram == size ? still good : ceiling found

Walk 2048 → 8192 → 16384 → 32768 → … until placement drops below 100%; your value is the last fully-resident rung.

Bake it in — per-request settings don't stick

Setting num_ctx in one client fixes one client. A tuned model variant fixes every client:

FROM qwen3.5:9b
PARAMETER num_ctx 32768

ollama create qwen3.5-tuned -f Modelfile
ollama run qwen3.5-tuned

Pair it with OLLAMA_FLASH_ATTENTION=1 in the server environment, and consider the KV cache type only after measuring.