# ollama-tune

Probe your hardware, **benchmark an Ollama model on the actual box**, detect the real
bottleneck, and emit an optimal config. It automates the "measure, don't assume" tuning
playbook — the same steps you'd otherwise do by hand and usually skip.

## Why

Small shops run Ollama on bad defaults and never know it: the context window silently
truncates (reads as "the model is dumb"), the KV-cache type backfires on their GPU, the
model quietly spills to CPU, or a reasoning model burns its whole budget "thinking." None
of that shows up unless you *measure*. This tool measures.

## Use

```
ollama-tune list
ollama-tune profile qwen3.5:9b
ollama-tune profile qwen3.5:9b --ctx 2048,8192,16384,32768 --emit tuned.Modelfile
ollama-tune profile qwen3.5:9b --deep --kv-types f16,q8_0   # A/B the KV cache type
ollama-tune apply qwen3.5:9b --num-ctx 32768                # bake <model>-tuned + persist env
ollama-tune serve                                           # web dashboard on :8770
```

The **dashboard** (`serve`) is the clickable version: pick a model, **Run Profile** or
**Run KV A/B**, compare the tables, and hit **Apply best** to bake the tuned model and
persist the config — plus a live proxy-health badge.

## What it does

1. **Probe hardware** — GPU name, total VRAM, and the compute backend Ollama *actually
   selected* (Vulkan / ROCm / CUDA / CPU), read from its own server.log — not from what
   libraries happen to be installed.
2. **Read the model** — params, quant, native context, and capabilities (thinking/tools)
   via `/api/show`.
3. **Benchmark** — sweep a context ladder; at each `num_ctx`, load the model and read
   `/api/ps` to see the GPU/CPU placement, and measure generation tok/s. Plus a prefill
   sample over a multi-thousand-token prompt.
4. **Analyze** — find the largest context that stays 100% on GPU (the sweet spot), flag
   default-context truncation, reasoning-model risk, the VRAM ceiling, and note when
   generation is memory-bandwidth-bound (so you don't chase kernel rewrites).
5. **Emit config** — a `FROM … / PARAMETER num_ctx …` Modelfile, env recommendations, and
   whether a serving layer (qwen-proxy) is needed.

## Layout (thin spine, one concern per file)

```
src/main.rs           CLI dispatch
src/cli.rs            args (clap)
src/ollama/           client + response types (show/ps/chat-timed/tags)
src/hardware/         best-effort GPU/VRAM/backend probe (fail-soft)
src/bench/            empirical context sweep (placement + speed)
src/analyze/          findings + recommendation
src/report/           render + emit Modelfile
```

## Honest limitations

- **Hardware probe is per-OS and fail-soft**: Windows (PowerShell + registry + Ollama's
  server.log), Linux (`nvidia-smi`, then amdgpu sysfs; backend via journald or
  `~/.ollama/logs`), macOS (`system_profiler`; Apple Silicon reported as unified
  memory). Unknown fields degrade the report, they don't crash it. Linux and macOS
  paths are compile-checked and parser-tested, not yet runtime-verified on real
  machines (Windows is).
- **Linux under systemd**: `--deep` and restarts REFUSE to fight the service manager —
  they print the exact `systemctl edit ollama` commands instead of silently mutating
  service config. On macOS the server restarts headless; relaunch the menubar app
  manually if you prefer it.
- **Localhost/http only** (no TLS backend built in) — the tool tunes the box it runs on.
- **Generation tok/s is noisy under GPU contention** (e.g. a training job sharing the
  card). Run it on an idle GPU, or average multiple runs, for stable numbers. The
  `--deep` "best" pick uses gen tok/s as the tiebreaker among all-GPU types, so noise
  can flip a near-tie — run `--deep` at your real target context (where q8's spill
  actually shows), not a small one.
- **`--deep` restarts Ollama** several times (disrupts anything using it mid-run) but
  always restores stock (f16) settings afterward, even on error.
- `apply` persists env per OS: Windows at **User scope** (new shell picks it up); Unix
  writes `~/.config/ollama-tune/env.sh` and tells you how to load it — it never edits
  your shell rc or systemd units.

## Roadmap

- Average multiple gen samples in `--deep` and weight prefill speed in the pick.
- Speculative-decoding advisor (draft-model pairing) — the real generation-speed lever.
- Live progress streaming in the dashboard for long `--deep` runs.
- Runtime verification of the Linux/macOS probes on real hardware.
