Local LLM Benchmarks on an RTX 3060: 35 Tokens a Second

By this point the box has SSH, Tailscale, a self-hosted CI runner, and a memory framework that indexes thousands of files. None of that answers the question that matters most for daily work: is the model on this RTX 3060 fast enough to code against, or is this a hobby project that happens to run Ollama.

Time to look at the actual numbers.

Tokens per second, measured

The daily-driver model is qwen2.5-coder:14b, quantized to Q4, about 9GB of weights. Ollama runs on the Windows host directly, not inside WSL, with flash attention and an 8-bit KV cache turned on, so a 16k context window fits on top of those weights inside the same 12GB card.

I measured it straight off the box’s GPU, over the tailnet, on 220-token generations:

qwen2.5-coder:14b          (14B dense, Q4_K_M, ~9GB)   ~35 tok/s
deepseek-coder-v2:16b-lite (MoE, ~2.4B active)         ~70 tok/s

Thirty-five tokens a second on the 14B is faster than I read, and that dense workhorse is the number that governs real coding. The MoE coder roughly doubles it, near 70, because only about 2.4B of its parameters fire per token; that’s the one wired to autocomplete-style prompts, where every millisecond of latency shows. Both sit comfortably under the card’s real ceiling: a 14B Q4 is about 9GB, the RTX 3060 moves roughly 360 GB/s, so memory bandwidth alone caps a 9GB model near 40 tok/s and 35 is most of the way there.

The measurement also killed an assumption I’d been carrying: that reaching the box from far away would be slower. Generation runs entirely on the card, so a request from the same room and a request from a café across the tailnet come back at the same tokens per second. The network adds a few milliseconds to the first token and nothing to the stream. Anywhere Tailscale reaches, the box answers at box speed.

The 12GB ceiling

The RTX 3060 in this box has 12GB of VRAM, and that’s the real constraint on the whole setup.

A 14B model at Q4 fits inside it with room left for a 16k-24k token context window. Ollama’s own settings for this box spell out the tradeoff:

OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KEEP_ALIVE=-1
OLLAMA_NUM_PARALLEL=1
OLLAMA_MAX_LOADED_MODELS=1

OLLAMA_MAX_LOADED_MODELS=1 is the honest line in that list. This card holds one model at a time. Swapping from the coder model to the vision model or the reasoning model means unloading one and loading the other, a few seconds of dead time on every switch.

Checked against the card directly, during the CI runner work earlier in this series: VRAM sat at 11919MiB / 12288MiB before and after a second service went live. That is the model’s full resident footprint, the ~9GB of weights plus the 16k context and its 8-bit KV cache, and it was already using effectively all of the card.

What didn’t work: the 32B attempt

Once the 14B model felt comfortable, the obvious next move was to try something bigger. qwen2.5-coder:32b is meaningfully stronger on harder refactors, and the box wasn’t doing anything else with that VRAM most of the day.

It loaded, and it didn’t fit. A 32B model at any usable quantization overruns 12GB, and the overflow spills into system RAM over a DDR4-3600 bus nowhere near GPU-memory bandwidth. It still answers, but at about 3 to 4 tokens a second, a tenth of the 14B’s pace. A normal back-and-forth turns into watching each word crawl out. Fine for something queued up before bed. Unusable for an interactive edit loop.

qwen3-coder:30b and anything in the 24B+ range hit the same wall. The ceiling doesn’t move because the box happens to be idle; it’s a property of the card, not the workload. Those models got reassigned to a batch tier instead, run overnight rather than chatted with. The 14B model stayed the daily workhorse for exactly that reason.

A shelf of models

Instead of a bigger card, I built a shelf of smaller models, each one sized to fit the card I have. A fast MoE coder (deepseek-coder-v2:16b-lite) answers autocomplete-style prompts quickly. A small vision model reads screenshots and mockups. A reasoning model handles the occasional debugging session that benefits from a model that thinks in steps. Each one runs alone inside the same 9-12GB window, loaded on demand rather than stacked.

Fifteen models lived on that drive then. Twelve gigabytes decides which one is running at any given moment.

How it feels next to Claude

Tokens per second answers whether the box is fast. It doesn’t answer whether a 14B model on a $300 GPU can replace paying for Claude on the hard days.

The honest answer is no, and the reason has nothing to do with speed. On routine work: rewriting a function, adding the boilerplate around an API call, explaining what a block of code does, translating a comment, the local model reads and writes about as usefully as I need, for $0 a call. On harder work: tracing a bug across a codebase it hasn’t seen in this context window, holding a long architectural back-and-forth, the difference shows up fast. The local model starts guessing where Claude asks the right clarifying question, or misses an inconsistency six files away that Claude catches.

Both models feel conversational; neither one keeps you waiting. The gap is depth on hard problems, and no amount of extra context window on the same 14B model closes it.

The practical split that fell out of this: the local model absorbs the volume of small, mechanical requests that used to burn a cloud quota for no good reason, and Claude gets reserved for the calls that need judgment. The weekly usage limit on my Claude plan, the one that used to run out in a couple of days, holds through most weeks now that the routine share of requests never touches it at all.

Deciding which bucket a given request belongs to, automatically, without me picking a model by hand every time, is the next problem: getting a router to make that call in real time.