QLoRA Fine-Tuning a Coding Assistant on an RTX 3060

The box already ran a local coding model wired to a RAG and memory layer: retrieval keeps it grounded in facts that change constantly, a long system prompt enforces the formatting and tone rules that barely change at all. It works well. It also raised a question worth testing rather than assuming the answer: if a rule almost never changes, why does it live in a prompt I have to re-send every conversation instead of in the model’s weights? I tested it with QLoRA, on the box’s RTX 3060.

That’s the actual pitch for fine-tuning over retrieval. RAG owns facts. Fine-tuning is supposed to own behavior — the commit message format, the tone, the review instincts that stay fixed for months at a time. Bake that into the weights once, and the system prompt should shrink or disappear.

I ran the experiment. Three attempts failed, each with an isolated root cause; the fourth worked, and the eval that followed came back split: a real win on one axis, a real regression on another. Nothing else moved. It shipped anyway, as an optional model I call on for one specific job.

Building a dataset I could trust

Fine-tuning needs instruction/output pairs, and I had three sources already sitting on disk: years of my own commit history across my personal repos, a conventions document I keep as the single written source of truth for my own code style, and a handful of hand-picked code samples. Diffs became “write a commit message for this change,” the conventions doc became question-and-answer pairs, and dedup ran on embedding similarity to drop anything too close to something else already in the set.

One early build of that dataset went wrong before it ever touched the GPU: it accidentally pulled in a project that had no business being in a personal training set. I caught it during review, purged it completely, and rebuilt the dataset clean before training started. Nothing from that first build was ever used.

The set that trained: 419 examples, 369 for training and 50 held out for eval. I dropped an entire planned category, hand-picked “lessons,” because it was too tangled with the excluded project to untangle safely. Better to lose a category than risk it.

QLoRA, restrained for a 12GB card

Base model: Qwen2.5-Coder-7B-Instruct, loaded 4-bit for training. QLoRA rank 16, with conservative alpha and dropout, targeting every linear layer instead of just attention, on the theory that style transfer needs more than the attention heads to move. I set batch size to 1 with gradient accumulation, used paged_adamw_8bit to keep optimizer state off the GPU, turned gradient checkpointing on, and ran bf16 compute.

Unsloth was the plan. The box’s system Python was 3.14, which has no ML wheels at all, and Unsloth’s current release wanted a newer torch than the CUDA stack on this card would tolerate. I bootstrapped an isolated Python 3.12 with uv and fell back to a pinned stack instead — plain PEFT, bitsandbytes, Transformers, and TRL, versions chosen specifically to work together on this GPU. The training environment itself went on the box’s local disk rather than the network share; pulling data over SMB was slower than just keeping it local.

Three ways to fail before it works

• Context-length OOM. The first attempt ran sequence length 2048, which fit fine for training itself but ran out of VRAM the moment it tried to run an in-training eval pass at the end of the first epoch. Fix: drop to sequence length 1024 and move eval to a separate pass after training finishes, so it never has to share VRAM with the optimizer state mid-run.
• A CUDA driver landmine. To claw back headroom, I set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, a normally safe flag for exactly this kind of VRAM pressure. On this box’s WSL2 CUDA driver paired with torch 2.4, it threw CUDA device not ready on the very first backward pass. I checked the GPU itself with a bare matmul-and-backward loop outside the training script; it passed without a hitch, so the card wasn’t the problem. The flag was. Pulling it out fixed the run.
• A WSL memory crash mid-merge. Training finished, then merging the LoRA adapter back into a full fp16 checkpoint (done on CPU, since the merged weights don’t fit in 12GB of VRAM) blew past WSL2’s default 16GB memory cap and wedged the whole VM. wsl --shutdown from the Windows side brought it back, Docker and everything else on the box included, none of it worse for wear. I raised the caps in .wslconfig afterward, memory to 26GB with a 12GB swap ceiling, so it wouldn’t happen again.

The run that finally worked

138 steps at sequence length 1024, about 59 minutes wall-clock, final train loss 0.888. I merged the adapter, converted to GGUF, and quantized to Q4_K_M, the same quant tier the box’s main coding model already runs at, landing a roughly 4.7GB model file. I deployed it into Ollama alongside the existing model, no system prompt attached on purpose. The whole point of the eval that followed was to see if it needed one.

The eval came back split

I compared three things on the same prompts: the untouched base model with no prompt, the fine-tuned model with no prompt, and my existing system-prompted model as the reference point. Twenty-three paired comparisons across four categories, on top of the 50-example held-out set used for loss.

• Commit messages: won. The fine-tuned model produced correctly formatted commit messages, unprompted, that matched my actual convention. This is the category the training data was heaviest in, and it showed.
• Convention questions: improved. Fewer hallucinated answers, more that matched what my written conventions doc says.
• Code review for known gotchas: regressed. The fine-tuned model got noticeably terser and more dismissive here than either the base or the system-prompted model. Commit messages made up 376 of the 419 training examples — the model learned that register a little too well and dragged it into a task that needed a different one.
• General coding questions, unrelated to any of this: unaffected. Against the base model on five held-out prompts, answers held steady or even improved slightly. Whatever the fine-tune changed, it didn’t damage the model’s general ability.

The decision rule going in was simple: if the fine-tuned model, with no prompt at all, doesn’t clearly approach what the system-prompted model does with one, the experiment doesn’t win outright. It approached that bar on two of the four fronts and lost ground on a third. That’s not a clean result in either direction.

What I run now

RAG stays the default. It handles facts, and after this pilot it’s still doing most of the behavior work too, through the system prompt, because that prompt still does something the fine-tune couldn’t fully replace.

The fine-tuned model is live, but only as a side model: I call it explicitly for commit messages, where it’s better unprompted, and leave everything else, especially anything that needs to reason about a gotcha instead of just format text, to the model still running its full system prompt. A rebalanced dataset, less commit-heavy, is the clear next experiment. It’s not one I’ve committed to running yet.