A Difficulty Router That Never Calls an LLM to Decide

The gateway had been up for a while, still without a router: a local gateway with a few local models and optional cloud slots, answering behind one OpenAI-compatible endpoint on the box, a test chat coming back “ok.” What it didn’t have was an opinion. Every session, I typed the model name myself: the smaller local model for a quick edit, the bigger one when I remembered a task might be heavier, Claude when I got tired of guessing.

That’s a manual routing decision, and a bad one. So a small script went in front of the gateway to make that call before any model gets touched.

Rules first, no network required

router.py runs a cheap pass before anything else: a keyword and file-size scan, no API call, no embedding, nothing that can time out.

Hard signals send the task straight to Claude: refactor, migrate, security, audit, architecture, race condition, deadlock, cross-module, or a touched file over 400 lines. Routine signals send it to the smaller local model: rename, format, typo, docstring, add test, as long as the file involved is 60 lines or under, or there’s no file at all.

python3 router.py "rename the variable foo to bar" --no-execute
# {"tier": "routine", "source": "rules"}

python3 router.py "refactor the authentication module architecture across services" --no-execute
# {"tier": "hard", "source": "rules"}
# -> escalate to Claude, no gateway call

Some tasks don’t match either list cleanly. “Improve the error messages in this component” isn’t a rename, and it isn’t a refactor either. That’s where the rules pass punts instead of guessing.

The fallback leans on an embedding

Asking a language model whether a task is hard or easy would defeat the point: a routing decision that itself costs a generation call cancels out the reason to route in the first place, which is spending compute only where it’s earned.

Instead, the fallback reuses the embedding model already installed on the box for the memory framework’s search index, qwen3-embedding:0.6b. The inconclusive task text gets embedded, then cosine-compared against two small sets of exemplar phrases, one written to read as routine work and one written to read as hard work, in the pattern Aurelio’s Semantic-Router uses. Closer to the routine cluster wins routine, closer to the hard cluster wins hard, and too close to call becomes medium.

python3 router.py "improve error messages in this component" --no-execute
# inconclusive on rules -> semantic step
# {"tier": "medium", "source": "semantic"}

No generative call happens anywhere in this path. Producing the embedding is a single encoder-model call that returns a vector and generates no text; the lookup is the cosine comparison that follows it. That’s the same discriminative-versus-generative line the memory framework already draws around the box’s other near-free tools.

Tier      Engine                Executed via
--------  --------------------  --------------------------------------------
routine   smaller local model   gateway, on the box
medium    bigger local model    gateway, on the box
hard      Claude Code, direct   not the gateway - escalation signal only

Executing the routine and medium tiers against the gateway is owner-gated: it needs a key set by hand before the call goes through. Without it, the router still decides and prints the decision; it just doesn’t place the call. --no-execute skips that gateway call on any tier — the two rules examples above need no network at all, but the semantic example still calls the embedding model to produce its vector.

The approach that got rejected: RouteLLM

RouteLLM exists and does something close to this: a learned router built for products serving unknown users at scale, trained to decide when a cheap model is good enough and when to escalate to an expensive one.

It’s also solving a different problem than the one on this desk. RouteLLM earns its complexity at the scale of a product serving unknown users and unknown queries, where nobody can hand-write the rules because nobody knows the query distribution in advance. My queries are my own coding tasks, on my own repos, and the task types are already known: renames look like renames, cross-module refactors look like cross-module refactors. Training a classifier to rediscover a distinction I can already write as a keyword list isn’t worth building for a problem this small.

So the decision layer is under 200 lines, standard library only: rules, then embeddings, no training step, no calibration set to maintain. It’s the coding version of the same cost-tiering the memory framework already uses elsewhere.

Why Claude stays outside the gateway

The gateway is plumbing, and Claude Max sits outside it deliberately.

The gateway’s cloud slots are built around API keys metered per call. Claude Max is a subscription, not a metered key, so putting it behind the same gateway would mean pretending it has a key it was never issued.

The router’s hard tier reflects that split: it doesn’t call anything through the gateway. It prints an escalation and stops. Claude Code takes the task directly, with its own full harness and session context, instead of a bare completion call through a proxy that was never built for it. Escalating to Claude doesn’t depend on the box either: if the box or the gateway is down, escalating still works, because Claude reaches out over the internet on its own.

Where this can misroute

A keyword list and a cosine similarity are heuristics, and heuristics misroute. “Improve the error messages” could be a one-line string change or a sign that the whole error-handling path needs rethinking, and the router has no way to tell those apart beyond how close the sentence sits to a couple of example phrases.

If the embedding call can’t reach anything at all — the box’s endpoint, or the Mac’s own local fallback — the router falls back to the middle tier rather than guessing wrong in either direction. That fallback isn’t free: at the time, the middle tier’s model was the 32B one Part 10 measures at three to four tokens a second, already called unusable for interactive work there, and it can’t run at all while the box itself is unreachable.

Where it stands

The gateway is live. The example decisions above were checked as a dry run; one routine task, a typo fix, ran end to end through the gateway, late August 2026. The gateway’s key was already set by hand by then; running the router itself is still a manual step, the same way the owner-gated steps earlier in this series needed a human to act once.

The decision happens on the Mac, where the router runs — not wired into Claude Code, not in front of every task. It runs by hand, when I remember to reach for it. The box’s job is smaller: supply the embedding call, and execute whichever local tier gets picked. What happens when the box itself reboots with nobody at the keyboard is a different problem, and it’s next.