polyrouter's routing engine is a layered, degradable pipeline. Layer 0 explicit routing is the reliable core that always works; the automatic layers refine model:"auto" and always fall back to explicit or default. L1 structural is on by default; L2 semantic and L3 cascade are opt-in. Every smart-layer fault degrades to exactly the path it would have taken with that layer disabled. The smart path never fails or stalls a request.
Decision layersexplicitmodel field onlyheaderx-polyrouter-tier · header rulesstructuralL1semanticL2cascadeL3defaultcatch-all
Routing precedence
Layer 0 is a pure function — no database, no dependency-injection, no clock — and it never faults. It resolves in five phases, first match wins. Whatever phase decides, your budgets and cost recording still apply — and every phase except a directly-named model resolves to a tier's ordered fallback chain.
1model fieldA direct model id, a provider-prefixed id, or a tier name in the request body — honored first.
2x-polyrouter-tierThe built-in tier header. Highest-precedence header — beats every other header rule.
3header rulesRules on other headers, matched in priority order — the first match wins.
4default ruleThe system's configured default routing rule, if one is set.
5default tierThe guaranteed catch-all — the default tier's entry chain. model:"auto" that reaches here hands off to the smart layers.
The model field takes three forms
Form
Example
Resolves to
Direct model id
gpt-5
That exact model on the single provider that exposes it — matched independently of tiers, with no fallback (a single target). If two providers expose the same id the request is a 404 ambiguous_model; qualify it as <providerId>:<model>. If the string matches no model but is a tier key, it resolves as that tier.
Provider-prefixed
<providerId>:gpt-5
That specific provider's copy of the model. <providerId> is the provider row's UUID — call GET /v1/models for the exact qualified ids.
Tier name
fast
The named tier's ordered entry chain.
Notex-polyrouter-tier has structural precedence: a non-empty value that resolves to a tier or remap rule wins regardless of any other header rule's priority — it beats every mechanism except the model field. It resolves in two sub-steps: first tier-header remap rules that match the sent value, then a direct lookup of an owned tier by that name. A matched-but-broken tier surfaces a clear error rather than silently rerouting. A value that matches neither a remap rule nor an owned tier key is advisory — the request silently falls through to the remaining phases and the default tier, with no error and no header recorded. The value match is exact and case-sensitive; tier keys are lowercase slugs.
Tiers & fallback chains
A tier is a named routing target. Exactly one tier — default — is seeded for you; fast and cheap below are tiers you would create. Separately, auto_low/auto_high are routing-rule match types (band targets) pointing at a tier you configure — they are not tiers. Each tier holds an ordered chain of up to five routing entries: position 0 is the primary, positions 1–4 are fallbacks. Manage them on the Routing page or via /api/routing/tiers and /api/routing/tiers/:tierId/entries — GET returns the ordered chain, PUT replaces it atomically; there is no per-entry create or delete.
Fallback chain · walked in position order until one responds
pos 0 · primaryclaude-sonnet-5
→
pos 1 · fallbackgpt-5-mini
→
pos 2 · fallbackllama-3.3-70b
Up to 5 entries — the chain is walked in position order.drag-to-reorder in the dashboard
What triggers a fallback
Fallback-eligible errors
authrate_limitunavailableunknown_modelcredentialcircuit open
The chain advances to the next entry — an upstream that's down or throttled shouldn't take the request with it. unknown_model is model-specific, so a model retired at one provider must not fail the request; credential (a revoked OAuth grant or an identity-provider outage) is eligible but breaker-neutral; circuit open skips the entry before the adapter is even built.
Terminal errors — no fallback
bad_requestcaller cancelledunclassified
Returned as-is. bad_request is the only provider kind that never falls back — a malformed request fails the same way on every entry. Caller cancellation is not a fault, and an error the router cannot classify is deliberately never retried.
How the chain is walked
1
Classify the error
Only fallback-eligible kinds advance; a terminal error stops here.
2
Check commit state
If the response already committed to a stream, no fallback — the model is locked.
3
Check the circuit breaker
If the next provider's breaker is open, skip that entry and keep walking.
4
Try the next entry
Walk the chain until one responds or the entries are exhausted.
NoteNever swapped mid-stream. Fallbacks happen freely before the first token. Once streaming begins the model is committed — an upstream failure ends the stream with a clear error, never a silent swap.
The auto_high & auto_low band targets
The smart layers don't pick a model directly — they pick a band, and each band points at a tier you configure. auto_low is the cheap tier; auto_high is the strong tier. Both are set on the Routing page's band-targets section, each showing its resolved chain (primary + fallback count) and flagging any degraded state — cascade needs both bands usable.
Smart layers & model:"auto"
Send model:"auto" and — if Layer 0 falls through to the default tier — the smart layers take over. They run in order, each one either routing the request or handing it down, and the whole stack lands on the default tier if every layer is disabled or unsure. Automatic routing is enhancement; it never becomes a new way for a request to fail.
GatingA layer runs only when all of these hold: the instance has it enabled; for Layer 2, the whole classifier is ready; the tenant has opted in (default-on when unset, except the learning loop which defaults off); and the request is model:"auto" that fell through Layer 0 to the default tier.
explicitLayer 0always on · never faults
Resolves the model field, tier header, header rules, and default rule. A named target routes immediately — only model:"auto" on the default tier continues down.
Refines only the ambiguous slice: embeds the request text and classifies it against anchor centroids. A confident band routes via auto_high/auto_low (decision_layer=semantic); otherwise it continues.
L2 ambiguous or skipped ↓
cascadeLayer 3opt-in · cheap-first
Runs the cheap tier, grades the result with a quality score, and escalates to the strong tier only if it fails — auto_high → fallback → default.
every layer off or unsure ↓
defaultDefault tierguaranteed catch-all
Where auto lands when the smart layers are unavailable or unsure. The request is never left without a route.
Turning layers on
One environment variable controls which layers exist on an instance. The default is structural; cascade and semantic both imply structural. All config is Zod-validated at boot — an unknown token rejects boot naming the offender, never a silent skip.
.env
# default is "structural"; cascade and semantic both imply structural
ROUTING_AUTO_LAYERS=structural,cascade,semantic
Layer 1 · structural
Structural classification is the cheap, language-neutral first pass — sub-millisecond, zero cost. It runs only for model:"auto" requests that reached the default tier, and scores a handful of local features to judge how heavy the task is.
The features it reads
Seven weighted sub-scores, each saturating at its own ceiling and combined into one score in [0, 1]. The default weights sum to 1 and are overridable with ROUTING_STRUCTURAL_WEIGHTS. Size is deliberately capped at 0.30 so a long prompt alone can never reach the top tier — high needs several signals at once.
Input size (characters, baseline-subtracted)0.30
Fenced code-block volume0.20
Tool count & parameter complexity0.20
Response-format / schema requirements0.10
Conversation depth (message count)0.10
Multimodal / vision content presence0.05
Requested max output tokens0.05
OverrideDeclared reasoning effort is not a weighted feature — it is a band rule that sits on top of the score. A maximal declaration (high/xhigh/max effort, or a thinking budget at or above saturation) forces the high band outright, and the reason string says so with declared=max. You asked for the heavy path; L1 does not second-guess it.
Per-agent baselines, subtracted
The system prompt is excluded from scoring outright; separately, each (agent, system-fingerprint) builds an EWMA of its own typical input size, subtracted from the size signal so routine traffic scores on the delta — the score reflects the task, not the boilerplate.
L1 structural · per-agent EMA baseline subtracted
sub-ms · $0.00
agent baseline · ~11,200 chars subtracted from the size signal
task
NoteThe declaration the override above reads comes from reasoning_effort (OpenAI), thinking budgets (Anthropic), or output_config.effort — whichever your protocol carries. A maximal declaration routes auto_high without further analysis. Undeclared requests classify byte-identically to before.
The three bands
highstructural
Complex request → route auto_high.
lowstructural
Simple request → route auto_low.
ambiguouscontinue
Uncertain → hand to Layer 2, then Layer 3 if still ambiguous.
Telemetry (structural_band / structural_score / structural_band_source / structural_epoch) is written for every evaluated row — even ambiguous ones that fall through — so nothing routes silently; the reason string is appended to routing_reason.
Layer 2 · semantic
New in v0.8
When Layer 1 lands on ambiguous, Layer 2 looks at what the request actually says. It embeds the request text with a small local model and classifies it against curated anchor centroids — turning a "this could go either way" into a confident high or low. It refines only the L1-ambiguous slice: it never re-scores a confident L1 band and never runs on a non-auto request.
How it works
01
Extract
Newest user turn first, bounded by char caps. System content is excluded; no evidence → skip.
02
Embed
A local ONNX model on CPU, ~5–20 ms, under a per-call deadline with bounded concurrency. Saturation → skip.
03
Classify
A three-band cosine score (simHigh − simLow) against anchor centroids. An invalid vector is a fault, never a band.
04
Decide
A confident band routes auto_high/auto_low; ambiguous or any fault degrades to cascade/default.
Band to route
Outcome
decision_layer
Action
high band, target resolves
semantic
Route auto_high — never cascades.
low band, target resolves
semantic
Route auto_low — never cascades.
Confident band, target missing
default
Verdict recorded; falls through to the default tier (mirrors L1 unroutable) — the Layer-0 decision is untouched, but the L2 verdict is still written to the semantic_* columns.
ambiguous band
downstream
Hand to Layer 3; the in-memory vector rides to the recorder for learning evidence.
Invalid / fault / unavailable
none
Skip to cascade or default — no verdict, no telemetry.
What the bands mean
high · reasoning-heavy
→ auto_high
Proofs, system designs, debugging, formal analyses — prompts that reward a stronger model.
low · quick
→ auto_low
Small talk, format conversions, lookups, rewrites — cheap and fast is the right call.
PrivacyNo prompt text or embedding vector is ever logged, persisted to Postgres, put in a metric, or returned by the API. The verdict reason is numbers only (s=hi=lo=src=). The first value that can ever reach Redis is a sum over at least MIN_COHORT embeddings — never a single raw vector.
Turning Layer 2 on
Default off — the baseline image ships no ONNX runtime and no model. There are two ways to enable it.
A · Batteries-included image
The -semantic variant pre-bakes a checksum-pinned reference MiniLM model and presets SEMANTIC_MODEL_PATH. Bring it up with a compose override.
B · Bring your own
Install the [email protected] peer, point SEMANTIC_MODEL_PATH at a v1 bundle (model.onnx + vocab.txt + manifest.json), and add semantic to ROUTING_AUTO_LAYERS.
$ docker compose -f docker-compose.yml -f docker-compose.semantic.yml up -d
NoteOnce an embedder is loaded, each tenant opts in per-account on the Routing page. The /api/routing/auto-layers endpoint reports semanticAvailable honestly — the whole classifier ready, not just a flag — so the toggle is a real control or honest "off instance-wide (set SEMANTIC_MODEL_PATH)" copy, never a dead switch. Reference model: all-MiniLM-L6-v2 (Apache-2.0, 384-dim).
Layer 2 · learning loop
New in v0.8
Optional and off by default: each tenant can let Layer 2 learn from its own traffic. The cascade's own outcomes weakly label the ambiguous requests it settles, and those labels nudge per-tenant learned centroids — always inside conservative rails, always revertible.
Cascade outcomes become weak labels
Cascade outcome
Weak label
Contributes?
Accepted — quality passed
low
yes
Escalated by the quality gate
high
yes
Escalated by a cheap provider fault
—
no
Cancelled / unknown quality
—
no
From outcome to centroid
1
Labelled embeddings accumulate in bounded volatile memory, grouped into cohorts.
2
A cohort flushes to Redis only as a sum of ≥ MIN_COHORT embeddings — never a single raw vector.
3
A daily sweep folds fresh evidence under rails — capped EMA, spherical drift clamp toward bundled, cooldown, revision match.
4
It applies crash-atomically — CAS to Postgres (generation bump, scalars-only audit) is authoritative, then the Redis stage is promoted.
The rails
Knob
Default
What it bounds
SEMANTIC_LEARNING_MIN_COHORT
8
Smallest sum that may reach Redis.
SEMANTIC_LEARNING_MIN_SAMPLES
50
Evidence needed before a fold.
SEMANTIC_LEARNING_ALPHA
0.2
EMA weight on fresh evidence.
SEMANTIC_LEARNING_MAX_DRIFT
0.35
Cosine distance the learned centroid may move from bundled.
SEMANTIC_LEARNING_COOLDOWN_H
24
Minimum hours between applies.
NoteLearned centroids supersede bundled only under read-time gates — learning on, and the stored (epoch, generation, revision) matching the decision-time gate. Any Redis fault, stale state, or gate mismatch falls back to bundled, never a skip.
Revert & honest source
One click reverts: it bumps a revocation epoch in Postgres first, which fences every in-flight sweep and stale reader before the Redis keys are even cleared. A learned centroid whose embedder or revision moved under it honestly reports source: bundled — the card never claims "learned" while the router is actually on bundled anchors.
Semantic learning
learning from 12 low · 5 highactive: bundled anchorsapplied 7/3/2026Revert to bundled
Audit rows (apply / discard_revision / revert) carry scalars only — counts, drift, and similarity — never text or vectors.
Layer 3 · cascade
Cascade is the last automatic layer, and it runs only when both L1 and L2 came back ambiguous (or L2 skipped). Its bet is simple: try the cheap tier first, check the result, and only pay for the strong tier if the cheap answer isn't good enough.
① cheapauto_lowwith timeout
Run the cheap tier and buffer the response before any tokens stream to the client.
② quality score ↓
score ≥ 0.5 · accept
Replay the buffered cheap response — done, no escalation.
score < 0.5 · escalate
③ Strong tier auto_high → fallback → default tier.
The quality gate
The score is a three-valued lattice — 0, 0.5, or 1. A response scores 0 on an error or content_filter stop, empty content, malformed tool arguments, or prose where the request demanded machine-parseable output; 0.5 on a length truncation with no hard failure; otherwise 1. Accept at or above the threshold (default 0.5), escalate below it.
Sharper gateRefinements: unparseable structured output escalates; a length-truncated answer grades 0.5; tool-calling turns are exempt from the empty/format checks.
Commit boundary
The cheap response is buffered and graded before streaming begins — the same commit rule as the main proxy, so no model is ever swapped after the first token.
Per-attempt cost ledger
Every attempt in an escalation is recorded to its own ledger (request_attempt), so an escalated request shows what each hop actually cost.
Per-tenant self-calibration
Opt-in and conservative: a daily sweep narrows your ambiguous band inward from your own outcomes, so more requests get a confident L1 answer over time. It's rail-bounded, fully audited, and one click reverts it — and it's degrade-shaped, so a poisoned or stale calibrated pair can never break or stall routing.
Calibration only ever contracts the ambiguous band
low → auto_low
→ ambiguous ←
high → auto_high
The edges move inward — never outward — under a drift cap, keeping a minimum gap between the two thresholds.
When a calibrated pair applies
The effective-threshold function is pure and fail-safe. A stored pair is used only when it's complete, finite, ordered, anchored to the current instance defaults (exact-float), and clean under the current rails — contraction direction, drift cap, minimum gap. Anything else reads as the instance defaults.
NoteOne-click revert (user-wins) snaps the calibrated pair back to the instance defaults. Escalations also record why they happened — quality_gate vs cheap_error — so the narrowing has honest signal to learn from.
Decision trail & inspector
Every routing decision is recorded so you can see exactly why a request went where it did. The request_log.routing_reason column carries the reason of the layer that actually decided. When the smart layers evaluated a request but did not route it — a default fall-through, or a cascade — their verdicts are appended behind a ; in L1 → L2 order, never overwritten. The dashboard's Requests inspector renders the whole string.
The ordered routing_reason
Scenario
decision_layer
routing_reason
Explicit model
explicit
explicit model gpt-5 — a tier name in the model field records explicit tier <key>
When a header decides, the inspector shows which one — but only what's safe to store. The built-in tier header records the owned config value that matched — the tier key on a direct lookup, or the remap rule's own header value (e.g. shopping) when a tier-header remap matched; never raw client bytes. A rule on any other header records the name only, because a configured rule value can itself be a credential. Fail-closed by design.
Metadata only by default — tokens, cost, routing decision, latency. Prompt and response bodies are never stored unless you turn on encrypted body capture (self-host only, off by default); Layer 2's embedding vectors are never persisted at all.
The inspector also surfaces the matched routing header, the aggregated attempt cost for escalated or fallback requests, and the semantic source provenance chip. Earlier chain failures appear as a ; fell back after: <kind>@<model> suffix on the routing reason. See Security & privacy for the full data model.