polyrouter's routing engine is a layered, degradable pipeline. Layer 0 explicit routing is the reliable core that always works; the automatic layers refine model:"auto" and always fall back to explicit or default. L1 structural is on by default; L2 semantic and L3 cascade are opt-in. Alongside the difficulty band, an auto request also records a workload class — what kind of work it is — which can claim the request outright or scope the bands to that class. Every smart-layer fault degrades to exactly the path it would have taken with that layer disabled. The smart path never fails or stalls a request.
Maintained by Anthony Izzo · Last updated
Layer 0 is a pure function — no database, no dependency-injection, no clock — and it never faults. It resolves in five phases, first match wins. Whatever phase decides, your budgets and cost recording still apply — and anything that resolves to a tier gets that tier's ordered fallback chain. A target that names a single model — the model field, but equally a header rule, a default rule, a band rule or a workload rule carrying model:<id> — resolves to one attempt with no fallback.
| Form | Example | Resolves to |
|---|---|---|
| Direct model id | gpt-5 | That exact model on the single provider that exposes it — matched independently of tiers, with no fallback (a single target). If two providers expose the same id the request is a 404 ambiguous_model; qualify it as <providerId>:<model>. If the string matches no model but is a tier key, it resolves as that tier. |
| Provider-prefixed | <providerId>:gpt-5 | That specific provider's copy of the model. <providerId> is the provider row's UUID — call GET /v1/models for the exact qualified ids. |
| Tier name | fast | The named tier's ordered entry chain. |
A tier is a named routing target. Exactly one tier — default — is seeded for you; fast and cheap below are tiers you would create. Separately, auto_low, auto_high and auto_workload are routing-rule match types pointing at a tier or model you configure — they are not tiers. The two band types may additionally carry a workload-class scope; see Workload routing. Each tier holds an ordered chain of up to five routing entries: position 0 is the primary, positions 1–4 are fallbacks. Manage them on the Routing page or via /api/routing/tiers and /api/routing/tiers/:tierId/entries — GET returns the ordered chain, PUT replaces it atomically; there is no per-entry create or delete.
The smart layers don't pick a model themselves — they pick a band, and each band points at a target you configure: normally a tier (so the band inherits its fallback chain), or a single model if you pick one, in which case there is no chain behind it. auto_low is the cheap tier; auto_high is the strong tier. Both live in the Band targets card of the Routing page's Auto section, each showing its resolved chain (primary + fallback count) and flagging any degraded state — cascade needs both bands usable. The same card carries a per-workload strong/cheap pair per class: when a request has a workload class, that class's scoped rows decide for it and the generic rows apply only where no scoped row exists — see Workload routing.
Send model:"auto" and — if Layer 0 falls through to the default tier — the smart layers take over. Layer 1 classifies first, producing two verdicts: a difficulty band and a workload class. A class with a target of its own claims the request there and then; otherwise the band resolves, Layer 2 refines what is still ambiguous, and Layer 3 cascades. The whole stack lands on the default tier if every layer is disabled or unsure. Automatic routing is enhancement; it never becomes a new way for a request to fail.
One environment variable controls which layers exist on an instance. The default is structural; cascade and semantic both imply structural. All config is Zod-validated at boot — an unknown token rejects boot naming the offender, never a silent skip. Workload routing has no token of its own: the structural classes ride structural and the semantic classes ride semantic.
# default is "structural"; cascade and semantic both imply structural
ROUTING_AUTO_LAYERS=structural,cascade,semanticStructural classification is the cheap, language-neutral first pass — sub-millisecond, zero cost. It runs only for model:"auto" requests that reached the default tier, and scores a handful of local features to judge how heavy the task is. The same feature vector yields a second, independent verdict — the request's workload class (Workload routing) — computed once and recorded alongside the band.
Seven weighted sub-scores, each saturating at its own ceiling and combined into one score in [0, 1]. The default weights sum to 1 and are overridable with ROUTING_STRUCTURAL_WEIGHTS. Size is deliberately capped at 0.30 so a long prompt alone can never reach the top tier — high needs several signals at once.
The system prompt is excluded from scoring outright; separately, each (agent, system-fingerprint) builds an EWMA of its own typical input size, subtracted from the size signal so routine traffic scores on the delta — the score reflects the task, not the boilerplate.
Telemetry (structural_band / structural_score / structural_band_source / structural_epoch, plus the workload quad workload_class / workload_score / workload_source / workload_revision) is written for every evaluated row — even ambiguous ones that fall through — so nothing routes silently; the reason string is appended to routing_reason. The quad is all-or-nothing and commits atomically with the structural columns: a database constraint rejects a partly-filled workload verdict.
The bands answer how hard. A workload class answers what kind. Every evaluated auto request records one, and you can act on it two ways: give a whole class its own target, or give a class its own strong/cheap band pair. Detection is never keyword-based — the structural classes fall out of the features Layer 1 already computed, the reserved classes out of the Layer-2 embedding. Unset classes change nothing.
| Class | Source | Fires when |
|---|---|---|
| vision | structural | An image content block is present in the scored window. |
| structured | structural | The request declares a JSON output format (OpenAI response_format or Anthropic output_config.format). |
| code | structural | Fenced-code chars are at least codeShare of the counted text and at least codeMinChars absolute — 0.30 and 200 by default. |
| research | semantic | The embedding's nearest class anchor, when it beats the runner-up by SEMANTIC_WORKLOAD_MARGIN (0.05) and clears SEMANTIC_WORKLOAD_MIN_SIM (0.20). |
| writing | semantic | Same rails as research; the semantic source never emits a structural class. |
| none | either | Nothing fired. Recorded for telemetry; it never claims and never scopes a band. |
When several structural signals fire, one class wins, in this order: vision first, then structured, then code. The structural source keeps precedence over the semantic one, which is consulted only when structural came back none. The taxonomy is fixed — a new class is a versioned change, never configuration.
Both sources emit a numbers-and-class-names-only reason; the deciding one rides the same atomic commit as the structural verdict.
workload:code score=0.42 share=0.42 codechars=1180 mm=0 rf=0
workload:research score=0.4871 m=0.0912 sim2=0.3959 top=research top2=writing src=semanticA scoped decision appends scope=<class> as the last fragment of routing_reason — after the quality marker, the fall-back trail and the classification trail — so it is always a reliable suffix to test for.
# known keys merge over the defaults; an unknown key or an out-of-range value rejects boot
ROUTING_WORKLOAD_THRESHOLDS={"codeShare":0.3,"codeMinChars":200}When Layer 1 lands on ambiguous, Layer 2 looks at what the request actually says. It embeds the request text with a small local model and classifies it against curated anchor centroids — turning a "this could go either way" into a confident high or low. It refines only the L1-ambiguous slice: it never re-scores a confident L1 band and never runs on a non-auto request. One thing did widen in v0.15: the embedding itself also feeds the semantic workload source, so a request whose structural workload came back none is embedded even when its band was confident — one vector, used for both, never two embeds.
| Outcome | decision_layer | Action |
|---|---|---|
| high band, target resolves | semantic | Route auto_high — never cascades. |
| low band, target resolves | semantic | Route auto_low — never cascades. |
| Confident band, target missing | default | Verdict recorded; falls through to the default tier (mirrors L1 unroutable) — the Layer-0 decision is untouched, but the L2 verdict is still written to the semantic_* columns. |
| ambiguous band | downstream | Hand to Layer 3; the in-memory vector rides to the recorder for learning evidence. |
| Invalid / fault / unavailable | none | Skip to cascade or default — no verdict, no telemetry. |
Default off — the baseline image ships no ONNX runtime and no model. There are two ways to enable it.
ROUTING_AUTO_LAYERS=structural,cascade,semantic
SEMANTIC_MODEL_PATH=/models/all-MiniLM-L6-v2 # bundle: model.onnx + vocab.txt + manifest.jsonOptional and off by default: each tenant can let Layer 2 learn from its own traffic. The cascade's own outcomes weakly label the ambiguous requests it settles, and those labels nudge per-tenant learned centroids — always inside conservative rails, always revertible.
| Cascade outcome | Weak label | Contributes? |
|---|---|---|
| Accepted — quality passed | low | yes |
| Escalated by the quality gate | high | yes |
| Escalated by a cheap provider fault | — | no |
| Cancelled / unknown quality | — | no |
| Knob | Default | What it bounds |
|---|---|---|
| SEMANTIC_LEARNING_MIN_COHORT | 8 | Smallest sum that may reach Redis. |
| SEMANTIC_LEARNING_MIN_SAMPLES | 50 | Evidence needed before a fold. |
| SEMANTIC_LEARNING_ALPHA | 0.2 | EMA weight on fresh evidence. |
| SEMANTIC_LEARNING_MAX_DRIFT | 0.35 | Cosine distance the learned centroid may move from bundled. |
| SEMANTIC_LEARNING_COOLDOWN_H | 24 | Minimum hours between applies. |
One click reverts: it bumps a revocation epoch in Postgres first, which fences every in-flight sweep and stale reader before the Redis keys are even cleared. A learned centroid whose embedder or revision moved under it honestly reports source: bundled — the card never claims "learned" while the router is actually on bundled anchors.
Audit rows (apply / discard_revision / revert) carry scalars only — counts, drift, and similarity — never text or vectors.
Cascade is the last automatic layer, and it runs only when no workload target claimed the request and both L1 and L2 came back ambiguous (or L2 skipped). Its bet is simple: try the cheap tier first, check the result, and only pay for the strong tier if the cheap answer isn't good enough. When the request carries a workload class with its own band pair, both legs stay inside that class's scope.
The score is a three-valued lattice — 0, 0.5, or 1. A response scores 0 on an error or content_filter stop, empty content, malformed tool arguments, or prose where the request demanded machine-parseable output; 0.5 on a length truncation with no hard failure; otherwise 1. Accept at or above the threshold (default 0.5), escalate below it.
Opt-in and conservative: a daily sweep narrows your ambiguous band inward from your own outcomes, so more requests get a confident L1 answer over time. It's rail-bounded, fully audited, and one click reverts it — and it's degrade-shaped, so a poisoned or stale calibrated pair can never break or stall routing.
The effective-threshold function is pure and fail-safe. A stored pair is used only when it's complete, finite, ordered, anchored to the current instance defaults (exact-float), and clean under the current rails — contraction direction, drift cap, minimum gap. Anything else reads as the instance defaults.
Every routing decision is recorded so you can see exactly why a request went where it did. The request_log.routing_reason column carries the reason of the layer that actually decided. When the smart layers evaluated a request but did not route it — a default fall-through, or a cascade — their verdicts are appended behind a ; in L1 → L2 order, never overwritten. When the workload stage claims the request, that verdict's reason is the recorded reason. And a decision made by a class-scoped band always ends with scope=<class> — appended last, after the quality marker, the fall-back trail and the classification trail, so the suffix is always testable. The dashboard's Requests inspector renders the whole string.
| Scenario | decision_layer | routing_reason |
|---|---|---|
| Explicit model | explicit | explicit model gpt-5 — a tier name in the model field records explicit tier <key> |
| Tier header | header | x-polyrouter-tier: fast |
| Tier-header remap, or a rule on any other header | header | header rule x-polyrouter-tier |
| Workload claim · structural class | workload | workload:code score=0.42 share=0.42 codechars=1180 mm=0 rf=0 |
| Workload claim · semantic class | workload | workload:research score=0.4871 m=0.0912 sim2=0.3959 top=research top2=writing src=semantic |
| L1 confident | structural | structural:high score=0.69 size=0.90 code=0.75 tools=0.50 schema=1.00 depth=0.40 mm=0.00 maxtok=0.60 think=-- rf=0.00 |
| L2 confident | semantic | semantic:low s=-0.1845 hi=0.3021 lo=0.4866 src=bundled |
| Class-scoped band decided | structural | structural:high score=0.71 … rf=0 — the scoped pair decided; the suffix is always last: scope=code |
| Cascade escalation | cascade | cascade: escalated cheap→strong (q=0.31); structural:ambiguous … ; semantic:ambiguous … src=bundled |
Two more fragment families ride the same column. Earlier chain failures append ; fell back after: <kind>@<model> — a member never contacted because its provider's circuit was open records skip@<model>, deliberately outside the error taxonomy. The output-cap guardrails append output_cap_clamped <ask>→<cap> (<model>) when a requested max-output is clamped to the model's known cap, and output_cap_deferred <model>(<cap><<ask>) when a chain member is passed over because its cap is smaller than the ask. One literal note on the cascade row above: cascade: escalated cheap→<tier> names the tier that actually served — the strong tier normally, the default tier when the reliable core rescued the request.
When a header decides, the inspector shows which one — but only what's safe to store. The built-in tier header records the owned config value that matched — the tier key on a direct lookup, or the remap rule's own header value (e.g. shopping) when a tier-header remap matched; never raw client bytes. A rule on any other header records the name only, because a configured rule value can itself be a credential. Fail-closed by design.
| Request | header name | value |
|---|---|---|
| x-polyrouter-tier: fast | x-polyrouter-tier | fast (owned tier key) |
| x-polyrouter-tier: shopping | x-polyrouter-tier | shopping (rule value; target tier heavy) |
| x-env: prod → tier fast | x-env | null (never recorded) |
| explicit model / default | null | null |
Metadata only by default — tokens, cost, routing decision, latency. Prompt and response bodies are never stored unless you turn on encrypted body capture (self-host only, off by default), in which case the drawer grows a Payload block; Layer 2's embedding vectors are never persisted at all.
The Decision block also surfaces the matched routing header, an escalated marker, the cascade's quality signal, the semantic source provenance chip and the workload chip — which reads · routed when that class claimed the request — and the aggregated attempt cost covers escalated or fallback requests. Every earlier attempt is listed line by line in the Fallback trail block: the mapped error kind with its HTTP status, or a plain "skipped" note where an open breaker meant the provider was never contacted. See Security & privacy for the full data model.