polyrouter
GitHub Self-host it Sign up
How it works Features Docs Compare Blog GitHub
Self-host it Sign up
Features

Everything polyrouter does.

A production LLM gateway in one container: deterministic routing, real reliability, cost control that blocks, dual-protocol translation, and privacy by construction.

Routing Semantic routing Workload routing Reliability Cost & budgets Protocols Subscriptions Security Operations
01 · routing

Deterministic by default, smart on demand.

Precedence is first-match-wins, so behavior is always predictable. Opt into tiers and auto when you want the router to choose — it reads both how hard a request is and what kind of work it is, and it always degrades back to something explicit.

01
Explicit model
A named model is always honored — the reliable core.
02
Tier header
x-polyrouter-tier pins a request to a tier's chain.
03
Header rules
A rule on any other header (highest priority first, then oldest), then a default-match rule.
04
default tier
The guaranteed catch-all every path can fall back to.
model: "auto" is not a precedence step — the smart layers refine a request that already landed on the default tier. One local pass yields two verdicts: how hard the request is, and what kind of work it is.
A configured workload target claims the request first; otherwise the band targets decide, then L2 semantic, then L3 cascade.
L1 structural runs by default; L2 and L3 are opt-in via ROUTING_AUTO_LAYERS. A disabled, skipped or faulted layer leaves that decision untouched.
L1 structural runs sub-millisecond on local features — the harness system prompt is excluded from scoring outright, and each agent's own rolling size baseline is subtracted, so a huge boilerplate prompt can't force everything into the top tier.
L2 semanticopt-in
A local ONNX model (CPU, ~5–20 ms) embeds the request once and classifies it against bundled anchor centroids.
One embed, two reads: for the slice L1 leaves ambiguous, a confident band verdict routes via auto_high / auto_low — the class-scoped pair when the deciding class has one — and stamps the row semantic.
The same vector also supplies the reserved research / writing workload classes for any request L1 read as no specialist workload, behind a margin and a minimum-similarity rail.
Anything unsure hands straight down to cascade; a failed embed simply skips both reads.
Privacy: no prompt text or embedding vector is ever logged, persisted to Postgres, or returned by the API.
L2 learning loopoff by default
Opt in and each tenant grows its own learned centroids from outcome-labeled traffic. Embeddings aggregate in memory and reach Redis only as a sum of at least SEMANTIC_LEARNING_MIN_COHORT (default 8) — never a single raw vector.
One-click revert fences stale state, and a centroid that drifted honestly shows bundled — never a fake "learned" badge.
Workload targetsroutes when set
The same local pass also reads what kind of work a request is: code (fenced code above the configured thresholds), vision (an image in the request), structured (a declared JSON output format). When several fire, one wins: vision, then structured, then code.
Bind a class to its own tier or model and it claims its auto requests before band targets, L2 and the cascade — stamped workload.
An explicit model or the tier header still wins; unset classes change nothing.
Detection is structural, never keyword matching. research and writing are reserved for the optional semantic source, which reuses the one embed L2 already takes.
Per-workload bandsoptional
For difficulty-aware routing within a class, give it its own strong/cheap pair: a scoped auto_high / auto_low is consulted before the generic rows for that class only, each band falling back to the generic target independently, and the cascade stays inside the same scope.
The recorded reason ends with scope=<class>.
A scoped target that is empty or unresolved makes that band unroutable for the class — the request falls through to default rather than silently borrowing the generic target.
Calibration A daily rail-bounded sweep can narrow each tenant's ambiguous band from its own outcomes — conservative, audited, and one click from reverted. It is degrade-shaped: a stale or poisoned pair simply reads as the instance defaults, so calibration can never fail or stall a request.
How a decision is made
diagram — not a dashboard screen
Precedence · first match wins
1explicit modelnot set
2x-polyrouter-tierabsent
3header ruleno match
4default-match rulenone
5default tiermatched
Landing on default — by rule or by the seeded tier — is what lets auto refine: workload target, then band targets (a class's own pair first), then L2, then the cascade. A request that matched at steps 1 to 3 never enters that path.
Decision trace
Incoming request
model:"auto" · 12,408 chars in
L1 structural
system excluded · baseline subtracted
Workload verdict
workload · code (structural) · no target set
Tier selected
balanced
Responded
claude-sonnet-5 · 200 ok
L1 structural · agent size baseline subtracted
sub-ms · $0.00
agent baseline · ~11,200 chars subtracted from the size signal
delta
localhost:3001
Routing
tiers, fallbacks & auto layers
New tier key
e.g. heavy
Add tier
balancedEveryday work
drag to reorder · max 5
Primaryclaude-sonnet-5$3 / $15 per 1M×
Fallback 1gpt-5-mini$0.15 / $0.6 per 1MMake primary×
Fallback 2llama-3.3-70bfreeMake primary×
cheapBulk & drafts
drag to reorder · max 5
Primarygpt-5-mini$0.15 / $0.6 per 1M×
Fallback 1llama-3.3-70bfreeMake primary×
Auto Tuning Rules
Automatic routing
Applies only when an agent asks for model auto. Explicit requests always win.
L1 · Structural<1ms, localon
Language-neutral features; system prompts fingerprinted and subtracted.
L2 · Semanticlocal embedon
Embedding classifier over the ambiguous slice — one shared embed also supplies the research / writing workloads.
L3 · Cascadecheap-firstoff
Ambiguous requests try the cheap model and escalate on a failed quality check.
If a smart layer is down, auto degrades to the default tier. Requests never fail because routing got clever.
Band targets
Route confident verdicts directly; the cascade tries cheap first for the rest.
STRONGauto_highTier balanced
CHEAPauto_lowTier cheap
PER-WORKLOAD BANDScode
a class's own strong/cheap pair — consulted before the generic rows for that class only, each band falling back to the generic target independently
STRONGauto_high · codeModel gpt-5
CHEAPauto_low · codeNot set — code requests use the generic cheap target
Busy/error state is per band (strong / cheap) across scopes. The Auto-performance savings basis stays the generic strong target.
Workload targets
Send a detected workload to its own tier or model. A set class claims its auto requests before band targets, Layer 2 and the cascade; an explicit model or the tier header still wins. Unset classes change nothing.
CODEauto_workload · codeNot set — the per-workload bands above decide
VISIONauto_workload · visionModel claude-sonnet-51,204 routed
STRUCTUREDauto_workload · structuredNot set — these requests follow the complexity path
RESERVED · SEMANTIC SOURCE
RESEARCHauto_workload · researchNot set
WRITINGauto_workload · writingNot set
Detection is structural, never keyword matching. For difficulty-aware routing inside a class, use the per-workload bands above.
The Routing page: a tier's ordered chain with a primary and up to four fallbacks, each showing its per-million input and output price, reorderable by drag.
02 · reliability

Built to stay up.

Circuit breakers
Per-provider, Redis-backed and shared across instances. Hung connects and stalled reads trip cleanly, and the half-open probe runs and leases at doubled patience (capped at 1 h) so a slow-but-healthy provider can actually close its breaker instead of skip-looping.
Safe mid-stream
Fallbacks happen freely before the first token. Once streaming begins the model is committed — an upstream failure ends the stream with a clear error, never a silent swap.
Graceful drain
In-flight streams drain on shutdown — the app gives them a 15 s deadline inside Compose's 45 s stop_grace_period — and streaming applies backpressure, so deploys don't sever live completions.
Fallback trail
Every attempt is on the record: a breaker-skipped member reads skip@model — never a fabricated failure — and error rows keep structured per-attempt metadata, both cascade legs included, rendered in the inspector as a step-by-step trail. No free text, no guessing which model actually answered.
Output-cap guardrails
A router-chosen chain skips members whose known output limit cannot satisfy the ask, and clamps with an honest length finish when nothing can — instead of dying on a guaranteed provider 400.
The routing reason records output_cap_deferred or output_cap_clamped. Explicit models are never second-guessed.
03 · cost & budgets

Know the cost. Cap the spend.

Every request stores its unit-price snapshot at request time, so history is immutable. Budgets are enforced with atomic Redis counters that stay correct across every proxy instance.

Immutable cost records
Later catalog updates never rewrite the past. Missing provider usage is flagged ~estimated, never silently nulled.
Budgets that block
Day / week / month windows, global or per-agent, set to alert-or-block at the threshold. Each budget also declares what it meters: cash (money actually owed) or notional (additionally counting prepaid subscription traffic at the vendor's API rate).
Auto-performance view
Band mix over time, the four-way cascade outcome split, and an estimated net-savings figure with its coverage shown — never a fabricated $0 —
Alongside that, the L2 semantic slice (evaluated, routed per band, bundled vs learned).
A per-agent L1 signal-quality read that names any agent whose structural score has collapsed to a near-constant.
A Workload mix breaks classified auto traffic down by class — share, requests, how many a Workload target actually claimed, and spend, with unpriced and coverage disclosed.
Savings stay pinned to the generic strong target, and say so when class-scoped bands exist.
Pricing that stays current
A daily automatic refresh from LiteLLM's public catalog keeps prices honest (on by default, self-host). New prices apply to new requests only — recorded costs never change.
localhost:3001
Limits
budgets that alert or block
New budget
Spend counters are atomic across instances — a blocked budget stops requests everywhere at once.
Team monthly
Block
$500.00/ monthGlobal
hard stop — requests rejected at limit
EditDeleteenabled
CI daily cap
Alert
$40.00/ dayAgent · ci-bot
notifies: ops-email
EditDeleteenabled
Batch weekly
Block
$60.00/ weekAgent · batch-01
notifies: ops-email, #alerts
EditDeleteenabled
Experiments
Alert
$120.00/ weekGlobal
no channels wired
EditDeletedisabled
The Limits page: budgets that alert or block, with spend counters that are atomic across instances, so a blocked budget stops requests everywhere at once.
04 · protocols & providers

OpenAI and Anthropic, both directions.

Any SDK that accepts a base URL works unchanged. Cross-protocol requests go through a translation core: multi-turn tool calls, system prompts, stop reasons, and usage carry across — locked by a golden-file contract suite. Protocol-specific controls (cache_control, thinking, output_config on Anthropic; response_format, reasoning_effort on OpenAI; image_url.detail) are carried opaquely on their own protocol and deliberately dropped — never mis-mapped — when crossing to the other.

OPENAI SDK /v1
client = OpenAI(
  base_url="https://you/v1",
  api_key="poly_…")

client.chat.completions.create(
  model="auto",
  messages=[…])
ANTHROPIC SDK /
client = Anthropic(
  base_url="https://you",
  api_key="poly_…")

client.messages.create(
  model="auto", max_tokens=256,
  messages=[…])
/v1/chat/completions /v1/messages /v1/models streaming & non-streaming

Bring a subscription, not just a key.

OAuth

Connect a Claude Pro/Max or ChatGPT Plus/Pro plan as a provider through a guided PKCE flow — sign in at the provider, paste the redirect back, done. Tokens land in an encrypted typed envelope and auto-refresh before they expire (single-flight, advisory-locked, rotation-safe). If a grant goes bad the card shows a one-click Reauthorize while your fallback chain keeps serving.

Claude speaks anthropic_compatible; ChatGPT speaks the streaming-only openai_responses backend, buffered transparently — three identity headers only, never a spoofed client.
ToS heads-up: driving flat-rate consumer subscriptions programmatically can violate provider terms — pair one with a pay-per-token fallback (the dashboard surfaces the risk).

Every provider kind, one adapter.

Provider kinds
api_key BYOK (OpenAI, Anthropic, or any compatible), subscription OAuth, custom — any OpenAI- or Anthropic-compatible base URL, and self-host-only local models (the one kind allowed to reach 127.0.0.1 — Ollama, LM Studio, llama.cpp).
Credentials are AES-256-GCM encrypted and decrypted only at call time.
Per-provider max-tokens
max_tokens_spelling picks the output-cap wire field — auto sends max_tokens to local gateways and max_completion_tokens everywhere else. Inbound accepts either; outbound emits exactly one, so a legacy gateway never silently drops your cap.
Per-provider timeouts
first_byte_timeout_ms / idle_timeout_ms (1 s–1 h, in a provider's Advanced options) give research-class models — Deep Research, Opus with thinking — the patience their long prefill needs without touching global defaults.
OpenRouter attribution
Requests to an openrouter.ai provider carry HTTP-Referer and X-OpenRouter-Title so polyrouter shows up in OpenRouter's rankings — non-secret, sent only to OpenRouter, never affecting auth.
05 · security & privacy

Private and locked down.

Metadata-only by default
The request log keeps tokens, model / provider ids, cost snapshots, routing metadata and latency — no prompt or response text.
A workload-classified request adds its class, score, source and a configuration digest to that row: what kind of work it was, never the text that said so.
A full database breach exposes no conversation content.
Opt-in body capture
Self-host only, off by default: turn on prompt / response capture behind an explicit consent gate — encrypted at rest, retention-bounded, with per-agent overrides. Cloud instances never capture.
Encrypted at rest
Provider and notification-channel credentials are encrypted before they touch the database.
SSRF-guarded egress
Every user-supplied URL is resolved and checked against private / loopback / link-local / metadata ranges — IPv6 included, with DNS-rebinding defense.
Tenant isolation
Every entity access is ownership-scoped through a central guard, covered by a dedicated e2e suite.
User administration
First-signup-wins claims the instance, then registration is invite-only.
Disabling a user deletes every session in one transaction and immediately blocks every agent key — the guard rejects any key whose owner is disabled.
Re-enabling restores them; rotate a key if you need it permanently dead. The last enabled admin is undeletable.
Two credential planes
Dashboard sessions (Better Auth, slow-hashed, optional OAuth) are kept separate from agent API keys (poly_…, HMAC-SHA256 + prefix lookup — fast per-request, never bcrypt on the hot path).
Layer 2 The optional semantic layer touches request text, so it ships a tight contract: no raw embedding ever reaches a Postgres column, a log, a metric, or an API response; Redis only ever sees a sum of at least SEMANTIC_LEARNING_MIN_COHORT embeddings; and a one-click revert fences every learned centroid. Every invariant above is enforced in code and pinned by dedicated suites (evidence-accumulator, learning-store-redis, learning.run, semantic-learning.e2e).
06 · operations

One container. Full visibility.

one containerSPA, API and proxy on one port, next to Postgres 16 + Redis.
metricsPrometheus /metrics + opt-in OpenTelemetry traces.
notificationsSMTP and Apprise channels for budget, provider-down and failure-spike alerts — queued off the request path.
multi-arch imageTagged releases publish an amd64 + arm64 image to GHCR — skip the local build.
-semantic imageA batteries-included Layer-2 variant bakes in the ONNX runtime and a checksum-pinned reference MiniLM model. The baseline image stays ORT- and model-free.
provider timeoutsPer-provider first-byte / idle bounds (1 s–1 h) let research-class models finish instead of tripping the global 30 s prefill limit.
ci/cdBuild, lint, typecheck and e2e against real Postgres + Redis on every push.
accessibleKeyboard-operable, WCAG-checked contrast, reduced-motion aware, and fully responsive down to a phone — an icon rail below 768 px, tables that reflow to stacked records, bottom-sheet dialogs, safe-area and on-screen-keyboard handling, and a 24 px minimum hit target that grows to 44 px on a touch pointer. All of it pinned by regression suites.

Run it yourself in minutes.

Docker with Compose v2 is all you need.

Read the quickstart View on GitHub