polyrouter
GitHub ↗ Self-host it Log in
How it works Features Docs Compare GitHub ↗
Self-host it Log in
Features

Everything polyrouter does.

A production LLM gateway in one container: deterministic routing, real reliability, cost control that blocks, dual-protocol translation, and privacy by construction.

Routing Semantic routing Reliability Cost & budgets Protocols Subscriptions Security Operations
01 · routing

Deterministic by default, smart on demand.

Precedence is first-match-wins, so behavior is always predictable. Opt into tiers and auto when you want the router to choose — and it always degrades back to something explicit.

01
Explicit model
A named model is always honored — the reliable core.
02
Tier header
x-polyrouter-tier pins a request to a tier's chain.
03
Header rules
A rule on any other header (highest priority first, then oldest), then a default-match rule.
04
default tier
The guaranteed catch-all every path can fall back to.
model: "auto" is not a precedence step — the smart layers refine a request that already landed on the default tier. L1 structural runs by default; L2 semantic and L3 cascade are opt-in via ROUTING_AUTO_LAYERS. A disabled, skipped or faulted layer leaves that decision untouched.
L1 structural runs sub-millisecond on local features — the harness system prompt is excluded from scoring outright, and each agent's own rolling size baseline is subtracted, so a huge boilerplate prompt can't force everything into the top tier.
L2 semanticopt-in
Only the slice L1 leaves ambiguous is embedded by a local ONNX model (CPU, ~5–20 ms) and classified against bundled anchor centroids. A confident read routes via auto_high / auto_low and stamps the row semantic; anything unsure hands straight down to cascade.
Privacy: no prompt text or embedding vector is ever logged, persisted to Postgres, or returned by the API.
L2 learning loopoff by default
Opt in and each tenant grows its own learned centroids from outcome-labeled traffic. Embeddings aggregate in memory and reach Redis only as a sum of at least MIN_COHORT (default 8) — never a single raw vector.
One-click revert fences stale state, and a centroid that drifted honestly shows bundled — never a fake "learned" badge.
Calibration A daily rail-bounded sweep can narrow each tenant's ambiguous band from its own outcomes — conservative, audited, and one click from reverted. It is degrade-shaped: a stale or poisoned pair simply reads as the instance defaults, so calibration can never fail or stall a request.
How a decision is made
diagram — not a dashboard screen
Precedence · first match wins
1explicit modelnot set
2x-polyrouter-tierabsent
3header ruleno match
4default-match rulenone
5default tiermatched
Decision trace
Incoming request
model:"auto" · 12,408 chars in
L1 structural
system excluded · baseline subtracted
Tier selected
balanced
Responded
claude-sonnet-5 · 200 ok
L1 structural · agent size baseline subtracted
sub-ms · $0.00
agent baseline · ~11,200 chars subtracted from the size signal
delta
localhost:3000
Routing
tiers, fallbacks & auto layers
New tier key
e.g. heavy
Add tier
balancedEveryday work
drag to reorder · max 5
Primaryclaude-sonnet-5$3 / $15 per 1M×
Fallback 1gpt-5-mini$0.15 / $0.6 per 1MMake primary×
Fallback 2llama-3.3-70bfreeMake primary×
02 · reliability

Built to stay up.

Circuit breakers
Per-provider, Redis-backed and shared across instances. Half-open probes survive long LLM streams; hung connects and stalled reads trip cleanly.
Safe mid-stream
Fallbacks happen freely before the first token. Once streaming begins the model is committed — an upstream failure ends the stream with a clear error, never a silent swap.
Graceful drain
In-flight streams drain on shutdown (45s grace) and streaming applies backpressure, so deploys don't sever live completions.
03 · cost & budgets

Know the cost. Cap the spend.

Every request stores its unit-price snapshot at request time, so history is immutable. Budgets are enforced with atomic Redis counters that stay correct across every proxy instance.

Immutable cost records
Later catalog updates never rewrite the past. Missing provider usage is flagged ~estimated, never silently nulled.
Budgets that block
Day / week / month windows, global or per-agent, set to alert-or-block at the threshold.
Auto-performance view
Band mix over time, the four-way cascade outcome split, and an estimated net-savings figure with its coverage shown — never a fabricated $0 — now including the L2 semantic slice (evaluated, routed per band, bundled vs learned).
Pricing that stays current
A daily automatic refresh from LiteLLM's public catalog keeps prices honest (on by default, self-host). New prices apply to new requests only — recorded costs never change.
localhost:3000
Limits
budgets that alert or block
New budget
Team monthly
Block
$500.00/ monthGlobal
hard stop — requests rejected at limit
EditDeleteenabled
CI daily cap
Alert
$40.00/ dayAgent · ci-bot
notifies: ops-email
EditDeleteenabled
Batch weekly
Block
$60.00/ weekAgent · batch-01
notifies: ops-email, #alerts
EditDeleteenabled
Experiments
Alert
$120.00/ weekGlobal
no channels wired
EditDeletedisabled
04 · protocols & providers

OpenAI and Anthropic, both directions.

Any SDK that accepts a base URL works unchanged. Cross-protocol requests go through a translation core: multi-turn tool calls, system prompts, stop reasons, and usage carry across — locked by a golden-file contract suite. Protocol-specific controls (cache_control, thinking, output_config on Anthropic; response_format, reasoning_effort on OpenAI; image_url.detail) are carried opaquely on their own protocol and deliberately dropped — never mis-mapped — when crossing to the other.

OPENAI SDK → /v1
client = OpenAI( base_url="https://you/v1", api_key="poly_…") client.chat.completions.create( model="auto", messages=[…])
ANTHROPIC SDK → /
client = Anthropic( base_url="https://you", api_key="poly_…") client.messages.create( model="auto", max_tokens=256, messages=[…])
/v1/chat/completions /v1/messages /v1/models streaming & non-streaming

Bring a subscription, not just a key.

OAuth

Connect a Claude Pro/Max or ChatGPT Plus/Pro plan as a provider through a guided PKCE flow — sign in at the provider, paste the redirect back, done. Tokens land in an encrypted typed envelope and auto-refresh before they expire (single-flight, advisory-locked, rotation-safe). If a grant goes bad the card shows a one-click Reauthorize while your fallback chain keeps serving.

Claude speaks anthropic_compatible; ChatGPT speaks the streaming-only openai_responses backend, buffered transparently — three identity headers only, never a spoofed client.
ToS heads-up: driving flat-rate consumer subscriptions programmatically can violate provider terms — pair one with a pay-per-token fallback (the dashboard surfaces the risk).

Every provider kind, one adapter.

Provider kinds
api_key BYOK (OpenAI, Anthropic, or any compatible), subscription OAuth, custom — any OpenAI- or Anthropic-compatible base URL, and self-host-only local models (the one kind allowed to reach 127.0.0.1 — Ollama, LM Studio, llama.cpp). Credentials are AES-256-GCM encrypted and decrypted only at call time.
Per-provider max-tokens
max_tokens_spelling picks the output-cap wire field — auto sends max_tokens to local gateways and max_completion_tokens everywhere else. Inbound accepts either; outbound emits exactly one, so a legacy gateway never silently drops your cap.
Per-provider timeouts
first_byte_timeout_ms / idle_timeout_ms (1 s–1 h, in a provider's Advanced options) give research-class models — Deep Research, Opus with thinking — the patience their long prefill needs without touching global defaults.
OpenRouter attribution
Requests to an openrouter.ai provider carry HTTP-Referer and X-OpenRouter-Title so polyrouter shows up in OpenRouter's rankings — non-secret, sent only to OpenRouter, never affecting auth.
05 · security & privacy

Private and locked down.

Metadata-only by default
The request log keeps tokens, model / provider ids, cost snapshots, routing metadata and latency — no prompt or response text. A full database breach exposes no conversation content.
Opt-in body capture
Self-host only, off by default: turn on prompt / response capture behind an explicit consent gate — encrypted at rest, retention-bounded, with per-agent overrides. Cloud instances never capture.
Encrypted at rest
Provider and notification-channel credentials are encrypted before they touch the database.
SSRF-guarded egress
Every user-supplied URL is resolved and checked against private / loopback / link-local / metadata ranges — IPv6 included, with DNS-rebinding defense.
Tenant isolation
Every entity access is ownership-scoped through a central guard, covered by a dedicated e2e suite.
User administration
First-signup-wins claims the instance, then registration is invite-only. Disabling a user deletes every session in one transaction and immediately blocks every agent key — the guard rejects any key whose owner is disabled. Re-enabling restores them; rotate a key if you need it permanently dead. The last enabled admin is undeletable.
Two credential planes
Dashboard sessions (Better Auth, slow-hashed, optional OAuth) are kept separate from agent API keys (poly_…, HMAC-SHA256 + prefix lookup — fast per-request, never bcrypt on the hot path).
Layer 2 The optional semantic layer touches request text, so it ships a tight contract: no raw embedding ever reaches a Postgres column, a log, a metric, or an API response; Redis only ever sees a sum of at least MIN_COHORT embeddings; and a one-click revert fences every learned centroid. Every invariant above is enforced in code and pinned by dedicated suites (evidence-accumulator, learning-store-redis, learning.run, semantic-learning.e2e).
06 · operations

One container. Full visibility.

one containerSPA, API and proxy on one port, next to Postgres 16 + Redis.
metricsPrometheus /metrics + opt-in OpenTelemetry traces.
notificationsSMTP and Apprise channels for budget, provider-down and failure-spike alerts — queued off the request path.
multi-arch imageTagged releases publish an amd64 + arm64 image to GHCR — skip the local build.
-semantic imageA batteries-included Layer-2 variant bakes in the ONNX runtime and a checksum-pinned reference MiniLM model. The baseline image stays ORT- and model-free.
provider timeoutsPer-provider first-byte / idle bounds (1 s–1 h) let research-class models finish instead of tripping the global 30 s prefill limit.
ci/cdBuild, lint, typecheck and e2e against real Postgres + Redis on every push.
accessibleKeyboard-operable, WCAG-checked contrast, reduced-motion aware — enforced by regression suites.

Run it yourself in minutes.

Docker with Compose v2 is all you need.

Read the quickstart → View on GitHub ↗