--- title: "One disagreement, one self-hosted LLM router" description: "Manifest deprecated rule-based routing. polyrouter kept automatic routing and hardened it. What a self-hosted LLM router adds, and how the two compare, in code." canonical: https://polyrouter.app/blog/self-hosted-llm-router-one-disagreement published: 2026-08-26 --- # One disagreement, one self-hosted LLM router Open the config file of any terminal agent. In OpenClaw it is `~/.openclaw/openclaw.json`, and the block that matters holds two values: a `baseUrl` and an `apiKey`. One provider, one key. Add a second provider and the block doubles. Add a second harness on the same machine and it doubles again. Each copy of each key answers to nobody about what it spent last month, and each one stops cold when its provider has a bad hour. That was my setup. Harnesses on my own machines, pointed either straight at the providers or through Manifest, an open-source LLM gateway that put automatic routing in front of the same keys. On June 22, 2026, Manifest published a post titled ["We are deprecating our rule-based routing"](https://manifest.build/blog/deprecating-rule-based-routing/). I agreed with almost every sentence in it. The one I did not agree with is why polyrouter exists: a self-hosted LLM router built to test that disagreement. The first commit landed on July 14, three weeks later, and v0.1.0 shipped on July 17. ## What Manifest said, and what I agreed with The post makes three claims. I will take them in order. First, harness system prompts break complexity rules. "Some harnesses like OpenClaw or Hermes often send a huge system prompt that can be enough to categorize all messages as complex." True. A rule that scores the whole request sees the same large preamble on every call and grades every call as hard. Second, rules only work in English. Rule-based routing "only applies to prompts written in English," Manifest wrote, and static rules cannot capture the complexity of human language. Also true, as long as the rules read words. Third, the obvious fix costs too much. "A credible alternative would be AI-powered routing," the post says, but "this adds extra latency and cost to all requests and most users do not need that kind of system." I agree with this one most of all. A router that calls a model to decide which model to call has doubled the thing it exists to reduce. The conclusion Manifest drew: retire the automatic layer, keep custom tiers that a client selects with an `x-manifest-tier` header, and remove the old routing on September 1. Here is where I disagreed. All three problems are problems with how the rules read the request. None of them is a problem with automatic routing as an idea. The diagnosis was right. The conclusion stopped one step early. ## The disagreement, as a mechanism polyrouter routes `model: "auto"` through a structural layer, Layer 1, which is on by default. It answers the objections one at a time. | The objection | What Layer 1 does | | --- | --- | | The harness system prompt grades everything as complex | Layer 1 excludes the system block from scoring. It reads the last six messages, and only those. | | Every harness has a different preamble, so no rule can cover them all | A keyed hash of the system prompt names a learned per-agent baseline, and Layer 1 subtracts that baseline from the size signal. It measures the delta, not the preamble. | | Rules only work in English | The features are sizes, counts and flags: input length, code length, tool count, an image block, a declared output format, a declared reasoning effort. None of them has a language. | | An AI router adds latency and cost | Layer 1 runs local and sub-millisecond, with no network call and no tokenizer. It costs nothing per request. | ### Features that cannot read The rules Manifest deprecated read words. Its scorer combines 31 dimensions, and 22 of them are lists of English phrases with a weight, matched through a keyword trie. The other nine are structural counts. This is the [shape of a keyword dimension](https://github.com/mnfst/manifest/blob/main/packages/backend/src/scoring/keywords/complexity.ts): ```ts export const COMPLEXITY_KEYWORDS: Record = { formalLogic: ['prove', 'proof', 'derive', 'derivation', 'theorem', /* … */], codeGeneration: ['write a function', 'implement', 'create a class', 'build a component', /* … */], simpleIndicators: ['what is', 'define', 'translate', 'thanks', 'thank you', 'yes', 'no', 'ok', /* … */], // … }; ``` A list like that has no answer for a prompt in Japanese, and the post above explains what it did with a harness preamble. Manifest said as much, and retired it. The polyrouter extractor runs on the normalized request, after protocol translation, so one function serves OpenAI and Anthropic clients alike. It never receives the system block. It walks the last six messages and stops after 32,000 characters: ```ts export const RECENT_WINDOW = 6; export const MAX_SCAN_CHARS = 32_000; export function extractStructuralFeatures(ir: NormalizedRequest): StructuralFeatures { const messages = ir.messages; const window = messages.length <= RECENT_WINDOW ? messages : messages.slice(messages.length - RECENT_WINDOW); const acc: ScanAcc = { chars: 0, code: 0, multimodal: false, budget: MAX_SCAN_CHARS }; for (const msg of window) walk(msg.content, acc); // … tool count, schema demand, declared reasoning effort return { effectiveInputChars: acc.chars, codeBlockChars: acc.code, /* … */ }; } ``` Every field in the result is a count, a size or a flag: `effectiveInputChars`, `codeBlockChars`, `toolCount`, `multimodalPresent`, `conversationDepth`, `maxOutputTokens`, `reasoningDemand`. A prompt in Japanese produces the same shape of vector as a prompt in English, because nothing in this file reads a word. That is the answer to the second objection, and it costs one pass over at most 32 KB of text. ### A baseline named by a hash The system block gets a different treatment. `canonicalizeSystem` folds it into one stable string, per block type and length, capped at 16,000 characters. The control plane then runs that string through an HMAC keyed with a server secret and keeps 128 bits of the digest as a field name. The field lives in a Redis hash per tenant and agent, and its value is an exponential moving average of `effectiveInputChars`. The update is one Lua script, so two proxy instances cannot race each other: ```lua local exists = redis.call('HEXISTS', KEYS[1], ARGV[1]) if exists == 0 then if redis.call('HLEN', KEYS[1]) >= tonumber(ARGV[4]) then local stale = redis.call('ZPOPMIN', KEYS[2]) if stale[1] then redis.call('HDEL', KEYS[1], stale[1]) end end redis.call('HSET', KEYS[1], ARGV[1], ARGV[2]) else local prev = tonumber(redis.call('HGET', KEYS[1], ARGV[1])) local a = tonumber(ARGV[3]) redis.call('HSET', KEYS[1], ARGV[1], a * tonumber(ARGV[2]) + (1 - a) * prev) end -- … touch the LRU zset, refresh both TTLs ``` Each agent holds at most 32 fingerprints, and a new one at the cap evicts the stalest. That branch exists because some harnesses interpolate a timestamp or a session id into the system prompt. Without eviction, those one-off fingerprints would fill the set and lock out the boilerplate that matters. Alpha is 0.2 by default, and a hash nobody touches for 30 days expires. The read side never waits on Redis. `read` is a synchronous lookup in an in-process LRU. A cold miss returns `null` and schedules a background seed, so the first request from a new agent classifies on raw features and the ones after it subtract the baseline. Nothing on the hot path awaits a network. That is why Layer 1 stays under a millisecond. The hash itself is never written to the request log. It is a Redis key and nothing else. ### A score that a long prompt cannot win alone With the baseline in hand, the classifier subtracts it from the size signal and scores what is left through saturating sub-scores: ```ts const sizeDelta = nonNeg(f.effectiveInputChars) - (baseline ? nonNeg(baseline.ewma) : 0); const sub = { size: sat(sizeDelta, SIZE_SAT), // 1.0 at 8,000 chars code: sat(f.codeBlockChars, CODE_SAT), // 1.0 at 4,000 chars tools: sat(f.toolCount, TOOLS_SAT), // 1.0 at 8 tools schema: f.toolSchemaDemand || f.responseFormatDemand ? 1 : 0, depth: sat(f.conversationDepth, DEPTH_SAT), // 1.0 at 20 messages multimodal: f.multimodalPresent ? 1 : 0, maxTokens: sat(f.maxOutputTokens, MAXTOK_SAT), }; ``` The default weights sum to one, and size carries 0.30 of it. A long prompt on its own therefore cannot reach the `high` threshold of 0.6. It needs company: code, tools, a schema, depth. That cap is the second answer to the harness problem, for the requests that arrive before the baseline has learned anything. A score at or above 0.6 is `high`, at or below 0.25 is `low`, and anything between is `ambiguous`. The reason line the inspector shows is a serialization of those sub-scores and nothing else: ``` structural:high score=0.65 size=0.60 code=1.00 tools=0.50 schema=1.00 depth=0.30 mm=0.00 maxtok=0.80 think=-- rf=1.00 ``` No prompt text and no hash of it. The reason line is where metadata-only storage is easiest to break by accident, so the classifier can only emit numbers. ### Every fault has the same shape A confident band steers the request to the tier you bound to it. An ambiguous band falls to the optional layers, a local embedding classifier and a cheap-first cascade, and when those are off or fail, the request lands on the default tier. The last piece is what happens when Layer 1 itself breaks. The whole classify call sits inside one try block: ```ts } catch { return { kind: 'skip' }; // degrade to Layer 0 — never fail or stall } ``` A `skip` carries no verdict and no telemetry. The proxy then asks Layer 0 for the route it would have taken with Layer 1 disabled, and Layer 0 is a pure function with no database and no clock — the five-phase precedence is in [the routing documentation](/docs/routing). Manifest was right that most users do not want a model in the routing path. So there is no model in the routing path. There is a function that reads sizes, and a hash that remembers what it read last time. ## How polyrouter improves over bare API keys The objection I hear most is not about Manifest. It is "I can just call the API directly." You can, and for one agent on one provider that is the right call. Here is what changes when the key moves behind a router. ### Every request gets a price tag A provider bill has one line per model and no idea which agent made the calls. polyrouter writes a request log row for every call, and the row carries the unit prices that applied at that moment. The cost math is a pure function over that snapshot: ```ts export function computeCost(usage: ResolvedUsage, price: PriceSnapshot | null): number | null { if (price === null) return null; if (price.isFree) return 0; // … a cache component whose rate the catalog lacks → null, never an understated number let cost = (usage.inputTokens / 1_000_000) * price.inputPricePer1m + (usage.outputTokens / 1_000_000) * price.outputPricePer1m; // … cache read and cache write components return cost; } ``` The `PriceSnapshot` is the point. The proxy resolves the price at request time and writes the numbers it used into the row: `input_price_snapshot`, `output_price_snapshot`, the two cache rates, `price_version_id` and `price_source`. The catalog can change tomorrow. The row cannot, because nothing ever recomputes a stored cost against current prices. When a provider omits usage, `resolveUsage` estimates tokens as `chars / 4` and sets `usage_estimated`, which the dashboard shows as `~est`. A cache component with no known rate returns `null` instead of a smaller number, because a wrong cost in an immutable ledger is worse than a missing one. The dashboard then groups spend by model, provider and agent, which is the view the provider bill cannot give you. ### Spend gets a ceiling An agent that retries a failing tool call all night spends until you wake up. polyrouter enforces spend limits over a day, a week or a month, global or per agent, and each limit either alerts or blocks at its threshold. The hot path increments nothing. A scheduler reconciles the request ledger into Redis once a minute, in integer micro-dollars, keyed per owner, scope, window and period, with a script that only ever moves a counter up: ```lua local exists = redis.call('EXISTS', KEYS[1]) local c = tonumber(redis.call('GET', KEYS[1]) or '0') local v = tonumber(ARGV[1]) if v > c or exists == 0 then redis.call('SET', KEYS[1], v) end redis.call('PEXPIRE', KEYS[1], ARGV[2]) -- … return the resulting counter ``` The request path reads that counter and compares it with the budget. Two instances read the same number, so neither can believe there is budget left when there is none. The interesting case is a stopped scheduler. A counter nobody has written for three minutes would read as zero, and zero admits everything, so the check treats a stale heartbeat as "enforcement unavailable" and applies the configured fail mode. The default admits the request and records the fault. `BUDGET_FAIL_OPEN=false` turns that into a 503. The trade is lag: a block engages within a minute of the ledger crossing the line, not on the exact request that crossed it. ### An outage becomes a retry With a bare key there is nowhere else to go. In polyrouter, every route resolves to a chain, and the chain is where fallback lives or does not: ```ts function modelDecision( model: RouteModel, decisionLayer: DecisionLayer, routingReason: string, tierKey: string | null = null, ): RouteDecision { return { // … chain: [target(model)], // a directly-named model has no fallback }; } ``` A tier builds its chain from its entries in position order. A named model builds a chain of one. Everything downstream walks `chain` and knows nothing else, so a request that names `gpt-5` gets `gpt-5` or an error, and a router that swaps models behind a benchmark is worse than none. Per-provider circuit breakers stop the walk from calling a provider that is already down. On a stream, the walk has a boundary. Each member opens behind a gate that waits for exactly one event: ```ts let first: IteratorResult; try { first = await nextWithTimeout(iterator, bound, abort, liveness); } catch (err) { await cleanup(); return { kind: 'error', error: err }; // raw — the chain classifies eligibility } // … a successful first event commits this member ``` Before the first event, a failure returns to the chain, and `fallbackEligible` decides whether to try the next member. A rate limit or a tripped breaker walks on. A bad request or a client disconnect stops. After the first event, the member is committed. A later failure yields one terminal frame in the client's own protocol, with a fixed message: "the upstream model failed mid-stream". No retry, no substitute, no second model finishing the first model's sentence. ### One key per agent, rotated in one place Every copy of a provider key is another file to find when that key leaks. In polyrouter each agent gets its own `poly_` key, and the provider keys live in one place, encrypted at rest. Verification is one HMAC and one constant-time compare: ```ts export function verifyAgentKey(key: string, storedHash: string, secret: string): boolean { const candidate = Buffer.from(hmacKey(key, secret), 'hex'); const stored = Buffer.from(storedHash, 'hex'); if (candidate.length !== stored.length) return false; return timingSafeEqual(candidate, stored); } ``` A key is `poly_` plus 32 base64url characters. The database stores the first 12 payload characters as a lookup prefix and an HMAC-SHA256 of the whole key, never the key itself. That is cheap enough to run on every request. Dashboard passwords go through a slow hash on a separate credential plane, and the two planes never meet. Disabling a user revokes their sessions and their agent keys in one action. None of these four mechanisms is clever. That is the point of them. They are the part of a router that has to be right before automatic routing earns the right to sit on top. ## polyrouter and Manifest, feature by feature Manifest is the project polyrouter learned the most from, so the fair comparison is a close one. Everything below comes from the two repositories as of August 26, 2026, and I link the Manifest code where I quote it. | | polyrouter | Manifest | | --- | --- | --- | | Automatic routing for `model: "auto"` | L1 structural on by default, L2 semantic and L3 cascade opt-in, workload classes | Deprecated. `auto` goes to the default tier, and the old scorer goes away on September 1 | | Tier pinning | `x-polyrouter-tier` header. A tier carries an ordered fallback chain | Header tiers on any header key and value. Each carries an override route and fallback routes | | Precedence | A model named in the body wins over every header | A matching header tier wins over a model named in the body | | A named model | One attempt, no fallback | One attempt, no fallback | | What triggers a fallback | Any provider error except `bad_request`, plus an open breaker | Any provider status of 400 or above | | Mid-stream | Commit at the first event, terminal error after | Warm-up peek at the first bytes, error frame after | | Where the decision shows up | The request log and the decision inspector. No response headers | `X-Manifest-Reason` and `X-Manifest-Confidence` response headers, plus the dashboard | | Cost record | Unit-price snapshot, version and source on every row | `cost_usd` per attempt from a models.dev catalog with OpenRouter fallback, including time-of-day tiers | | Spend limits | Day, week or month, global or per agent, alert or block. Redis counters shared across instances | Per-agent rules on tokens or cost over an hour, day, week or month. Notify, block or both. Postgres sums cached in-process for 60 s | | Request bodies | Metadata only. Body capture is opt-in, encrypted, self-host only | Full request and response recording per attempt, on by default for new agents, per-agent switch, retention window | | Agent keys | `poly_`, HMAC-SHA256 plus prefix lookup on every request | `mnfst_`, scrypt with a per-key salt, verified keys cached for five minutes | | Providers | Any OpenAI- or Anthropic-compatible endpoint, local models, Claude and ChatGPT subscriptions over OAuth | 32 built-in providers, 18 subscription flows, local models, aggregators, custom endpoints | | Client protocols | OpenAI chat completions and models, Anthropic messages | OpenAI chat completions and Responses, Anthropic messages | | Request repair | None | Autofix repairs a malformed request and resends it | | Services | App, PostgreSQL 16, Redis | App, PostgreSQL 16, recording storage on disk or S3 | | License | AGPL-3.0 | MIT | Three of those rows are worth the code. ### Pinning a tier The client side is the same idea with a different header name. Manifest, from its own announcement: ```ts const client = new OpenAI({ baseURL: "https://app.manifest.build/v1", apiKey: "mnfst_XXXX", defaultHeaders: { "x-manifest-tier": "custom-value" }, }); ``` polyrouter: ```ts const client = new OpenAI({ baseURL: "https:///v1", apiKey: "poly_XXXX", defaultHeaders: { "x-polyrouter-tier": "fast" }, }); ``` The difference is what happens when the body also names a model. Manifest resolves the header tier first, on purpose. Its [resolver](https://github.com/mnfst/manifest/blob/main/packages/backend/src/routing/resolve/resolve.service.ts) explains that "a header rule is a deliberate override the operator configured, and it outranks the model an SDK happens to name": ```ts private async resolveExplicitModel( agentId: string, tenantId: string, requestedModel: string, headers: ProxyRequestOptions['headers'], ): Promise { if (headers) { const headerTier = await this.resolveService.resolveHeaderTier(agentId, tenantId, headers); if (headerTier) return headerTier; } // … then the model the body asked for } ``` polyrouter checks the body first, and a named model ends the search before it reads any header: ```ts export function resolveRoute(snap: RoutingSnapshot, parsed: ParsedRoute): RouteDecision | RouteError { const mf = parsed.modelField; if (mf.length > 0 && mf !== AUTO_ALIAS) { // … an explicit model or tier resolves here, or returns unknown_model } // … x-polyrouter-tier, then other header rules, then the default rule, then the default tier } ``` Both are defensible. Manifest treats the header as the operator overriding the SDK. polyrouter treats the body as the caller saying "no substitutions", because the caller is often a benchmark or a test, and an operator who wants to steer it can name a tier instead of a model. Pick the one that matches who you trust more, the operator or the caller. ### What a fallback chain looks like Manifest stores a tier as one override route and an optional list of fallbacks, as JSON on the [tier row](https://github.com/mnfst/manifest/blob/main/packages/backend/src/entities/header-tier.entity.ts): ```ts @Entity('header_tiers') export class HeaderTier { // … @Column('varchar') header_key!: string; @Column('varchar') header_value!: string; @Column('jsonb', { nullable: true }) override_route!: ModelRoute | null; @Column('jsonb', { nullable: true }) fallback_routes!: ModelRoute[] | null; // … } ``` polyrouter stores a tier as a set of entries with positions, and the resolver sorts them into the chain: ```ts const entries = [...(snap.entriesByTierId.get(tier.id) ?? [])].sort( (a, b) => a.position - b.position, ); const primary = entries.find((e) => e.position === 0); if (!primary) return { error: 'empty_tier', detail: tier.key }; ``` Same shape, two storage choices. The larger difference is the trigger. Manifest walks to the next route on any status of 400 or above, and `shouldTriggerFallback(status)` is one line. polyrouter excludes `bad_request`, on the theory that a malformed request stays malformed on the next provider too, and it skips a member whose breaker is open without dispatching a call. Both projects reached the same conclusion on streams. Manifest peeks at the first bytes for up to 15 seconds before committing to the client, and a socket that dies after that writes an error frame. Neither swaps a model mid-response. ### What the request log keeps This is the row I would read first if I had to choose. Manifest records the full request and response body of every provider attempt, gzipped to disk or S3, and the [switch](https://github.com/mnfst/manifest/blob/main/packages/backend/src/entities/agent.entity.ts) is on for new agents by default: ```ts // Enabled for newly created agents. Existing agents keep their persisted // choice because the migration changes only the column default. @Column('boolean', { default: true }) record_messages!: boolean; ``` That is a feature. The docs describe reading back any attempt "as a conversation", and for debugging an agent it is exactly what you want. It is also a store of your prompts with a retention window, on a box you now have to protect. polyrouter starts from the other end. The request log holds tokens, cost, latency, the decision layer and the numbers-only reason, and nothing else. Body capture exists, but it is opt-in, encrypted at rest, retention-bounded and self-host only. The trade is the debugging view. Without capture you get the numbers for a bad request and not the conversation. Where Manifest is ahead is not subtle. Thirty-two built-in providers against two wire shapes, eighteen subscription flows against two, a Responses API surface, request repair, eleven deploy templates and an MIT license. If your providers sit outside the OpenAI and Anthropic shapes, Manifest is the shorter path today. ## Where this leaves you A self-hosted LLM router is one more thing to run. polyrouter is one container next to PostgreSQL and Redis. That is three services a bare key does not need, and one hop in the path of every request. If that trade is worth it to you, the install is one line: ```bash curl -fsSL https://polyrouter.app/install.sh | sh ``` Point a harness at the OpenAI-compatible endpoint, `https:///v1`, with a `poly_` key, set the model to `auto`, and open the decision inspector. Every request shows the layer that decided and the numbers that drove it. The source is at [github.com/izzoa/polyrouter](https://github.com/izzoa/polyrouter) under AGPL-3.0, and the quickstart is here: [self-host polyrouter on my box](/docs).