Batch inference, at the batch rate

You have forty thousand records to classify. You do not need them back in two seconds, and every provider that runs an asynchronous batch tier charges about half for exactly that patience. Until last week polyrouter had no door for it. Every request went out synchronously, at the synchronous rate, whether an agent was waiting on a keystroke or a nightly job nobody would read until morning.

v0.17.0 opens the door. One POST carries the whole job, the router resolves it once, and the provider’s own batch API does the work:

curl https://<your-instance>/v1/batches \
  -H "Authorization: Bearer poly_your_key" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "/v1/chat/completions",
    "model": "openai/gpt-6-astra",
    "requests": [
      {"custom_id": "row-1", "body": {"messages": [{"role":"user","content":"…"}], "max_tokens": 256}},
      {"custom_id": "row-2", "body": {"messages": [{"role":"user","content":"…"}], "max_tokens": 256}}
    ]
  }'

Same poly_ key your agents already use. Same budgets, same request log, same cost records, at the batch rate instead of the synchronous one.

What the call does

Each item body is an ordinary chat or messages request. They pass through the same translation module a synchronous call uses, so an OpenAI-shaped batch aimed at an Anthropic provider reaches the upstream in Messages shape and comes back in chat shape, byte-equivalent to what the sync path would have produced.

An item carries a unique custom_id and a body, and three things it may not do: set stream, ask for more than one choice, or name a model other than the batch model. The job has one target, so an item naming a second one is a mistake worth refusing at the door.

The endpoint answers 202 with a batch object. GET /v1/batches/{id} tracks it, /results streams the outcomes as newline-delimited JSON, and POST /v1/batches/{id}/cancel stops it. Results are never stored. They are read from the provider on demand and passed through, which means the provider’s retention window is the only one, and the job object reports it rather than inventing one.

Four rules shape a submission, and each one is a deliberate narrowing.

endpoint and model must appear before requests. The body is parsed as a stream, so a 50,000-item batch never lands in memory as one object, and the parser needs the routing decision before the first item arrives.

auto is refused, with batch_auto_not_allowed. Every smart routing layer sits outside the batch path. Layer 1 scores a request to pick a model for that request, and a batch is one target for tens of thousands of them.

The model resolves exactly once, at submission. A tier resolves to its primary. There is no fallback across a 24-hour completion window, because once an upstream has accepted the job, walking to a second provider would mean paying for the work twice.

The resolved provider must carry a batch adapter. OpenRouter, Anthropic and OpenAI ship one. A custom or local provider has no batch seam, and says so at submit time with batch_not_supported rather than failing hours later.

Batch is a mode, not a model

Three upstreams spell the same capability three ways:

Provider	Where batch lives	Model id it takes
OpenRouter	POST /api/beta/batches, requests inline	the plain slug, never :batch
OpenAI	JSONL upload to /v1/files, then a batch referencing the file	the same id as a sync call
Anthropic	POST /v1/messages/batches, requests inline	the same id as a sync call

Only one of the three has a separate id at all, and even there the id prices the tier rather than naming a target. So batch could not be a property of a model in polyrouter. It had to be a mode on the call, which is why the endpoint above takes one model for the whole job and every item inherits it.

That single sentence is also what the model catalog had to be taught.

The row that was never a model

Open OpenRouter’s model list and scroll to the OpenAI section. OpenAI: GPT-6 Astra is there. Directly under it sits OpenAI: GPT-6 Astra (batch), at half the price: five dollars in and twenty-five out, against ten and fifty.

I counted the live catalog on September 4. 431 models, 69 of them :batch, 19 :free. A :batch entry is byte-identical to the model it shadows except for three things: the id, the display name, and prices at exactly 50 percent. Its endpoint entry reports null latency, null throughput and null uptime. No synchronous traffic has ever flowed through it, because none can.

The reason is on OpenRouter’s own page for the model. The “make your first request” example for a :batch id posts to POST /api/beta/batches with "model": "openai/gpt-6-astra", the plain slug. The suffixed id is never sent anywhere. It is a price row wearing a model’s clothes.

polyrouter had been syncing it as a model, which made it selectable as a routing target and, being the cheapest row for every model that has one, the row a price-driven choice lands on. Two jobs, then, and one fact underneath:

The row as stored	The row as it is
A model, openai/gpt-6-astra:batch	A price fact about openai/gpt-6-astra
Listed at $5 / $25, routable	The batch rate: $5 / $25
Nothing can call it	The base model is the only target

The lazy version of this is to hide 69 rows. The useful version is to reclassify them from models into prices, because the price recovered is exactly what the new endpoint needed to bill honestly.

What you can name, and what you cannot

A batch names the base model, the same id a synchronous call names. The twin is no longer routable at all, and that starts with a parser:

export const MODEL_VARIANTS: readonly string[] = [
  'batch', 'free', 'nitro', 'floor', 'extended', 'thinking', 'online', 'exacto',
];
export const NON_ROUTABLE_VARIANTS: readonly string[] = ['batch'];

export function parseModelVariant(externalModelId: string): ParsedModelVariant | null {
  const id = externalModelId.trim();
  const colon = id.lastIndexOf(':');
  if (colon <= 0 || colon === id.length - 1) return null;
  const suffix = id.slice(colon + 1).trim().toLowerCase();
  if (!MODEL_VARIANTS.includes(suffix)) return null;
  // Only the suffix is lower-cased. The base keeps the id's original casing,
  // because `external_model_id` is stored and matched exactly as given.
  const base = id.slice(0, colon).trim();
  if (base === '') return null;
  return { base, variant: suffix };
}

An allowlist, never a shape. “Text after the last colon” would read anthropic.claude-haiku-4-5-20251001-v1:0 as variant 0. An unknown suffix classifies as nothing and the model stays routable, so a token OpenRouter invents next month falls into today’s behavior rather than into a refusal.

The parse is scoped to aggregator providers, through the same host-to-family map the pricing keys use. A self-hosted gateway may legitimately serve a model named foo:batch, and unknown beats wrong.

Naming a twin anyway returns 400, not 404, and the message names the way out:

const suffix =
  err.baseIsRoutable === true
    ? `; use "${err.baseModelId}" instead`
    : `; its base model is "${err.baseModelId}"`;

Two details there are load-bearing. The status is 400 because the row exists and the dashboard shows it with a price, so “model not found” would be a lie. And the base id is derived by the resolver from your own configuration, never echoed from the client’s model field. The message offers it as an alternative only when a routable model bearing that id actually exists on the same provider. Naming a route that is not there would be the same dishonesty from the other direction.

GET /v1/models drops them too, which shrinks the advertised list for an OpenRouter tenant by up to 16 percent. Every id it removed could not have served a request.

Rows that synced before the upgrade are classified by a boot pass, not by the migration. Migrations here are SQL files, while the variant allowlist and the host-to-family map are TypeScript. Writing either one again in SQL would fork the definition and drift on the first host added, so the pass runs after migrations and before traffic, and it writes only rows whose stored value differs. Once converged it is a read-only no-op on every later boot. An instance that synced an aggregator catalog last month behaves correctly on this boot rather than on its next sync.

The price the twin was holding

Here is where the reclassification pays for itself. The twin’s captured rate is not discarded. It becomes the batch price for the model it shadows, and the batch service looks for it by pairing:

const twin = models.find(
  (m) =>
    m.providerId === provider.id &&
    m.variant === 'batch' &&
    parseModelVariant(m.externalModelId)?.base === model.externalModelId,
);

That rate is the last resort, not the first. Batch-mode resolution walks its own precedence: the exact catalog row’s batch pair, then the native-family row’s batch pair, then the twin’s captured listed rate, then nothing.

if (catalogRow !== null) {
  const exact = fromRow(catalogRow, catalogRow.source as PriceSource);
  if (exact !== null) return exact;
}
if (nativeCatalogRow !== null) {
  const native = fromRow(nativeCatalogRow, 'native_family');
  if (native !== null) return native;
}
const li = opts.listedBatchInputPricePer1m ?? null;
const lo = opts.listedBatchOutputPricePer1m ?? null;
if (li !== null && lo !== null) {
  // the twin's captured rate, recorded as source `listed` with no version id
  return { /* … */ inputPricePer1m: li, outputPricePer1m: lo, mode: 'batch' };
}
return null;

A synchronous rate never substitutes for a missing batch rate. Where the batch tier is half price, charging a batch item at the sync rate is wrong by a factor of two, and a cost record that is wrong is worse than one that is absent.

The ledger says which rule paid for each row:

// Which pricing rule the snapshot came from. 'sync' is resolved when the request
// completed. 'batch' is the job's submit-time snapshot, copied verbatim at
// settlement. Null predates the column and reads as 'sync'.
priceMode: text('price_mode'),

check(
  'request_log_batch_price_mode_compat',
  sql`${t.priceMode} IS DISTINCT FROM 'batch' OR ${t.batchId} IS NOT NULL`,
),

The database refuses a batch-priced row that names no job. A batch snapshot without a settlement is a cost nobody can trace, and the constraint makes that unrepresentable rather than unlikely. Batch and synchronous spend then separate cleanly in analytics, which is the whole point of recording the mode.

What it costs you

Three trades, stated plainly.

Anthropic’s batch rate is not in LiteLLM’s catalog. Every batch pair there belongs to OpenAI, Azure, Gemini, or Bedrock and Vertex-hosted Claude, so a direct Anthropic provider resolves an unknown batch rate unless you supply the pair through the manual override. Recording Anthropic’s published 50 percent as if it were catalog data would be inventing a price, which is the one thing the cost rules forbid. Under a block budget an unknown rate means the submission is refused with batch_unbounded, because a budget cannot bound what it cannot price.

Submitting reserves a ceiling rather than charging it. The reservation is the worst-case cost of every item at the snapshotted rate, held as a distinct pending component of the same atomic Redis counter your synchronous spend uses. Recording cost only at settlement would let a fifty-dollar cap wave through a five-hundred-dollar batch, and would let two concurrent batches each fit the same remaining budget. The dashboard says which of the two numbers you are looking at. An item with no max_tokens on a model with no known output cap cannot be bounded either, and gets the same refusal.

Once an upstream accepts a job, it is bound to that provider for the window. A tier’s fallback chain applies at submission and never again.

Where this leaves you

v0.17.0 runs four additive migrations on boot and needs no configuration. Batch is on by default. BATCH_ENABLED=false refuses new submissions while the poller, reads, results and cancel keep serving, so in-flight jobs drain instead of stranding.

curl -fsSL https://polyrouter.app/install.sh | sh

Point a nightly job at /v1/batches and read the Batches page while it runs. If you already run polyrouter against an OpenRouter provider, the model list also gets shorter on the first boot after the upgrade, and every id it lost was one you could not have called anyway. The source is at github.com/izzoa/polyrouter under AGPL-3.0.