---
title: "Batch inference, at the batch rate"
description: "polyrouter now runs batch jobs on the upstream batch API, at the batch rate. Getting there meant reclassifying 69 OpenRouter rows from models into prices."
canonical: https://polyrouter.app/blog/batch-inference-at-the-batch-rate
published: 2026-09-06
---
# Batch inference, at the batch rate

You have forty thousand records to classify. You do not need them back in two seconds,
and every provider that runs an asynchronous batch tier charges about half for exactly
that patience. Until last week polyrouter had no door for it. Every request went out
synchronously, at the synchronous rate, whether an agent was waiting on a keystroke or
a nightly job nobody would read until morning.

v0.17.0 opens the door. One `POST` carries the whole job, the router resolves it once,
and the provider's own batch API does the work:

```bash
curl https://<your-instance>/v1/batches \
  -H "Authorization: Bearer poly_your_key" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "/v1/chat/completions",
    "model": "openai/gpt-6-astra",
    "requests": [
      {"custom_id": "row-1", "body": {"messages": [{"role":"user","content":"…"}], "max_tokens": 256}},
      {"custom_id": "row-2", "body": {"messages": [{"role":"user","content":"…"}], "max_tokens": 256}}
    ]
  }'
```

Same `poly_` key your agents already use. Same budgets, same request log, same cost
records, at the batch rate instead of the synchronous one.

## What the call does

Each item body is an ordinary chat or messages request. They pass through the same
translation module a synchronous call uses, so an OpenAI-shaped batch aimed at an
Anthropic provider reaches the upstream in Messages shape and comes back in chat
shape, byte-equivalent to what the sync path would have produced.

An item carries a unique `custom_id` and a body, and three things it may not do: set
`stream`, ask for more than one choice, or name a model other than the batch model.
The job has one target, so an item naming a second one is a mistake worth refusing at
the door.

The endpoint answers 202 with a batch object. `GET /v1/batches/{id}` tracks it,
`/results` streams the outcomes as newline-delimited JSON, and
`POST /v1/batches/{id}/cancel` stops it. Results are never stored. They are read from
the provider on demand and passed through, which means the provider's retention window
is the only one, and the job object reports it rather than inventing one.

Four rules shape a submission, and each one is a deliberate narrowing.

`endpoint` and `model` must appear before `requests`. The body is parsed as a stream,
so a 50,000-item batch never lands in memory as one object, and the parser needs the
routing decision before the first item arrives.

`auto` is refused, with `batch_auto_not_allowed`. Every smart routing layer sits
outside the batch path. Layer 1 scores a request to pick a model for that request, and
a batch is one target for tens of thousands of them.

The model resolves exactly once, at submission. A tier resolves to its primary. There
is no fallback across a 24-hour completion window, because once an upstream has
accepted the job, walking to a second provider would mean paying for the work twice.

The resolved provider must carry a batch adapter. OpenRouter, Anthropic and OpenAI
ship one. A custom or local provider has no batch seam, and says so at submit time
with `batch_not_supported` rather than failing hours later.

## Batch is a mode, not a model

Three upstreams spell the same capability three ways:

| Provider | Where batch lives | Model id it takes |
| --- | --- | --- |
| OpenRouter | `POST /api/beta/batches`, requests inline | the plain slug, never `:batch` |
| OpenAI | JSONL upload to `/v1/files`, then a batch referencing the file | the same id as a sync call |
| Anthropic | `POST /v1/messages/batches`, requests inline | the same id as a sync call |

Only one of the three has a separate id at all, and even there the id prices the tier
rather than naming a target. So batch could not be a property of a model in
polyrouter. It had to be a mode on the call, which is why the endpoint above takes one
`model` for the whole job and every item inherits it.

That single sentence is also what the model catalog had to be taught.

## The row that was never a model

Open OpenRouter's model list and scroll to the OpenAI section. `OpenAI: GPT-6 Astra`
is there. Directly under it sits `OpenAI: GPT-6 Astra (batch)`, at half the price:
five dollars in and twenty-five out, against ten and fifty.

I counted the live catalog on September 4. 431 models, 69 of them `:batch`, 19 `:free`.
A `:batch` entry is byte-identical to the model it shadows except for three things: the
id, the display name, and prices at exactly 50 percent. Its endpoint entry reports null
latency, null throughput and null uptime. No synchronous traffic has ever flowed
through it, because none can.

The reason is on OpenRouter's own page for the model. The "make your first request"
example for a `:batch` id posts to `POST /api/beta/batches` with
`"model": "openai/gpt-6-astra"`, the plain slug. The suffixed id is never sent
anywhere. It is a price row wearing a model's clothes.

polyrouter had been syncing it as a model, which made it selectable as a routing
target and, being the cheapest row for every model that has one, the row a price-driven
choice lands on. Two jobs, then, and one fact underneath:

| The row as stored | The row as it is |
| --- | --- |
| A model, `openai/gpt-6-astra:batch` | A price fact about `openai/gpt-6-astra` |
| Listed at $5 / $25, routable | The batch rate: $5 / $25 |
| Nothing can call it | The base model is the only target |

The lazy version of this is to hide 69 rows. The useful version is to reclassify them
from models into prices, because the price recovered is exactly what the new endpoint
needed to bill honestly.

## What you can name, and what you cannot

A batch names the base model, the same id a synchronous call names. The twin is no
longer routable at all, and that starts with a parser:

```ts
export const MODEL_VARIANTS: readonly string[] = [
  'batch', 'free', 'nitro', 'floor', 'extended', 'thinking', 'online', 'exacto',
];
export const NON_ROUTABLE_VARIANTS: readonly string[] = ['batch'];

export function parseModelVariant(externalModelId: string): ParsedModelVariant | null {
  const id = externalModelId.trim();
  const colon = id.lastIndexOf(':');
  if (colon <= 0 || colon === id.length - 1) return null;
  const suffix = id.slice(colon + 1).trim().toLowerCase();
  if (!MODEL_VARIANTS.includes(suffix)) return null;
  // Only the suffix is lower-cased. The base keeps the id's original casing,
  // because `external_model_id` is stored and matched exactly as given.
  const base = id.slice(0, colon).trim();
  if (base === '') return null;
  return { base, variant: suffix };
}
```

An allowlist, never a shape. "Text after the last colon" would read
`anthropic.claude-haiku-4-5-20251001-v1:0` as variant `0`. An unknown suffix classifies
as nothing and the model stays routable, so a token OpenRouter invents next month falls
into today's behavior rather than into a refusal.

The parse is scoped to aggregator providers, through the same host-to-family map the
pricing keys use. A self-hosted gateway may legitimately serve a model named
`foo:batch`, and unknown beats wrong.

Naming a twin anyway returns 400, not 404, and the message names the way out:

```ts
const suffix =
  err.baseIsRoutable === true
    ? `; use "${err.baseModelId}" instead`
    : `; its base model is "${err.baseModelId}"`;
```

Two details there are load-bearing. The status is 400 because the row exists and the
dashboard shows it with a price, so "model not found" would be a lie. And the base id
is derived by the resolver from your own configuration, never echoed from the client's
`model` field. The message offers it as an alternative only when a routable model
bearing that id actually exists on the same provider. Naming a route that is not there
would be the same dishonesty from the other direction.

`GET /v1/models` drops them too, which shrinks the advertised list for an OpenRouter
tenant by up to 16 percent. Every id it removed could not have served a request.

Rows that synced before the upgrade are classified by a boot pass, not by the
migration. Migrations here are SQL files, while the variant allowlist and the
host-to-family map are TypeScript. Writing either one again in SQL would fork the
definition and drift on the first host added, so the pass runs after migrations and
before traffic, and it writes only rows whose stored value differs. Once converged it
is a read-only no-op on every later boot. An instance that synced an aggregator catalog
last month behaves correctly on this boot rather than on its next sync.

## The price the twin was holding

Here is where the reclassification pays for itself. The twin's captured rate is not
discarded. It becomes the batch price for the model it shadows, and the batch service
looks for it by pairing:

```ts
const twin = models.find(
  (m) =>
    m.providerId === provider.id &&
    m.variant === 'batch' &&
    parseModelVariant(m.externalModelId)?.base === model.externalModelId,
);
```

That rate is the last resort, not the first. Batch-mode resolution walks its own
precedence: the exact catalog row's batch pair, then the native-family row's batch
pair, then the twin's captured listed rate, then nothing.

```ts
if (catalogRow !== null) {
  const exact = fromRow(catalogRow, catalogRow.source as PriceSource);
  if (exact !== null) return exact;
}
if (nativeCatalogRow !== null) {
  const native = fromRow(nativeCatalogRow, 'native_family');
  if (native !== null) return native;
}
const li = opts.listedBatchInputPricePer1m ?? null;
const lo = opts.listedBatchOutputPricePer1m ?? null;
if (li !== null && lo !== null) {
  // the twin's captured rate, recorded as source `listed` with no version id
  return { /* … */ inputPricePer1m: li, outputPricePer1m: lo, mode: 'batch' };
}
return null;
```

A synchronous rate never substitutes for a missing batch rate. Where the batch tier is
half price, charging a batch item at the sync rate is wrong by a factor of two, and a
cost record that is wrong is worse than one that is absent.

The ledger says which rule paid for each row:

```ts
// Which pricing rule the snapshot came from. 'sync' is resolved when the request
// completed. 'batch' is the job's submit-time snapshot, copied verbatim at
// settlement. Null predates the column and reads as 'sync'.
priceMode: text('price_mode'),

check(
  'request_log_batch_price_mode_compat',
  sql`${t.priceMode} IS DISTINCT FROM 'batch' OR ${t.batchId} IS NOT NULL`,
),
```

The database refuses a batch-priced row that names no job. A batch snapshot without a
settlement is a cost nobody can trace, and the constraint makes that unrepresentable
rather than unlikely. Batch and synchronous spend then separate cleanly in analytics,
which is the whole point of recording the mode.

## What it costs you

Three trades, stated plainly.

Anthropic's batch rate is not in LiteLLM's catalog. Every batch pair there belongs to
OpenAI, Azure, Gemini, or Bedrock and Vertex-hosted Claude, so a direct Anthropic
provider resolves an unknown batch rate unless you supply the pair through the manual
override. Recording Anthropic's published 50 percent as if it were catalog data would
be inventing a price, which is the one thing the cost rules forbid. Under a `block`
budget an unknown rate means the submission is refused with `batch_unbounded`, because
a budget cannot bound what it cannot price.

Submitting reserves a ceiling rather than charging it. The reservation is the
worst-case cost of every item at the snapshotted rate, held as a distinct pending
component of the same atomic Redis counter your synchronous spend uses. Recording cost
only at settlement would let a fifty-dollar cap wave through a five-hundred-dollar
batch, and would let two concurrent batches each fit the same remaining budget. The
dashboard says which of the two numbers you are looking at. An item with no
`max_tokens` on a model with no known output cap cannot be bounded either, and gets the
same refusal.

Once an upstream accepts a job, it is bound to that provider for the window. A tier's
fallback chain applies at submission and never again.

## Where this leaves you

v0.17.0 runs four additive migrations on boot and needs no configuration. Batch is on
by default. `BATCH_ENABLED=false` refuses new submissions while the poller, reads,
results and cancel keep serving, so in-flight jobs drain instead of stranding.

```bash
curl -fsSL https://polyrouter.app/install.sh | sh
```

Point a nightly job at `/v1/batches` and read the Batches page while it runs. If you
already run polyrouter against an OpenRouter provider, the model list also gets shorter
on the first boot after the upgrade, and every id it lost was one you could not have
called anyway. The source is at
[github.com/izzoa/polyrouter](https://github.com/izzoa/polyrouter) under AGPL-3.0.