Quickstart: OpenAI-compatible API

One line changes. Point your existing client at the Airmet sidecar running in your VPC. Started with no policy named — on the command line or in a config file — it twinlegs at α = 0.3: on about 30% of requests both legs race from the first byte and you are streamed whichever produces a token first. The other leg is not cancelled for being slower: it runs to its own finish, unless the runaway guard stops it for looping or running far past the winner, and its answer is discarded, so its prompt cache stays warm for the conversation's next turn. The remaining ~70% of requests go to one leg alone. α is the one dial: the fraction of requests that race.

This page covers two of the three setups: two provider keys (one leg per provider — the independence the race depends on) and one provider (one key, raced against itself: both legs on one provider's queue — the fallback, and the shape of run R1). For the first setup — one marketplace key with each leg pinned to a different underlying provider — see provider pinning.

# before: base_url = "http://your-engine:8000/v1"
client = OpenAI(base_url="http://airmet:8080/v1")

resp = client.chat.completions.create(
    model="qwen3-8b", messages=msgs,
    extra_headers={"X-Hedge-N": "2"},  # optional tuning
)
# resp includes hedge: {winner, n_launched, arms[].ttft_ms, cost.upstream_requests, …}

The hostname above is just the address of your own Airmet sidecar — call it whatever your network calls it. Prompts, completions and keys go only to the providers you configure, never to us. airmet up listens on 127.0.0.1 unless told otherwise: a sidecar your app reaches over the network needs --host 0.0.0.0 and, since it spends your keys, --auth-token-env (see install channels).

The sidecar sits between two things that already speak OpenAI HTTP — your client and your providers. Nothing about your app, prompts, or tools changes.

Start the sidecar

Lead with policy: twinleg in a hedger.yaml. Twinleg does not cancel the slower leg when the winner arrives, so it needs no cancel-billing verdict: direct provider keys race with the defaults, as do a marketplace key with pinned legs and a pair of engines you run. Two provider keys need a key per leg:

replicas:
  - id: leg0                       # the leg with the HIGHER rate limit goes first
    engine: openai
    base_url: https://api.provider-a.example/v1
    model: <model-id>
    api_key_env: PROVIDER_A_KEY
    extra_body: {reasoning_effort: low}   # only if the model takes it — see below
  - id: leg1
    engine: openai
    base_url: https://api.provider-b.example/v1
    model: <model-id>
    api_key_env: PROVIDER_B_KEY
    extra_body: {reasoning_effort: low}
hedge:
  policy: twinleg                  # keeps the slower leg running, warm
  alpha: 0.3                       # the fraction of requests that race

$ airmet up --config hedger.yaml

Four things in that file are deliberate:

• Opaque leg ids (leg0, leg1). Telemetry, on by default, reports per-leg counts under each id exactly as written here, so an id like the provider's name would leave the machine as a vendor name.
• The leg with the higher rate limit first. At α = 0.3 about 70% of requests go to the first leg alone. From release 0.11.0, when that leg fails before it sends a token (a connection error, a timeout, a 429, a 5xx, or a 401, 402, 403, 404 or 408 answer), the request is sent once to the second leg, and the response's hedge.rescue reads solo_failed. A first leg that sends nothing at all for half the first-token deadline (15 s of the default 30 s) is raced by the second (solo_hung). Nothing is sent again once the first leg has sent a token, and a request relayed unraced (below, X-Airmet-Bypass) goes to the first leg only and is not retried. The retry is a second upstream request, so the leg with the higher limit still goes first. hedge.rescue_solo: false turns it off.
• Per-leg request fields go in extra_body. A request carrying a field the sidecar does not know — reasoning_effort and metadata are two — is relayed to the first leg alone, unraced, with X-Airmet-Bypass: unknown_key, and a failure on that leg is not retried on the second. Set such a field on each leg in the file and those requests race.
• The keys stay in the environment. The file names each key's variable, never the key.

Confirm both legs started. The boot log prints one admitted line per leg; check for both. The banner says 2 leg(s) even when a leg failed to start. With one of two legs down the sidecar keeps serving from the other, and marks the requests that would have raced hedge.passthrough: degraded_pool; proxy.on_unhealthy: error answers 503 (no_agreeing_pool) instead. Until a dead first leg is taken out of the pool (two failed health probes, about 20 seconds), the requests that run on it alone are sent to the second leg instead of failing, except those relayed unraced. A leg whose key is refused (401, 402 or 403) on two requests in a row is taken out as well, although its model list still answers; it returns once a warm-up request with that key succeeds.

… replica leg0 admitted L0, guarantee=stateless
… replica leg1 admitted L0, guarantee=stateless

Confirm a race. A raced request's hedge block reads n_launched: 2 (1 means it ran alone) and names the winner; X-Airmet-Bypass: <reason> on a response means it was not raced, and says why. A request the second leg took over after the first leg failed also reads n_launched: 2, but its block has a rescue object. airmet status shows how many legs are healthy and the wins from your local records.

A config file races the policy it names. One that names none gets twinleg at α = 0.3 anyway, so the hedge block above is optional. hedge.default_n: 1 (or hedge.alpha: 0) serves one leg alone, with no retry on the other, and the banner says what each run races.

One provider takes flags alone. Give its endpoint once and the sidecar races it against itself: a twin leg on the same endpoint and model, under its own id, so it keeps its own connections and its own session id — and with them its own warm cache. The key is named by its environment variable and never appears in argv:

$ airmet up --providers https://api.provider/v1 \
    --model-name <model-id> --api-key-env PROVIDER_KEY
note: one provider — twinleg races it against itself: legs r1 and r1-twin are the same endpoint and model, …
airmet up · Go data plane · 2 legs on 1 provider · twinleg α=0.3 · /v1 on 127.0.0.1:8080 · run dir airmet-results/up-…

--hedge-n 1 serves the provider alone instead. Listing the endpoint twice gives the same race under ids you choose.

When a conversation's next turn arrives while the leg kept from its last turn is still finishing, that turn runs on one leg: a leg never carries two requests of the same conversation at once. Clients that send no task id are told apart by their prompt cache key, else by the conversation's opening messages.

Cost on metered APIs

If your upstream meters requests (RPM/RPD/TPM tiers), the default spends three things, all set by α:

• Request units. A raced request issues two upstream requests and a solo one issues one, so request units run at 1 + α: about 1.3× at the default. A solo request whose first leg fails is sent to the second leg and issues two as well; those extra requests follow your failure rate, not α.
• Tokens. The slower leg is not cancelled, so a raced request bills both legs — in full, except a leg the runaway guard stops for looping or running far past the winner. The warm cache is what keeps a race near the price of one more request rather than a cold re-read of the whole prompt — but a leg that sat out some turns still reads the turns it missed, uncached, the next time it races, so on prompt-heavy conversations tokens grow faster than request units do.
• Tier steps. On quota-tiered plans the real bill delta is a tier step, not a token delta: restoring the throughput you had before racing means buying the next tier.

By the mechanism, cost ≈ 1 + α. Measured within one run on a cross-vendor pair (n = 1,000 per arm, run hy3-xvendor-alpha-sweep-20260911), it was 1.08 / 1.19 / 1.40 / 2.22× at α = 0.1 / 0.3 / 0.5 / 1; your workload sets its own figure.

Every response's hedge block counts the upstream requests it made (cost.upstream_requests), so the request multiplier is in your own records from request one. On this plane the block's token fields are your provider's own counts: arms[].tokens is each leg's completion, and cost.extra_tokens and cost.extra_prompt_tokens add up the legs that did not win. A leg still running when the response is written (the kept loser) reads null, not yet known, and so do the sums that need it; its final counts land in your local records once it finishes (cost.kv_gb_s_est is always null here). Lower α (say 0.1) on a request-capped key, or raise it to 1 to race every request; on keys with hard daily request caps, mostly don’t race at all.

A policy that cuts the loser — not on direct keys

Name a policy and the sidecar runs it instead of twinleg. --policy deferred-p95 (the default before twinleg) fires the second leg only once the first is already slow; --policy immediate races every request. Both cancel the loser, and so race only legs whose abort-billing verdict is a measured refund. No direct-key route has one, so on direct keys the second leg never fires under these policies: stay on twinleg. --on-no-refund allow races regardless, the loud override for a pilot, and the cancelled leg may still bill. airmet status shows each leg's verdict.

A cancelled loser still consumes its full prompt prefill against request/TPM quota and (typically) your token bill: a cancel saves only the output the loser had not yet produced, and only on providers that propagate cancellation.

Tuning headers

Every response declares the winner, each leg’s first-token time (arms[].ttft_ms) and how many upstream requests it made in the hedge block, so you can measure the tail you just cut on request one.