The prompt-cache redundancy layer for LLM inference

Near-Pareto LLM cost-latency.Same models, same providers.Change one URL.

On a share α of your calls (30% by default), Airmet races a second copy across your providers and streams whichever answers first. The other leg is not cancelled for being slower: it stays a warm second copy of your conversation's prompt cache on another provider, synced at rate α, cost ≈ 1+α. If a provider drops, the other is already warm.

Join the pilotInstall →

runs in your infra · prompts never reach us

1,000 turns · one conversationmeasured · run gemini-2.5-flash-lite-selfhedge-20260909
solo · Provider P
solo · Provider P
twinleg · α = 1.0
hover a turncommitted — ✓first token · leg 0 — · leg 1 —standby — cachedserved by —
leg 0 · Provider Pleg 1 · Provider Psolo turn

width and height: the turn's time · colour: the leg that answered · twinleg row is shorter than the best solo

turn1,000of 1,000
sync rate
1.2×p99 first token vs best solo, at α = 1
≈ 1+αcost vs solo: a second leg on a share α of turns

measured · run gemini-2.5-flash-lite-selfhedge-20260909

The control room

Behind every tape, our observatory: one monitor per model · provider · workload, always-on · measured continuously.

Our observatory, per model · provider · workload. Colour on the live row is the measured tail band; the lines are schematic.

control roomalways-on · measured continuously
ingestper-request records
aggregateper model · pair
calibratelearned priors
servesigned bundle
the forecast stage stays internal ▸ only the priors leave

One bundle, every sidecar. What the control room learns is cached in each sidecar and consulted only while its own live window is cold — live measurement always wins, and nothing touches the request path. Pseudonymous per-request records in, priors out; prompts never travel.

one sidecar · the requesttwinleg · n=2
Simulated: pairs drawn from run R1's measured solo first-token timessample 1/200
leg 0
leg 1
slower draw: —first token: —
your dashboardmeasured · run R1
Measured: time to first token · run R1, mid-run read
raced n=2, same providersolo, same provider

A mid-run read of run R1 (main-20260812T033124Z, requests up to 2026-08-12 09:41 UTC): first-token times of one provider raced against itself (n=4,169) vs that provider solo (n=4,378), same model; that run cancelled the slower leg. 4 solo requests never answered by 15s — the solo curve tops out below 1. Every raced request answered. The finished run's p99s are in the numbers below. Log x-axis; curves smoothed monotonically through the empirical steps (shown faint); requests that missed the deadline are annotated, never interpolated.

524 msp99 first token, vs 9.8 s solo · run R1, n=7,638
≈ 1+αcost vs solo · measured 1.08 / 1.19 / 1.40 / 2.22× at α = 0.1 / 0.3 / 0.5 / 1 · within-run, cross-vendor pair, n = 1,000 per arm, run hy3-xvendor-alpha-sweep-20260911
0 / 15,257requests died while racing · run R1

how it works ↓the numbers, scoped ↓

How it works · one request, two legs
first token ✓streamed to youcompleteskept warm · standby
Point your client at Airmet.One base URL. Prompts, tools and model stay exactly as they are.
Every turn can get a second leg.On an α fraction of turns a copy goes to a second provider. Each leg keeps its own session, so both prompt caches stay warm.
First token wins. The other leg keeps going.The tail disappears; the loser is not cancelled for being slower — it finishes in the background, unless the runaway guard stops it for looping or running far past the winner, and stays warm for the next turn. Cost ≈ 1+α.
Start here

Two legs, one URL. Pick where the legs come from.

Path one · one marketplace key, two pinned providers

The shape our pilot runs: one account on a major model marketplace, each leg pinned to a different underlying provider, so the two legs fail and stall independently.

# install and start the sidecar in your infra (pip: inside a virtual environment)
$ pip install --index-url https://airmet.ai/simple/ airmet
$ airmet up --config hedger.yaml
airmet up · Go data plane · 2 leg(s) · twinleg α=0.3 · /v1 on 127.0.0.1:8080 · run dir airmet-results/up-…
# hedger.yaml — one marketplace key, two pinned providers
replicas:
  - id: leg0
    engine: openai
    base_url: https://marketplace.example/api/v1
    model: <model-id>
    api_key_env: MARKETPLACE_API_KEY
    provider_routing:              # pin leg 0
      only: [provider-slug-a]
      order: [provider-slug-a]
      allow_fallbacks: false
  - id: leg1
    engine: openai
    base_url: https://marketplace.example/api/v1
    model: <model-id>
    api_key_env: MARKETPLACE_API_KEY
    provider_routing:              # pin leg 1 elsewhere
      only: [provider-slug-b]
      order: [provider-slug-b]
      allow_fallbacks: false
hedge:
  policy: twinleg                  # keeps the slower leg running, warm
  alpha: 0.3                       # sync rate: the fraction of turns that get the second leg
  mode: throughput                 # or ttft
Path two · two provider keys

Keys for two providers that serve the same model: the same file, one key per leg. Nothing else changes.

# hedger.yaml — two provider keys, one leg each
replicas:
  - id: leg0                       # the leg with the higher rate limit first
    engine: openai
    base_url: https://api.provider-a.example/v1
    model: <model-id>
    api_key_env: PROVIDER_A_KEY
  - id: leg1
    engine: openai
    base_url: https://api.provider-b.example/v1
    model: <model-id>
    api_key_env: PROVIDER_B_KEY
hedge:
  policy: twinleg                  # keeps the slower leg running, warm
  alpha: 0.3                       # sync rate: the fraction of turns that get the second leg
  mode: throughput                 # or ttft

Same provider twice — the shape of run R1, when you hold one key and want the tail cut today:

$ airmet up --replicas https://api.provider/v1,https://api.provider/v1 \
    --engine openai --model-name <model-id> --api-key-env PROVIDER_KEY

Your own engines — two vLLM or SGLang replicas you already run (never a single pool):

$ airmet up --replicas http://gpu1:8000,http://gpu2:8000
Then, in your app

The sidecar is a local proxy, so your language is irrelevant — Node, Go, and Python clients all just change base_url. Prompts, tools and model choice stay exactly as they are.

client = OpenAI(base_url="http://127.0.0.1:8080/v1")
Join the pilot
$ pip install --index-url https://airmet.ai/simple/ airmet

The install page walks a bare box through every channel — pip (inside a virtual environment), npm, Docker, a bare download — and the docs cover vLLM / SGLang, LiteLLM, n8n and coding tools.

Measured, with the caveats attached

Four numbers, each scoped to what it actually proves.

524 ms vs 9.8 sp99 first-token latency, the same provider raced against itself vs that provider solo — same model, same key, n=2 (run R1, finished 2026-08-12; n=7,638 raced vs 7,571 solo). The tail it removes is the provider's own variance.
911 ms vs 9.8 sp99 first-token latency, a cross-provider race vs Provider A solo, same run and model (n=7,619 vs 7,571). The significance claim is the per-block sign test — the race won 26 of 32 paired blocks, p=0.0005 — not the pooled delta.
0 / 15,257requests died while racing, both racing arms of run R1. With zero observed, the failure rate's Wilson 95% upper bound is 0.025% — an upper bound, not a proof of zero.
100%of turns completed while one provider's free-tier front refused about 40% of them: 231/231 and 320/320 in two pairs containing it, against 193/321 (60.1%) for that front alone. Observatory runs of 2026-08-10; the refusals were rate limits under our own pacing.
What it costs · measured

≈ 1+α. The second leg runs on a share α of turns — the rate you set — and is not cancelled for being slower: it runs to its own finish unless the runaway guard stops it for looping or running far past the winner, so its prompt cache stays warm. Measured, the dial cost 1.08 / 1.19 / 1.40 / 2.22× at α = 0.1 / 0.3 / 0.5 / 1 (within-run, cross-vendor pair, n = 1,000 per arm, run hy3-xvendor-alpha-sweep-20260911, 2026-09-11).

Confirmed with the standard tool

A real race on the field's standard serving-latency benchmark (vLLM benchmark_serving, 400 attempted per arm, 2026-08-31, GLM-5.2, run glm-5.2-2026-08-31): two pinned providers on a major model marketplace, the same prompts, only the race differing. End-to-end p99 69.6 s → 9.3 s (7.5×) over completed, non-empty responses (386 solo, 400 raced); first-token p99 11.1 s → 2.7 s. 13 solo requests hit the 300-second cap and 1 returned nothing, against none raced. The median is about tied — end-to-end p50 7.8 s solo vs 6.8 s raced (1.14×) — and solo finished first in 98 of 386 pairs: one model, one pair, one window. Method and denominators on the evidence page.

The first three tiles are run R1 (main-20260812T033124Z) — completed requests on llama-3.3-70b, the same model on both legs, through a major model marketplace. That run cancelled the slower leg, which is no longer the default: today's twinleg keeps it running, which is what the dial above prices. Every p99 here uses one convention (completed requests, linear-interpolated; deadline misses counted separately). Vendor identities are anonymized and shareable under diligence.

Privacy & consent

The sidecar runs in your infra. We never see your prompts.

In your infra

Plainly: the sidecar runs in your infra. We never see prompts, keys, or outputs. The one URL change points at your own instance. Nothing you type in the signed-in dashboard reaches a model either: its demo replays recorded data.

Telemetry · on by default

Telemetry is on by default. It sends one record per request: its timings and sizes, its outcome (the HTTP status and error class), each leg's route (provider, model, region), token counts and cost, the decision taken, and a session link (a random session id, the turn, the turn it continues), with labels as you configured them — pool, scenario, your legs' ids as you named them — plus a random install id created on your machine at first run (am_…), not derived from your hardware, name or account. Each push also carries the sidecar's version, your policy settings and each leg's route, its request counters, the state of its priors feed and any cost-guard trip. Never prompts, completions, keys, error bodies or URLs. Opt out anytime with --no-telemetry: the pushes stop and the racer works identically without it; AIRMET_TELEMETRY_SCHEMA=v1 sends aggregate windows instead.

The route feed

The live route feed comes with airmet login, which links the machine to your org — not with telemetry. Linking is identity, not consent: a linked install that has opted out pushes no telemetry, and whenever it registers or polls the feed it still sends its token and its install id — and, to the route feed, the public catalog routes its legs use (for a marketplace endpoint, with the price its catalog lists).

FAQ

The questions infra owners actually ask.

Do I route my traffic through you?

Self-hosted: no. The sidecar runs in your infra next to your app, and the one URL change points at your own Airmet instance. Prompts and completions travel between your app, the sidecar and the providers you configured — never through us — and your keys go only to the providers they belong to. What does reach us is the telemetry above, which you can turn off, and what the feed needs once you link. The signed-in dashboard sends nothing you type to a model: its demo replays recorded data.

Which two providers?

Two that fail and stall independently. The win comes from one leg being fast at the moment the other is slow, so two providers with their own queues beat any single one, and one marketplace key pinned to a different provider per leg gives you that without a second account. Two uniformly slow providers barely help — our in-loop runs showed it — and the same provider raced against itself is the fallback when you hold one key: that is the shape of run R1. A marketplace's own routing sorts providers by an average and sends you to one of them; twinlegging is tail-case racing for the request already in flight, and it can run on top of any route.

On Vercel or Cloudflare Workers?

There is no place for a long-lived sidecar on an edge platform, so the edge door is protocol-first: an edge-worker template that runs the same race inside your worker. Honestly: it is coming next, not shipped. If your app lives on an edge platform, say so when you join the pilot — it decides what we build next.

What does the pilot cost?

Nothing. The pilot is free: the sidecar and every policy it ships, its Prometheus metrics and local records, and the full live route feed for your org for 90 days from its first linked install. What the pilot covers, and how we handle your data, is in the pilot terms.

Do you store my prompts?

No prompt custody, ever — it is a deliberate non-feature. The self-hosted sidecar sends prompts only to your providers, and the telemetry it sends us carries timing, size, token-count and cost metadata, never prompt or completion text and never keys. The local records it writes on your own machine are a different matter: they hold the text each leg answered with (its reasoning and tool calls too) and the start of any error a provider returned, so keep them as you keep your app's logs. Every field telemetry carries is listed on What we send, and airmet traffic shows how to check the telemetry pushes byte for byte in your own packet capture. Latency and cost observability without holding your prompts.

The pilot

Run it on your own traffic, with us beside you.

Join the pilot

Or start on your own: Start here, the install page, and the method and denominators behind the standard-tool race on the evidence page.

No analytics until you say so. One first-party cookie remembers this choice; allowing adds EU-hosted usage analytics and error reporting (what we send); declining changes nothing about how the site works.