Quickstart: vLLM / SGLang

One command. Front the replicas you already run. Airmet health-checks each endpoint, then races them, so a request stuck behind one slow replica can be answered by another. It runs in your VPC next to the replicas — prompts and completions never leave your network.

You need at least 2 replicas — don't race a single saturated pool. With one replica, the second leg lands on the same queue that made the first request slow: you double the load on the saturated engine and cut nothing. Two or more replicas give the race independent queues to draw from. Given one URL, airmet up races it against itself by default — right for a provider's API, with a fleet behind it, and wrong for a single GPU of your own: front one replica with --hedge-n 1.

$ pip install --index-url https://airmet.ai/simple/ airmet                   # inside a virtual environment
$ airmet up --replicas http://gpu1:8000,http://gpu2:8000,http://gpu3:8000 \
    --model-name <model-id>
airmet up · Go data plane · 3 leg(s) · twinleg α=0.3 · /v1 on 127.0.0.1:8080 · run dir airmet-results/up-…

The sidecar listens on 127.0.0.1:8080. For clients on other machines in the VPC, add --host 0.0.0.0 --auth-token-env NAME; they then send Authorization: Bearer <token> (see install channels).

What happens on each request

1. Race about 30% of requests. With no policy named, Airmet twinlegs at α = 0.3: on about 30% of requests it sends the request to two replicas (n=2 by default) and streams whichever produces the first token. The rest go to one replica alone.
2. Keep the loser warm. The slower leg is not cancelled for being slower. It runs to its own finish, unless the runaway guard stops it for looping or running far past the winner, and its answer is discarded, so that replica's prefix cache holds the conversation for its next turn. On your own GPUs the price is compute: each raced request runs one more generation.
3. Audit. Every response carries the winner, each leg’s first-token time (arms[].ttft_ms) and how many upstream requests it made, so you can measure the tail you cut. Its token fields are the engine's own usage counts; a leg still running when the response is written reads null, not yet known, and its final usage lands in your local records once it finishes.

To cancel the loser instead, name a policy that cuts it: --policy deferred-p95 fires the second leg only once the first is already slow, and --policy immediate races every request. Replicas given by URL carry no abort-billing verdict, so on this plane a cutting policy races them only with --on-no-refund allow — on your own GPUs a cancel frees the replica rather than billing it. The native SGLang plane (--engine sglang) keeps its classic default, deferred-p95, and cancels the loser.

SGLang and vLLM pools are both fronted the same way — any OpenAI-compatible engine works. Production fleets already run ≥2 replicas for availability, so on your own GPUs the replicas race each other and the extra load lands on stranded capacity, not new rental.