Install Airmet with a coding agent

For the agent reading this. Your human asked you to install Airmet. This page is your runbook. Follow it in order, ask your human at every step marked Ask, and finish with the "Done when" report at the end. It describes release 0.11.2. The same page as plain markdown, for fetching: https://airmet.ai/install-for-agents.md.

1. What Airmet is

Airmet is one static binary, airmet, that runs next to your human's app as a local OpenAI-compatible endpoint. It sends a share of the requests (α, 30% by default) to two model providers at once, streams whichever answers first, and lets the slower one keep running so its prompt cache stays warm (only the runaway guard stops one that loops or runs far past the winner). The app changes one setting: its base URL.

Prompts, completions and provider keys go only to the providers your human configures. A raced request is billed by both providers, so upstream requests run at about 1 + α of what they were.

The other requests go to the first leg alone. From release 0.11.0 such a request is not lost when that leg fails before it sends a token: it is sent once to the second leg, and the response says so. The exception is a request the sidecar relays unraced, which has an X-Airmet-Bypass header (section 6). A first leg whose key is refused is taken out of the pool. Section 6 has the details and check 8 in section 9 shows it working.

2. Rules for you, the agent

1. Never print, log or commit a key. Keys live in environment variables. hedger.yaml names the variable (api_key_env), never the value. Do not cat an env file, run env or printenv without a name, or type a key into a command line. Check that a variable is set without showing it: [ -n "$(printenv NAME)" ] && echo set || echo missing.
2. Never change providers, models, keys, α or the policy without asking. Stay on policy: twinleg: it does not cancel the slower leg when the winner arrives, which is why it races on any pair of keys with the defaults.
3. Telemetry is your human's decision. Do not start airmet up before they have chosen (section 3) from what the installed binary says it sends (airmet traffic), and never add or drop --no-telemetry, --telemetry or AIRMET_TELEMETRY_SCHEMA on your own.
4. Your human signs in, not you. airmet login is optional. If they want it, you relay the link and the code it prints, and they approve it in their own browser.
5. Listen on 127.0.0.1 only, unless your human says the app runs on another host or in a container. Then require a token (section 7).
6. No system changes without asking: no sudo, no system packages, no pip --break-system-packages, no change to macOS quarantine or firewall settings.
7. Change only the app setting your human pointed you to (the base URL), and show them the diff.
8. Verify every download against SHA256SUMS before you run it. If a check fails, delete the file and stop.
9. When a step fails twice, stop and ask. Report the exact error lines, with any key redacted.

3. Ask your human first

Ask these before you install anything. Defaults are in brackets.

Before I install Airmet I need a few decisions (defaults in brackets):

1. Setup: (a) one marketplace key, each leg pinned to a different provider,
   (b) two provider keys, one leg each, or (c) two OpenAI-compatible servers
   you run (vLLM, SGLang, ...)?
2. For each of the two legs: the OpenAI-compatible base URL, the model id there
   (the same model on both legs), and the NAME of the environment variable that
   holds its key. Not the key itself. For (a), also the two provider slugs to pin.
3. Which leg is the more reliable (the higher rate limit, the fewer errors)?
   It goes first: it serves every request that does not race. A request that
   fails there before its first token is sent once to the other leg, at the
   cost of a second upstream request. (A request the sidecar cannot race, for
   example one with a field it does not know, is relayed to the first leg only
   and is not retried.) [the first one you named]
4. Alpha, the share of requests that race both legs [0.3]. A raced request is
   billed by both providers, so upstream requests run at about 1 + alpha.
   With alpha above 0, the requests that do not race are still sent to the
   second leg when the first fails (the exception is in question 3), so alpha
   is a speed and cost setting, not a reliability one.
5. Telemetry is on by default (summary below). Keep it on, opt out, or send the
   older aggregate format instead?
6. Where does your app set its base URL (a file, an env var, a config key)?
   Does the app run on this machine, in a container, or on another host?
7. If one leg goes down: keep serving from the healthy leg [yes], or answer 503?
8. Install channel: bare binary [default], pip, npm or Docker?
9. May I send about 40 short test requests (16 tokens each) through your
   keys to verify it?
10. Link this machine to your Airmet org with `airmet login`? [no] You would
    approve it in your browser. It turns on the live route feed.

Read your human this telemetry summary as written, then record their choice:

Telemetry is ON by default. From release 0.11.0 it sends one record per
request it serves (telemetry v2), pushed gzipped every 30 seconds. The
first `airmet up` prints a notice listing all of it (shown once, and once more
if what is sent changes). A record describes a request, never its content:
  - timing: arrival, time to first token and to the end, each leg's times
  - outcome: the HTTP status and an error class (never an error's message
    or body), which leg won, whether a rescue answered
  - usage: each leg's token counts and the cost its provider reported
  - route: how each leg is reached, its provider, model and region (a host
    airmet does not know, or a model id naming your account, reports as
    "custom")
  - session: a random conversation id, the turn's number, the turn it continues
  - shape: request class, message counts and byte sizes
  - decision: the policy, how many legs it launched, why one was declined
  - labels as configured: pool, scenario, surface, and each leg's id exactly
    as named in hedger.yaml
Every push also carries a random install id (am_...) made on this machine at
first run and kept across restarts.
It never sends prompts, completions, API keys, error bodies, URLs or hostnames,
generation ids, or any hash of your text.

Options:
  keep it on        nothing to add
  opt out           --no-telemetry on the first start; saved for later runs
  aggregate format  AIRMET_TELEMETRY_SCHEMA=v1 on every start (the older
                    10-second aggregate windows instead of per-request records)
Racing works the same with telemetry off. Opting out is immediate and stops
every push; it does not delete the install id, which stays in the local config
file. If this machine is linked (airmet login), it sends its token and install
id whenever it registers or polls the route feed, and the public catalog routes
its legs use when it polls, even with telemetry off.

The full inventory of what leaves the machine is traffic transparency. The binary also describes itself: once it is installed (section 5), run airmet traffic, which makes no network calls, and show your human its payload: paragraph before the first start. That is this build's own list of what it sends. Where it differs from the summary above, the binary's list is the one your human decides on.

4. Detect the platform

Run these and keep the two values for the rest of this page:

uname -s; uname -m
case "$(uname -s)/$(uname -m)" in
  Linux/x86_64)              PLAT=linux-amd64;  NPMPLAT=linux-x64 ;;
  Linux/aarch64|Linux/arm64) PLAT=linux-arm64;  NPMPLAT=linux-arm64 ;;
  Darwin/arm64)              PLAT=darwin-arm64; NPMPLAT=darwin-arm64 ;;
  Darwin/x86_64)             PLAT=darwin-amd64; NPMPLAT=darwin-x64 ;;
  *)                         PLAT=unsupported ;;
esac
# A terminal under Rosetta reports x86_64 on Apple silicon:
if [ "$(sysctl -n hw.optional.arm64 2>/dev/null)" = 1 ]; then PLAT=darwin-arm64; NPMPLAT=darwin-arm64; fi
echo "PLAT=$PLAT NPMPLAT=$NPMPLAT"

Windows has a bare binary (airmet-windows-amd64.exe); this runbook covers macOS and Linux, and WSL counts as Linux. Anything else: stop and tell your human.

Many agent shells start fresh for every command, so PLAT and NPMPLAT do not carry over: set them again (PLAT=… with the value printed above) at the top of each command that uses them. The install blocks stop with an error when one is unset.

Also check what is available:

command -v curl; command -v sha256sum || command -v shasum
command -v python3 && python3 -c 'import sys; print(sys.version_info >= (3, 9))'
command -v node && node --version          # npm needs Node 18 or later
command -v docker
ldd --version 2>&1 | head -1               # "musl" (Alpine): use the bare binary

If command -v curl prints nothing (a stock python:3.12-slim image has none), stop and ask your human to install curl, or to let you: every install block in section 5 and most checks in section 9 use it, and a system package is theirs to allow (rule 6). On Debian and Ubuntu the packages are curl and ca-certificates.

Pick the channel your human chose. With no preference, use the bare binary: one file, one checksum, nothing else to install.

5. Install, with checksum verification

Work in a directory of its own, so the run records (airmet-results/) never land in the app's repository. Every channel below leaves the command at $HOME/airmet/bin/airmet, and every later block starts with cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH", so a shell that starts fresh for each command still finds it:

mkdir -p "$HOME/airmet" && cd "$HOME/airmet"

Every file comes from https://airmet.ai/dl/v0.11.2/, and its SHA256SUMS lists every artifact of the release. Each block below defines sha256check (sha256sum on Linux, shasum -a 256 on macOS) and installs or runs nothing unless every check prints <file>: OK (a file listed twice prints it twice). On anything else (FAILED, no properly formatted … lines) the block deletes the download and prints CHECKSUM FAILED: stop there and tell your human.

Bare binary (the default)

cd "$HOME/airmet"
: "${PLAT:?set PLAT to the value section 4 printed}"
sha256check() { if command -v sha256sum >/dev/null 2>&1; then sha256sum -c -; else shasum -a 256 -c -; fi; }
curl -fsSLO "https://airmet.ai/dl/v0.11.2/airmet-$PLAT"
curl -fsSLO "https://airmet.ai/dl/v0.11.2/SHA256SUMS"
if grep " airmet-$PLAT\$" SHA256SUMS | sha256check; then     # airmet-<platform>: OK
  mkdir -p bin && mv "airmet-$PLAT" bin/airmet && chmod +x bin/airmet
  export PATH="$HOME/airmet/bin:$PATH"
  airmet version
else
  rm -f "airmet-$PLAT"; echo "CHECKSUM FAILED: nothing installed. Stop and tell your human."; false
fi

airmet version prints airmet v0.11.2 (go… <os>/<arch>).

pip (Python 3.9 or later; macOS, or Linux with glibc)

cd "$HOME/airmet"
sha256check() { if command -v sha256sum >/dev/null 2>&1; then sha256sum -c -; else shasum -a 256 -c -; fi; }
python3 -m venv .venv
.venv/bin/pip download --no-deps --only-binary=:all: \
    --index-url https://airmet.ai/simple/ "airmet==0.11.2" -d wheels
curl -fsSLO "https://airmet.ai/dl/v0.11.2/SHA256SUMS"
ok=no
for w in wheels/airmet-0.11.2-*.whl; do
  if grep " ${w#wheels/}\$" SHA256SUMS | (cd wheels && sha256check); then ok=yes; else ok=no; break; fi
done
if [ "$ok" = yes ]; then
  .venv/bin/pip install --no-index --no-deps wheels/airmet-0.11.2-*.whl
  mkdir -p bin && ln -sf "$HOME/airmet/.venv/bin/airmet" bin/airmet
  export PATH="$HOME/airmet/bin:$PATH"
  airmet version
else
  rm -rf wheels; echo "CHECKSUM FAILED: nothing installed. Stop and tell your human."; false
fi

pip install --index-url https://airmet.ai/simple/ airmet is the one-line form: pip checks each file against the hash the index lists. The download above adds the check against SHA256SUMS. The wheel is the same binary plus a short Python shim with no dependencies.

npm (Node 18 or later)

cd "$HOME/airmet"
: "${NPMPLAT:?set NPMPLAT to the value section 4 printed}"
sha256check() { if command -v sha256sum >/dev/null 2>&1; then sha256sum -c -; else shasum -a 256 -c -; fi; }
# a package.json of its own: npm refuses to install "airmet" into a project named airmet
[ -f package.json ] || printf '{"name":"airmet-sidecar","private":true,"version":"0.0.0"}\n' > package.json
curl -fsSLO "https://airmet.ai/dl/v0.11.2/airmet-cli-$NPMPLAT-0.11.2.tgz"
curl -fsSLO "https://airmet.ai/dl/v0.11.2/airmet-0.11.2.tgz"
curl -fsSLO "https://airmet.ai/dl/v0.11.2/SHA256SUMS"
ok=yes
for f in "airmet-cli-$NPMPLAT-0.11.2.tgz" "airmet-0.11.2.tgz"; do grep " $f\$" SHA256SUMS | sha256check || ok=no; done
if [ "$ok" = yes ]; then
  npm install --no-audit --no-fund "./airmet-cli-$NPMPLAT-0.11.2.tgz" "./airmet-0.11.2.tgz"
  mkdir -p bin && ln -sf "$HOME/airmet/node_modules/@airmet/cli-$NPMPLAT/bin/airmet" bin/airmet
  export PATH="$HOME/airmet/bin:$PATH"
  airmet version
else
  rm -f "airmet-cli-$NPMPLAT-0.11.2.tgz" "airmet-0.11.2.tgz"; echo "CHECKSUM FAILED: nothing installed. Stop and tell your human."; false
fi

bin/airmet links the platform package's binary itself, so every airmet command on this page runs as written. Use that, not npx airmet, for the long-running airmet up: npx waits in front of the binary as one more process to supervise. Up to release 0.11.1, stopping npx on Linux left the binary running; from release 0.11.2 the launcher stops it with its parent.

Docker (the image is linux/amd64)

On an arm64 host the image runs under emulation: prefer the bare binary there.

cd "$HOME/airmet"
sha256check() { if command -v sha256sum >/dev/null 2>&1; then sha256sum -c -; else shasum -a 256 -c -; fi; }
curl -fsSLO "https://airmet.ai/dl/v0.11.2/airmet-image-v0.11.2.tar.gz"
curl -fsSLO "https://airmet.ai/dl/v0.11.2/SHA256SUMS"
if grep " airmet-image-v0.11.2.tar.gz\$" SHA256SUMS | sha256check; then
  docker load -i "airmet-image-v0.11.2.tar.gz"     # Loaded image: airmet:v0.11.2
  docker run --rm airmet:v0.11.2 version
else
  rm -f "airmet-image-v0.11.2.tar.gz"; echo "CHECKSUM FAILED: nothing installed. Stop and tell your human."; false
fi

The whole archive is checked before docker load, so the image's configuration and every layer are covered, not only the binary. If SHA256SUMS has no line for the image archive, it cannot be verified: do not load it, and use the bare binary instead.

6. Write hedger.yaml

Write $HOME/airmet/hedger.yaml from the template for your human's setup. These are the tested pilot configs, with the provider details left for you to fill in. Every template follows the same rules:

• Opaque leg ids: leg0, leg1. Telemetry reports per-leg counts under each id exactly as written, so an id named after a provider or a host would leave the machine as that name.
• The more reliable leg first. At α = 0.3 about 70% of requests go to the first leg alone, so put the provider with the higher rate limit and the fewer errors there. When it fails, the retry in the next bullet costs a second upstream request and the time the failed attempt took.
• A request the first leg fails is sent to the second leg. From release 0.11.0, a request that runs on the first leg alone (α above 0) and fails before that leg sends a token is sent once to the second leg: after a connection error, a timeout, a 429, a 5xx, or a 401, 402, 403, 404 or 408 answer. Its response carries hedge.rescue with "kind":"solo_failed", and cost.upstream_requests reads 2. A first leg that sends nothing at all, not even response headers, for half the first-token deadline (15 s of the default 30 s) is raced by the second leg ("kind":"solo_hung"), and the silent leg is cut if the second answers first. A request is never sent again once the first leg has sent a token (a leg that dies mid-answer still ends the stream), when the first leg refuses it as malformed (400, 413, 422), or when only one leg is healthy. It is on under policy: twinleg; hedge.rescue_solo: false turns it off, and only your human decides that. It covers the requests the sidecar serves itself. A request it relays unraced (its response carries X-Airmet-Bypass: <reason>: a field the sidecar does not know, response_format without supported_params, n above 1, and a few more, which the troubleshooting table in section 12 lists) goes to the first leg only, as before: nothing is retried on the second.
• A refused key takes its leg out. From release 0.11.0, a leg whose key is refused (401, 402 or 403) on two requests in a row is taken out of the pool until a warm-up request with that key succeeds. Some providers answer the model-list probe with 200 whatever the key, so the probe alone never sees a revoked key. Those requests are retried on the other leg like any other failure.
• Keys from environment variables only: api_key_env names the variable.
• policy: twinleg. The policies that cancel the loser (immediate, deferred-p95, deferred-adaptive, tied) race only legs whose cancel billing has a verdict, and direct keys have none, so on them the second leg never fires.
• Per-leg request fields go in extra_body. A request that carries a field the sidecar does not know, such as reasoning_effort or metadata, is relayed to the first leg alone, unraced, and not retried on the second if that leg fails. Set the field on each leg instead.
• model.name is what /v1/models advertises. The name is checked, not routed: each leg sends its own model, and a client may send any name.

(a) One marketplace key, each leg pinned to a different provider

replicas:
  - id: leg0
    engine: openai
    base_url: <the marketplace's OpenAI-compatible base URL>
    model: <model id on the marketplace>
    api_key_env: MARKETPLACE_API_KEY
    provider_routing: {only: [<provider-slug-a>], order: [<provider-slug-a>], allow_fallbacks: false}
  - id: leg1
    engine: openai
    base_url: <the same base URL>
    model: <the same model id>
    api_key_env: MARKETPLACE_API_KEY
    provider_routing: {only: [<provider-slug-b>], order: [<provider-slug-b>], allow_fallbacks: false}
model:
  name: <the model name your human's app sends>
hedge:
  policy: twinleg
  alpha: 0.3

Pick a model that two or more providers serve on the marketplace, and keep allow_fallbacks: false: otherwise both legs can land on the same provider and the race is no longer independent. provider_routing is forwarded as each request's provider field on that leg.

(b) Two provider keys, one leg each

replicas:
  - id: leg0                       # the leg with the higher rate limit
    engine: openai
    base_url: <provider A's OpenAI-compatible base URL, ending in /v1>
    model: <model id at provider A>
    api_key_env: PROVIDER_A_API_KEY
    extra_body: {reasoning_effort: low}   # only for a reasoning model; else delete this line
  - id: leg1
    engine: openai
    base_url: <provider B's OpenAI-compatible base URL, ending in /v1>
    model: <model id at provider B>
    api_key_env: PROVIDER_B_API_KEY
    extra_body: {reasoning_effort: low}   # keep both legs the same
model:
  name: <the model name your human's app sends>
hedge:
  policy: twinleg
  alpha: 0.3

(c) Two OpenAI-compatible servers you run

replicas:
  - id: leg0
    engine: openai                 # openai even for vLLM or SGLang
    base_url: http://<server-1>:8000/v1
    model: <model id served by both>
  - id: leg1
    engine: openai
    base_url: http://<server-2>:8000/v1
    model: <model id served by both>
model:
  name: <model id served by both>
hedge:
  policy: twinleg
  alpha: 0.3

Two servers on different GPUs or hosts race independent queues; two processes on one GPU mostly do not. Add api_key_env: NAME to a leg only if that server requires a bearer key.

Optional: the proxy block

Add it only for a choice your human made:

proxy:
  listen: 0.0.0.0:8080             # the default is 127.0.0.1:8080
  auth_token_env: AIRMET_TOKEN     # required whenever listen is beyond 127.0.0.1
  on_unhealthy: error              # the default, passthrough, serves from the healthy leg

Check the file

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
grep -n '<' hedger.yaml || echo "no placeholders left"
grep -o 'api_key_env: *[A-Za-z_][A-Za-z0-9_]*' hedger.yaml | awk '{print $2}' | sort -u |
  while read -r v; do if [ -n "$(printenv "$v")" ]; then echo "$v: set"; else echo "$v: MISSING"; fi; done

Every placeholder must be gone and every variable set. A variable your human keeps in a file is loaded in the same command that starts airmet, without printing it: set -a; . /path/they/gave/you; set +a.

7. Start it

Build the start command from two answers in section 3. Telemetry:

Where the app runs:

The second case puts the sidecar on the network, and whoever reaches it without the token could spend your human's keys. Ask first. Make the token once, without printing it, and ask your human where to keep it: the app needs the same value as its API key. In each later command that starts or checks the sidecar, load it from there.

export AIRMET_TOKEN="$(openssl rand -hex 32)"

Start it in the background, with its log in a file:

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
nohup airmet up --config hedger.yaml > airmet.log 2>&1 &   # plus your flags from the two tables
echo $! > airmet.pid
for i in $(seq 1 60); do grep -q 'serving /v1' airmet.log && break; kill -0 "$(cat airmet.pid)" 2>/dev/null || break; sleep 1; done
grep -E 'airmet up ·|admitted|failed L0 admission|rescue_solo|serving /v1|^error|address already in use' airmet.log

Expected, with your own run id and model:

airmet up · Go data plane · 2 leg(s) · twinleg α=0.3 · /v1 on 127.0.0.1:8080 · run dir airmet-results/up-…
… replica leg0 admitted L0, …
… replica leg1 admitted L0, …
… rescue_solo ON: a solo turn (alpha gate closed, or the lagging-arm guard) whose leg fails before its first token …
… airmet-hedgerd serving /v1 on 127.0.0.1:8080 (policy=twinleg α=0.3 (immediate, keep_losers) n=2 pool=[leg0 leg1] …)

Check for one admitted line per leg: the banner says 2 leg(s) even when a leg failed to start. The rescue_solo ON line says a failed first leg will be retried on the second (a hedger.yaml with hedge.rescue_solo: false has no such line). If your tool has its own way to run a command in the background, it can stand in for nohup … &, as long as the log lands in airmet.log. With telemetry on, the first start also prints the one-time notice: show it to your human. With --no-telemetry the log reads telemetry: OFF (flag; saved …).

Ask how it should keep running after your session: a terminal or tmux of their own, their service manager, or Docker with -d --restart unless-stopped.

Docker

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
docker run -d --name airmet -p 127.0.0.1:8080:8080 \
  -v airmet-data:/data \
  -v "$HOME/airmet/hedger.yaml:/data/hedger.yaml:ro" \
  -e PROVIDER_A_API_KEY -e PROVIDER_B_API_KEY \
  airmet:v0.11.2 up --config /data/hedger.yaml    # one -e per api_key_env in hedger.yaml
for i in $(seq 1 60); do docker logs airmet 2>&1 | grep -q 'serving /v1' && break; sleep 1; done
docker logs airmet 2>&1 | grep -E 'admitted|failed L0 admission|rescue_solo|serving /v1|^error'

-e NAME passes each key variable from your shell by name, so the value never appears in the command line. Inside the container the sidecar listens on every interface and the port mapping decides who reaches it: 127.0.0.1:8080:8080 keeps it on this machine. For an app on another host or in another container, publish it with -p 8080:8080, and add -e AIRMET_TOKEN to docker run and --auth-token-env AIRMET_TOKEN after up. Leave proxy.listen out of a hedger.yaml the container runs. The airmet-data volume holds the telemetry choice, the install id and the request records, so every container started on it is the same install: an opt-out given once with --no-telemetry holds for the containers after it. Keep -v airmet-data:/data in every docker run of this install. Without it a new container starts over: it shows the notice again, turns telemetry back on and counts as a new install. A host directory mounted on /data instead has to be writable by the image's user, uid 65532; when it is not, the error says so and how to fix it. -e AIRMET_TELEMETRY_SCHEMA=v1 selects the aggregate format, on each docker run. Run the other commands on this page as docker exec airmet /airmet ….

8. Point the app at it

The base URL is http://127.0.0.1:8080/v1. The API key can be any string: the sidecar sends each leg's own key and ignores the client's, unless it was started with --auth-token-env. The app keeps its model name. Change only the setting your human named, and show them the diff.

Python (OpenAI SDK):

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="unused")
r = client.chat.completions.create(model="<model.name>",
        messages=[{"role": "user", "content": "hello"}])
print(r.hedge["winner"], r.hedge["n_launched"])   # the SDK keeps unknown fields

Node (OpenAI SDK):

import OpenAI from "openai";
const client = new OpenAI({ baseURL: "http://127.0.0.1:8080/v1", apiKey: "unused" });
const r = await client.chat.completions.create({ model: "<model.name>",
  messages: [{ role: "user", content: "hello" }] });
console.log(r.hedge.winner, r.hedge.n_launched);

LiteLLM, as a proxy (config.yaml) or from the SDK:

model_list:
  - model_name: <the name your callers use>
    litellm_params:
      model: openai/<model.name>
      api_base: http://127.0.0.1:8080/v1
      api_key: unused

import litellm
r = litellm.completion(model="openai/<model.name>", api_base="http://127.0.0.1:8080/v1",
                       api_key="unused", messages=[{"role": "user", "content": "hello"}])

Tools that read the environment: OPENAI_BASE_URL=http://127.0.0.1:8080/v1 (the OpenAI SDKs and most OpenAI-compatible tools).

A streaming response carries the hedge block in its last chunk.

The app uses wrap() instead. The pip and npm packages also ship airmet.wrap(client, model=…) (Python) and wrap(client, {model}) (Node), which start a sidecar themselves when the app calls them. The sidecar logs to a file, and from release 0.11.0 wrap() relays its first-run telemetry notice to the app's stderr: once, and once more if what is sent changes. A sidecar that wrap() reuses prints nothing. The notice arrives only once the app is running, so take your human's telemetry choice first (section 3). If they opt out, pass it on: extra_args=["--no-telemetry"] (Python) or extraArgs: ["--no-telemetry"] (Node).

The app runs in a container or on another host, and section 7 started the sidecar with --host 0.0.0.0 --auth-token-env AIRMET_TOKEN: the app uses http://<this host>:8080/v1 and sends the token as its API key (Authorization: Bearer …).

9. Verify

Run every check. Replace <model.name> with the value in hedger.yaml, and use your port if it is not 8080.

Started with --auth-token-env? Then /v1 and /admin answer 401 without the token (/readyz and airmet health need none). Define this helper, which hands curl the token on its standard input so it never appears on a command line, and use acurl wherever a check below says curl:

acurl() { printf 'header = "Authorization: Bearer %s"\n' "$(printenv AIRMET_TOKEN)" | curl -K - "$@"; }

airmet status cannot send the token in 0.11.2, so it answers error: …/admin/replicas -> 401. In check 5 use acurl -s http://127.0.0.1:8080/admin/replicas instead: it lists the configured, healthy and down legs.

1. Both legs admitted.

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
grep -c ' admitted L0' airmet.log          # 2
grep 'failed L0 admission' airmet.log      # nothing

With Docker the log is the container's: docker logs airmet 2>&1 | grep -c ' admitted L0'.

2. Ready.

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
curl -s http://127.0.0.1:8080/readyz
airmet health

{"ready":true,"healthy_replicas":2,"configured_replicas":2,…}
ready: true (HTTP 200) healthy_replicas=2 configured_replicas=2 …

3. One request answers, with its hedge block.

curl -s -D - http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<model.name>","max_tokens":16,"messages":[{"role":"user","content":"Say OK."}]}'

Expect HTTP/1.1 200 OK, an X-Hedge-Winner: leg0 (or leg1) header, an X-Hedge-Request-Id, and a body ending in a "hedge":{…} object with n_launched and winner. An X-Airmet-Bypass: <reason> header means the request was not raced, and says why.

On a reasoning model the 16 tokens can all go to its reasoning. The answer then reads "content":"" with "finish_reason":"length", and the winning leg's entry in hedge.arms carries "quality":["empty"]. That is the token cap, not a failure: this check reads the status, the headers and the hedge block.

4. Requests race. At α = 0.3 about 30% of requests race, so send 20 (at a lower α send more: 50 at α = 0.1):

for i in $(seq 1 20); do
  curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
    -d "{\"model\":\"<model.name>\",\"max_tokens\":16,\"messages\":[{\"role\":\"user\",\"content\":\"Say OK ($i).\"}]}" \
    | grep -o '"n_launched":[0-9]'
done | sort | uniq -c

     14 "n_launched":1
      6 "n_launched":2

The counts vary. At least one "n_launched":2 is expected: none in 20 happens about once in 1,250 runs at α = 0.3, so run it again before you conclude anything. A raced request's block reads like this (trimmed):

"hedge": {"n": 2, "n_launched": 2, "policy": "immediate", "winner": "leg0",
  "arms": [{"replica": "leg0", "outcome": "won", "ttft_ms": 18.7, "e2e_ms": 92.0},
           {"replica": "06GMP9GG3H78CXE9F2PCNFDZK0-leg1", "outcome": "completed_loser"}]}

policy reads immediate under twinleg: that is how twinleg launches a raced request. The losing leg keeps running, under the name <request id>-<leg>, and its timings land in the local records (airmet-results/up-*/raw/requests-0.jsonl) once it finishes.

A request that the second leg took over after the first leg failed also reads "n_launched":2, but its block carries a rescue object (check 8). With both legs healthy you should see none.

5. Status. Run it from $HOME/airmet, where the local records are.

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
airmet status

airmet status · http://127.0.0.1:8080
ready:       yes (2/2 replicas healthy)
replicas:
  • leg0       healthy  …
  • leg1       healthy  …
telemetry:   ON → pushed … (or push OFF / OFF, as your human chose)
…
your wins (vs unhedged baseline, from local records):

6. Telemetry is what your human chose.

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
airmet traffic | grep '^consent'

consent: ON (saved choice; …) if they kept it on, consent: OFF (saved choice; nothing is ever sent) if they opted out. airmet traffic prints the full inventory, and airmet traffic --verify the steps to confirm it in a packet capture.

7. A dead leg does not take the app down. Run a second, throwaway sidecar whose second leg points at a closed port, with its own config directory and telemetry off, so your human's saved choice is untouched:

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
cp hedger.yaml hedger.deadleg.yaml
# edit hedger.deadleg.yaml: in leg1, set base_url: http://127.0.0.1:9/v1 (nothing
# listens there) and delete its api_key_env line; in a proxy: block, delete the
# listen and auth_token_env lines and keep on_unhealthy, your human's choice
CHECK="$(mktemp -d)"
AIRMET_CONFIG_DIR="$CHECK/config" XDG_STATE_HOME="$CHECK/state" \
  nohup airmet up --config hedger.deadleg.yaml --host 127.0.0.1 --port 8089 \
  --no-telemetry --results-dir "$CHECK/results" > "$CHECK/airmet.log" 2>&1 &
echo $! > "$CHECK/pid"
for i in $(seq 1 60); do grep -q 'serving /v1' "$CHECK/airmet.log" && break; kill -0 "$(cat "$CHECK/pid")" 2>/dev/null || break; sleep 1; done
grep -E 'admitted|failed L0 admission' "$CHECK/airmet.log"
for i in $(seq 1 10); do
  curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8089/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"<model.name>\",\"max_tokens\":16,\"messages\":[{\"role\":\"user\",\"content\":\"Say OK ($i).\"}]}"
done | sort | uniq -c
airmet status --admin http://127.0.0.1:8089 | head -6
kill "$(cat "$CHECK/pid")"

… replica leg0 admitted L0, …
… replica leg1 failed L0 admission (excluded from the pool): health (/models) failed
     10 200
ready:       yes (1/2 replicas healthy)
    leg1       DOWN     …

Every request is answered from the healthy leg, and the ones that would have raced carry "passthrough":"degraded_pool" in their hedge block: serving from the legs that are up is the default (proxy.on_unhealthy: passthrough). If your human chose on_unhealthy: error, expect 503 (no_agreeing_pool) instead. The same happens while running: a leg that fails two liveness probes is demoted (about 20 seconds) and comes back on its own once it answers again. Until a dead first leg is demoted, the requests that run on it alone are sent to the second leg (check 8), except those relayed unraced. A leg whose /models still answers, as in a 429 storm, stays in the pool, and its requests are sent to the second leg the same way. A leg whose key is refused twice in a row is taken out (section 6). With Docker, run the check as a second container on -p 127.0.0.1:8089:8080 with hedger.deadleg.yaml mounted, --no-telemetry, and only leg0's key, and without the airmet-data volume: on it, that --no-telemetry would be saved as your human's choice.

8. A first leg that fails is retried on the second. This check needs no keys and sends nothing to your human's providers. It starts a sidecar of its own on two throwaway stand-in servers made with python3 (without python3, skip it and say so), on ports 8090 to 8092, with telemetry off and its own config directory. It kills the first stand-in and sends 12 requests. With Docker, skip it too unless a bare binary is installed as well: a container does not reach the stand-ins on the host's 127.0.0.1.

Run the block as one command: CHECK does not carry over to a fresh shell. Like section 7, it starts every process with nohup, so none of them dies with its shell if your tool has to run the block in pieces. Its last line stops the sidecar and the stand-in that is still running.

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
CHECK="$(mktemp -d)"
cat > "$CHECK/leg.py" <<'EOF'
import json, sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class Leg(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    def log_message(self, *args): pass
    def reply(self, body, ctype):
        self.send_response(200)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def do_GET(self):
        self.reply(b'{"object":"list","data":[{"id":"stub","object":"model"}]}', "application/json")
    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length") or 0))
        def chunk(delta, fin):
            return "data: " + json.dumps({"id": "s", "object": "chat.completion.chunk", "model": "stub",
                "choices": [{"index": 0, "delta": delta, "finish_reason": fin}]}) + "\n\n"
        sse = chunk({"role": "assistant", "content": "OK"}, None) + chunk({}, "stop") + "data: [DONE]\n\n"
        self.reply(sse.encode(), "text/event-stream")

ThreadingHTTPServer(("127.0.0.1", int(sys.argv[1])), Leg).serve_forever()
EOF
cat > "$CHECK/hedger.yaml" <<'EOF'
replicas:
  - {id: leg0, engine: openai, base_url: "http://127.0.0.1:8091/v1", model: stub}
  - {id: leg1, engine: openai, base_url: "http://127.0.0.1:8092/v1", model: stub}
model: {name: stub}
hedge: {policy: twinleg, alpha: 0.3}
EOF
nohup python3 "$CHECK/leg.py" 8091 > "$CHECK/leg0.log" 2>&1 & echo $! > "$CHECK/leg0.pid"
nohup python3 "$CHECK/leg.py" 8092 > "$CHECK/leg1.log" 2>&1 & echo $! > "$CHECK/leg1.pid"
for i in $(seq 1 20); do
  curl -s -o /dev/null http://127.0.0.1:8091/v1/models && curl -s -o /dev/null http://127.0.0.1:8092/v1/models && break; sleep 1
done
AIRMET_CONFIG_DIR="$CHECK/config" XDG_STATE_HOME="$CHECK/state" \
  nohup airmet up --config "$CHECK/hedger.yaml" --host 127.0.0.1 --port 8090 \
  --no-telemetry --results-dir "$CHECK/results" > "$CHECK/airmet.log" 2>&1 &
echo $! > "$CHECK/pid"
for i in $(seq 1 60); do grep -q 'serving /v1' "$CHECK/airmet.log" && break; kill -0 "$(cat "$CHECK/pid")" 2>/dev/null || break; sleep 1; done
grep -E ' admitted L0|failed L0 admission|rescue_solo ON' "$CHECK/airmet.log" | cut -c1-110
kill "$(cat "$CHECK/leg0.pid")"      # the first leg dies
for i in $(seq 1 12); do
  curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:8090/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"stub","max_tokens":16,"messages":[{"role":"user","content":"Say OK."}]}' |
    grep -oE '"kind":"[a-z_]+"|^HTTP [0-9]+' | paste -sd' ' -
done | sort | uniq -c
kill "$(cat "$CHECK/pid")" "$(cat "$CHECK/leg1.pid")"

… replica leg0 admitted L0, …
… replica leg1 admitted L0, …
… rescue_solo ON: a solo turn (alpha gate closed, or the lagging-arm guard) whose leg fails before its …
     8 "kind":"solo_failed" HTTP 200
     4 HTTP 200

Every request answers 200, although the first leg is gone. The ones that ran on the first leg alone (about 70%) carry this in their hedge block:

"rescue":{"kind":"solo_failed","from":"leg0","to":"leg1"}

The rest raced, and the second leg won. The counts vary. Send the 12 requests right after the kill: the first leg is taken out of the pool within about 20 seconds, and the requests after that are not retries any more. No solo_failed in 12 would mean the retry is not running: read the rescue_solo line of the log, and check that the config has policy: twinleg and no hedge.rescue_solo: false.

9. The app goes through it. Note the count of local records, run one request from the app itself, and check that the count grew by one:

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
wc -l "$(ls -td airmet-results/up-* | head -1)/raw/requests-0.jsonl"

With Docker the records are in the airmet-data volume: docker exec airmet /airmet status | grep 'source:' reads source: <N> request(s) from /data/airmet-results/…. With Docker and a token, status cannot authenticate (above): check instead that the app's response carries the hedge field or an X-Hedge-Request-Id header.

10. Optional: link this machine (airmet login)

Only if your human asked for it (section 3, question 10):

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
airmet login

It prints a link and a short code, then waits. Give both to your human; they open the link in any browser, sign in and confirm the code. The command ends with linked to <org> — token saved in …/config.json. If your shell cannot leave a command waiting, ask your human to run airmet login in their own terminal. Then restart airmet up (section 11) and check:

cd "$HOME/airmet" && export PATH="$HOME/airmet/bin:$PATH"
airmet login --status                      # linked: <org> — ingest token saved in …

Linking turns on the live route feed, the fleet-tuned routing data that a running install refreshes. A linked install sends its token and install id whenever it registers or polls the feed, and the public catalog routes its legs use when it polls the route feed, whatever the telemetry choice. airmet login --clear unlinks it.

11. Stop, restart, upgrade, uninstall

Stop. kill "$(cat "$HOME/airmet/airmet.pid")". The log ends with airmet-hedgerd stopped cleanly. Docker: docker stop airmet.

Restart. Start it again exactly as in section 7. The saved telemetry choice applies; AIRMET_TELEMETRY_SCHEMA=v1 has to be given again. Docker: docker start airmet restarts the same container, with its saved choice and install id (docker run --name airmet again would fail: the stopped container still holds the name).

Upgrade. Fetch this page again (https://airmet.ai/install-for-agents.md): it names the current release. Repeat section 5 with it, stop the old process, start the new one as in section 7 and run section 9 again. hedger.yaml carries over. Docker: remove the old container first (docker rm -f airmet), then run section 7's docker run with the new image tag. The new container keeps the telemetry choice and the install id from the airmet-data volume. A container started without that volume (the 0.11.0 command) kept them inside itself, and they go with it: pass your human's choice once more on the first docker run with the volume.

Uninstall. Ask first, then:

1. Stop it, and put the app's base URL back the way it was.
2. Remove what you installed: rm -rf "$HOME/airmet/bin" (every channel but Docker), plus rm -rf "$HOME/airmet/.venv" "$HOME/airmet/wheels" (pip), rm -rf "$HOME/airmet/node_modules" "$HOME/airmet/package.json" "$HOME/airmet/package-lock.json" (npm), docker rm -f airmet && docker rmi airmet:v0.11.2 (Docker).
3. Only if your human wants it gone too, the local state: ~/.config/airmet/ holds the telemetry choice, the install id and any login token ($XDG_CONFIG_HOME/airmet/ when that is set); ~/.local/state/airmet/ holds caches ($XDG_STATE_HOME/airmet/); $HOME/airmet/airmet-results/ holds the local request records. Docker keeps all three in its volume: docker volume rm airmet-data. Run airmet login --clear before you delete a linked install's config.

12. Troubleshooting

Test one leg without putting its key on a command line (printf is a shell built-in, so the key reaches curl on its standard input):

printf 'header = "Authorization: Bearer %s"\n' "$(printenv PROVIDER_A_API_KEY)" |
  curl -sK - -o /dev/null -w '%{http_code}\n' "<leg0 base_url>/models"

200 is a working key and URL, 401 or 403 a key problem, 404 a wrong base URL, 000 no connection.

Done when

Report this checklist to your human, one line each, with the evidence:

• Installed from https://airmet.ai/dl/v0.11.2/: the channel, airmet version output, and the SHA256SUMS check that printed OK.
• hedger.yaml in $HOME/airmet: the setup (a, b or c), leg ids leg0 and leg1, no key in the file, policy: twinleg, the α your human chose.
• Telemetry as your human chose: the start flag used, and the airmet traffic consent line.
• One admitted line per leg, and airmet status reads 2/2 replicas healthy.
• A raced request seen: the "n_launched":2 count out of 20.
• The dead-leg check answered 200 from the healthy leg (or 503, if your human chose error), and the check's sidecar is stopped.
• The retry check (check 8): the solo_failed count out of 12 and the rescue_solo ON log line, or why it was skipped.
• The app's base URL changed (the diff), and one request from the app itself appeared in the local records.
• How it keeps running (whose terminal, service or container), and whether this machine is linked (airmet login --status).
• Anything that failed twice, with its exact error line, keys redacted.