AI Infrastructure / Platform · 2026 · Active

Headroom

An LLM gateway and control plane — virtual keys, atomic budgets, semantic caching, provider failover, per-request cost — and the measurement almost nobody publishes: how often that cache silently returns the wrong answer

Role

Architect · Sole Engineer

Timeline

2026 · twelve phases, P0–P11 · 16 PRs · one operator, one coding agent, fresh session per phase

Stack

Python 3.12 · FastAPI · asyncio (true SSE passthrough, never buffering)PostgreSQL 16 + pgvector (the ledger, cache entries, embeddings)DynamoDB — budgets and token buckets as single atomic conditional writesNext.js 15 + TypeScript (the operator console, reading the gateway's own admin API)Anthropic API + an OpenAI-dialect chain served by vLLM on two local RTX 4090ssentence-transformers (in-container CPU embedding for the semantic cache)Terraform in two roots split by lifetime: data survives, compute is disposableAWS: ECS Fargate · RDS · DynamoDB · ECR · Lambda · ALB · CloudWatch · Secrets ManagerKubernetes: EKS via eksctl · a Helm chart · IRSA · an NLB with instance targetsTailscale — one egress pod, so a cluster in us-east-1 reaches two GPUs on a deskGitHub Actions: six keyless jobs with Postgres and DynamoDB service containersuv · ruff · mypy --strict · 1,449 testsPython 3.12FastAPIasyncioPostgreSQL 16pgvectorDynamoDBNext.js 15TypeScriptsentence-transformersvLLMAnthropic APIDocker ComposeGitHub ActionsTerraformAWS ECS FargateRDSAWS LambdaALBEKSHelmeksctlIRSANLBTailscaleSecrets ManagerCloudWatch

Responsibilities

Gateway architecture: the request pipeline, dialect adapters, SSE passthrough semantics, and the rule that a stream past its first token is never re-routed
Money: reservation-based budget enforcement on DynamoDB conditional writes, settlement against real usage, dated price-book pricing, and the stampede tests that hold it
The experiments: pre-registered hypotheses, a versioned paraphrase corpus with an operator spot-check gate, offline replay of the admission decision, and adjudication of every gate verdict against evidence
Infrastructure: both Terraform roots, the Helm chart, the eksctl config, and runbooks written for a stranger to repeat — then executed by hand, step by step
Verification: chaos smoke over the internet, load loops with negative controls, a three-run find-fix-verify on rolling-upgrade drops, and a live failover demo with a GPU killed mid-run
The paper trail: decision log, phase log, per-phase evidence directories, and doc-pinning tests that fail when the README stops following from its artifacts

Deliverables

The running gateway: two dialects, streaming passthrough, virtual keys, budgets, rate limits, exact + semantic caching, same-dialect failover, and a ledger that prices every request
The cache-safety experiment: 130 answer-keyed questions, 520 probes, the admission decision replayed across 0.70–0.99, and the committed curve showing no threshold separates right from wrong
The parity experiment: the full 133-question suite re-run through the gateway — 93.7 vs 93.3 direct against a bound fixed before the run — plus a two-meter cross-check $0.000855 apart with the residual identified
The operator console: live traffic by upstream, breaker state, spend by tenant, cache dispositions — all read from the gateway's own admin API
Two hand-executed deployments, both destroyed the same day with empty-check receipts: ECS Fargate with a nightly rollup Lambda, then a Helm chart on EKS reusing the same managed services
A 100+-entry decision log, a phase log recording every deviation, and a README whose numbers are recomputed from committed artifacts by CI

Problem

Every team putting an LLM in front of users eventually builds the same layer: keys, budgets, rate limits, a cache, a fallback provider, a bill nobody can attribute. Headroom is that layer built properly — and then pointed at its own most-recommended feature. Semantic caching is shipped everywhere and measured almost nowhere, because measuring it needs a large question set with exact ground truth. Backline is one, so Headroom borrows it: 130 answer-keyed questions seeded into the cache, 520 probes embedded once, and the admission decision replayed offline at every threshold from 0.70 to 0.99. At the industry-default 0.90, the cache answered 98 of 130 never-before-seen questions from a neighbouring entry — and 92 of those answers were provably wrong. On the same corpus at the same threshold it served 389 of 390 genuine paraphrases, 382 of them correctly. Both halves are the finding: a control surface that looks excellent on the traffic you tested it with, and poisons the traffic you did not, is not a safe control surface.

Approach

One pipeline, two dialects, real SSE streaming: authenticate, scope, route, rate limit, cache, budget gate, failover, passthrough, meter. Virtual keys are per-tenant and scopeable to model patterns. The budget gate is a reservation against a DynamoDB conditional write — reserve before the call, settle the real cost after — so concurrent requests cannot spend the same dollar twice. The token buckets are the same shape. Both are raced by a 64-request stampede on every pull request, beside two deliberately broken implementations that must fail the identical test, because atomicity is necessary and it is not sufficient. Failover is same-dialect and honest about it: a circuit breaker per upstream, hops recorded on the ledger row and surfaced as response headers, and a hard rule that a stream which has already emitted a token is never silently retried onto another provider. Cost attribution is exact and per-request, priced from a dated price book rather than a constant — which is how the gateway's meter and the provider's own, run over one $7.54 experiment, land $0.000855 apart with the entire residual traced to a single identified request. The semantic cache ships with its own safety curve instead of a recommended threshold, because the measurement says no safe threshold exists: the closest wrong answer scores 0.999539, the furthest correct one 0.889850. The bands overlap. One cosine number cannot separate them.

Ask the question the industry answers by vibes: how often does a semantic cache serve a confidently wrong answer — measured against an exact answer key, not argued
Build the control plane that makes the measurement possible: virtual keys, per-tenant budgets and rate limits, exact and semantic caching, provider failover, per-request cost attribution
Enforce money on single atomic conditional writes, and prove it by racing them — a 64-request stampede on every PR, beside deliberately broken twins that must fail the same test
Prove the extra hop is free twice over: accuracy through the gateway against a pre-registered noise bound, and passthrough overhead measured live on every runtime it ships to
Run the same code on Compose, ECS Fargate, and EKS — and let each runtime break differently, because the bug one runtime hides is the bug the next one finds
Hold the README to the code's standard: every claim recomputed from a committed artifact by a test on every pull request, so the front door cannot overclaim without turning the build red

Build

The cache that lies politely

Semantic caching is recommended everywhere and measured almost nowhere, because measuring it needs a large question set with exact ground truth. Headroom borrows Backline's: 130 answer-keyed questions seeded into the cache, 520 probes embedded once, and the admission decision replayed offline at every threshold from 0.70 to 0.99 — so the curve is a property of the data, not of a lucky run. At the industry-default 0.90, the cache answered 98 of 130 never-before-seen questions from a neighbouring entry, and 92 of those answers were provably wrong. On the same corpus at the same threshold it served 389 of 390 genuine paraphrases, 382 of them correctly. Both halves are the finding: excellent on the traffic you tested, poisonous on the traffic you did not. And no threshold fixes it. The closest wrong answer scores 0.999539 — two questions differing in one period token — while the furthest correct one scores 0.889850. The bands overlap, so the recommended safe threshold, defined by a rule fixed before the curve was drawn, does not exist anywhere in the sweep. The embedding is not broken; it is right. The failure is using a similarity score as an admission decision when the characters it correctly calls negligible are the entire answer — a property of every application that templates its prompts.

92 wrong of 98 hitsno safe threshold0.999539 vs 0.889850520 probes

Money, on one atomic write

A budget that is checked and then spent is a budget two concurrent requests can blow through together. Headroom's gate is a reservation against a single DynamoDB conditional write — reserve the estimate before the call, settle the true cost after — and the token buckets are the same shape. One write, or it does not count. The claim is raced rather than asserted. Every pull request fires a 64-request stampede at a tenant with room for a fraction of them, and runs the identical test against two deliberately broken implementations — a read-then-write and a non-conditional update — both of which must fail it. Atomicity is necessary; a test that only ever passes proves nothing about whether it is sufficient. Cost attribution is priced from a dated price book rather than a constant, so a rate change is a data change, never a code change. That discipline is what makes the cross-check meaningful: the gateway's own ledger and the provider's reported usage, summed independently over one $7.54 experimental run, land $0.000855 apart — and the entire residual is one identified pre-flight request, not noise.

reserve then settle64-request stampedetwo broken twinsdated price book

A GPU killed from orbit

Failover is same-dialect on purpose: a circuit breaker per upstream, every hop recorded on the ledger row and surfaced as a response header, and a hard rule that a stream which has already emitted a token is never silently retried onto another provider — a caller who has seen half an answer must not be handed the start of a different one. The demo is the proof. Two vLLM instances on two RTX 4090s on a desk in California, reached from a Kubernetes cluster in us-east-1 through a single Tailscale egress pod, with a load loop streaming real 27B inference through the gateway. Ninety seconds in, one GPU is killed outright — docker kill, no drain, no warning. The result: 92 requests, 92 ok, zero dropped, zero caller-visible 5xx. The breaker opens on the dead upstream, traffic lands on its sibling within one request, and when the instance is resurrected a probe re-admits it and traffic flows back. The console shows the whole arc as a colour change in the live view. Every pre-first-token fault was hidden from the caller — the claim the pre-registration made in words, measured here in requests.

92/92, 0 droppedbreaker open → probe → re-admit0 caller-visible 5xxus-east-1 → a desk

What the hop costs

A gateway is an extra hop, and a hop has to justify itself twice: it must not cost accuracy, and it must not cost time. Both bounds were fixed before either run. Accuracy: the full 133-question Backline suite, re-run through the gateway, scored 93.7 against 93.3 direct — a difference of +0.4 against a pre-registered noise bound of 3.0. That means the hop did not cost accuracy. It does not mean the hop helped; there is no paired control, and the write-up says so rather than letting the plus sign flatter. Time: passthrough overhead over 462 live requests came in at p50 0.0612 ms, p95 0.1176, p99 0.1644 — the gateway's own cost, isolated from the model's. The same column reads 0.0249 ms on a live request through ECS Fargate behind an ALB, and 0.0175 ms through EKS behind an NLB — live rows on live infrastructure, cited to their ledger entries. Against inference measured in seconds, the control plane is free at three significant figures — stated as its own measurement, not as a comparison to a system nobody benchmarked.

93.7 vs 93.3 directΔ +0.4, bound 3.0p50 0.0612 msmeasured on all three runtimes

Same code, three runtimes, three different bugs

The gateway ran on Compose, then on ECS Fargate with RDS, real DynamoDB, and a nightly rollup Lambda, then on EKS through a Helm chart that reuses those same managed services rather than rebuilding them as pods — because managed-services-from-Kubernetes is the realistic architecture, and Postgres-as-a-pod would demo something nobody does. Terraform is split into two roots by lifetime: a data root that survives — VPC, RDS, DynamoDB, ECR, secrets — and a compute root created and destroyed at will. After destroying forty-three compute resources, the data root's plan reads "No changes." That sentence is the design working, and it is committed as evidence. The third runtime earned its keep in the first hour. The Tailscale egress pod crash-looped seven times because dropping ALL capabilities removed CAP_MKNOD, which the container needs to create its own tunnel device on nodes that do not expose one. Then every real request failed with a region error despite the region being set — because Fargate injects a second region variable for free, Kubernetes injects nothing, and the AWS SDK's environment resolution reads only the name Fargate happened to provide. The code had never been region-correct on its own; one runtime had been quietly covering for it. Both fixes were found at first contact, committed with their story, and hardened with tests.

Compose → Fargate → EKStwo Terraform rootsFargate's free variable"No changes."

The rolling upgrade that dropped one request

"Zero dropped requests during a rolling upgrade" is usually asserted. Headroom measured it, and the first measurement said one. Run one, 8,331 streamed requests with a helm upgrade mid-flight: one dropped, at the instant a pod was replaced. Run two, with the pre-stop drain window tripled: two dropped — one per replaced pod. The sleep was irrelevant, and that was the finding. A pod is removed from the endpoint list and signalled at the same moment, so the sleep only covers new connections; established flows are pinned by connection tracking, and a client holding a keep-alive connection spends the entire drain window talking to the doomed pod, losing whatever it had written when the server closes it. The fix went in at the right layer — a drain sentinel, and a draining pod answering Connection: close so clients retire those connections themselves. Proving it needed a rig, and the rig's first version measured zero drops on the broken build: on loopback the server's closing packet always wins, because the race window is one round trip. At two milliseconds of emulated latency the arms finally separate — baseline 2, 1, 1 dropped; drained 0, 0, 0. Run three on the real cluster: 8,342 requests, zero dropped. All three runs are committed, because a zero only means something if the instrument could have said otherwise.

8331 → 1 dropped8326 → 2 dropped8342 → 0 droppedconntrack, not sleep

The repo that cannot overclaim

Every number in the README is recomputed from a committed artifact by a test on every pull request — the curve values from the experiment results, the parity verdict and overhead percentiles from the analysis file, the cost table from the billing capture. A claim that no longer follows from its evidence turns the build red. One test even tightens itself: while the billing screenshot was missing, the cost table was required to read "pending", and the day the capture landed the test demanded the number instead. A stranger's cold clone reaches a working keyless demo in one command — verified from a temporary directory with no .env, no virtualenv, and no image, then torn down. The operational epilogue is priced. Both cloud phases were applied by hand, measured, and destroyed the same day, ending in per-service empty checks — because tombstones lie, and the only proof of a clean teardown is asking every service directly. The entire deployment story — Fargate, RDS, a Lambda, an EKS cluster, two load balancers — cost $3.56 against a budget of $20–25 pre-registered before any of it existed. The undershoot is the window, not efficiency: the cluster lived fourteen hours instead of three days, and a test now fails if any document lets that read the other way.

1,449 testsH-000 … H-102one command, keyless$3.56 of cloud

Decisions & tradeoffs

01The semantic cache ships a safety curve instead of a recommended threshold, because the measurement says no safe threshold exists — the bands of right and wrong overlap at every value from 0.70 to 0.99
02A stream that has emitted its first token is never failed over: a caller who has seen half an answer must not silently receive the beginning of a different one
03Budgets are reserved before the call and settled after it, on single conditional writes — and the tests race two deliberately broken implementations that must fail, because a green test on a correct implementation proves nothing on its own
04Prices come from a dated price book, never a constant: a sabotage test moves a rate in config without touching the README, and the README fails anyway — proving the numbers are computed, not typed
05The paraphrase corpus was regenerated, not patched, when an operator spot-check caught two systematic generation failures the mechanical checks had passed — scope compression and negation inversion — and the corpus carries a rubric version so a bad batch can never silently mix with a good one
06Every experiment's hypothesis and noise bound were fixed in a pre-registration before the run, and the write-ups adjudicate the ones the data refuted rather than quietly dropping them
07The two-Terraform-root split is proven, not asserted: after destroying forty-three compute resources, the data root's plan reads "No changes."
08Cost allocation tags label spend only from activation forward — an activation lag is a deadline, not a delay, and 72.4% of the window's bill came back unattributable on resources that were all configured to carry the tag
09A test that hard-coded two dates went red the day the calendar advanced past them; the fix froze the clock rather than the calendar, and the audit that followed found one more bomb set to detonate on a price-schedule rollover — with a wrong number instead of an error