The Gate Failed Both Runs

Last time I wrote about Backline, the story was that its eval suite spent a week catching my bugs instead of the model's. This is the sequel, and it starts with an uncomfortable question about every number in that post:

All of them were measured in my house.

My dual-4090 rig, my Docker Compose, my residential fiber. The suite says the platform scores 94.8. Does any of that survive contact with infrastructure I don't own? Or is "94.8" secretly "94.8 on Sergio's machine" — a number with an asterisk nobody can see?

There's only one honest way to find out, and it isn't redeploying and eyeballing the dashboard. It's an experiment. So I ran one — and because I'd spent a week learning that my own harness lies to me more often than the models do, I wrote the hypothesis down before collecting any data:

Pre-registered, before either run started: re-run the full 133-question suite on AWS, and the overall score lands within 3.0 points of a same-day local control run — the same-model noise bound already documented in the repo's benchmark notes.

Also written down in advance: what would falsify it. An overall gap bigger than 3. A systematic one-directional degradation. Or my named prime suspect — the cross-encoder reranker choking on Fargate's CPU and timing out, showing up as correlated failures across every retrieval-heavy category.

Spoiler on the suspect: not only innocent, but faster.

Hold the database constant

The design problem with "does it work on AWS" is that a naive redeploy changes ten variables at once — new database, re-generated data, re-computed embeddings, different index, different everything. If the score moves, you learn nothing, because anything could have moved it.

So the migration is the experimental control. I pg_dumped my exact local world — 468,160 statement lines, the real bge embeddings, the ivfflat index, and the answer key itself — and restored it byte-for-byte into RDS. Seventeen tables verified by count, the embedding model tag verified by query, the index verified present. Same database state, same suite hash, same judge rubric, same day, identical flags, both runs at concurrency 4.

Hold the database constant and exactly one variable remains: the environment. My rig versus ECS Fargate talking to RDS over forced TLS. That's the whole experiment.

One confession that belongs in this section: I wrote my first line of Terraform the morning of the deploy. What made a one-day AWS deployment possible for someone who'd never touched it was doing the archaeology first — the deploy plan opens with a table of fourteen facts verified by literally executing the repo on a cold sandbox before any AWS resource existed. That's how I knew, before spending a dollar, that the eval module wasn't in the Docker image, that the model weights weren't baked, that the data-seeding shortcut would silently skip file generation against a restored database, and that RDS forces SSL on Postgres 15+ so every connection string needs sslmode=require or nothing connects. Each of those is an hour of confused debugging that got converted into one line of a table.

The stack, briefly

ECS Fargate for the API and UI, RDS Postgres 16 with pgvector, a two-listener ALB (the API bakes its address into the Next.js bundle at build time, so it needs a DNS name that exists before the UI image does), everything in Terraform, ingress locked to my home IP and nothing else — a live API key behind a public URL is fine when the URL only answers to one house.

Two details I'm disproportionately fond of. The task role — the IAM identity the application code can reach — has zero policies attached. Backline's agents talk to Postgres and to Anthropic; they call no AWS API, so there is no policy to audit because there is no policy. Least privilege as an empty list.

And the eval itself doesn't run as a service. It's a one-off Fargate task: same image, command override, --gate at the end. Which means the container's exit code is the gate's verdict. The task that ran my suite on AWS shows STOPPED, exit code 1, in the ECS console — and that exit code is the finding, surfaced exactly where an operator would look for it.

The baked model weights turned a cold start into a 42-second image pull. The suite ran 133 questions plus judging in 11 minutes 48 seconds.

The result

Δ 0.8, against a pre-registered bound of 3.0. Both runs completed everything with zero infrastructure errors — no heal pass, no outage wearing a model costume. And the reranker-timeout suspect I'd named in advance? AWS median latency came in 170 milliseconds faster than my rig. A 2-vCPU Fargate task out-p50'd two RTX 4090s, because the latency here was never about local compute — it's API round trips, and Oregon sits closer to the API than my living room does.

The claim this supports, stated exactly as narrowly as the evidence allows: the platform's measured behavior is environment-invariant. Not "it works on AWS" — the instrument reads the same in both rooms.

That could have been the whole post. Then the gate spoke up.

The gate failed the AWS run

The strict regression gate compares a run against the committed baseline and fails any category that drops more than 3 points. It failed the AWS run, loudly:

gate: FAIL
  ✗ contract_terms: 77.0 vs baseline 85.0 (-8.0 pts > 3)
  ✗ reconciliation: 86.7 vs baseline 96.7 (-10.0 pts > 3)
  ✗ 7 T2 violation(s) — process assertions failed
  · adversarial: improved 93.3 → 100.0

The move everyone makes here — the move I could feel my hand reaching for — is the quiet re-roll. Run it again, get a friendlier draw, publish that one. Eight dollars and forty minutes buys a clean table and nobody ever knows.

Instead I ran the gate on the local control, which I hadn't bothered to gate because it was the control:

gate: FAIL
  ✗ abstention: 90.0 vs baseline 100.0 (-10.0 pts > 3)
  ✗ multi_step: 65.0 vs baseline 72.8 (-7.8 pts > 3)
  ✗ 2 T2 violation(s) — process assertions failed

Both fresh runs fail the strict gate. On completely disjoint categories.

Sit with that, because it's the cleanest evidence in the whole experiment. A broken environment degrades systematically — the same categories, the same direction, a traceable mechanical cause. What actually happened is scatter in both directions: AWS lost contract_terms and reconciliation, and gained ten points of abstention and seven of multi_step — the exact categories the local run dropped. Noise scatters. Breakage accumulates. Two runs failing one gate in different places isn't two failures; it's one measurement of the noise floor, taken twice.

The per-category arithmetic was already documented in the repo before this experiment, and it predicts exactly this. Reconciliation is F1-scored over anomaly flags, so a couple of misses swing the category by double digits — and the AWS run's 86.7 actually beats the committed benchmark row's 83.3 on that category; it's my local control's 98.3 that was the lucky roll. Abstention is ten questions, so one question is ±10 points. Adversarial is three questions; both runs scored 100, above baseline. When your category has three questions in it, the gate isn't measuring regressions. It's measuring dice.

So both results are published exactly as measured, gate failures and all. Nothing was re-rolled until it passed. The last post ended with "rulers don't get edited after the measurement" — this is the sequel's corollary: readings don't get re-taken until they flatter.

What broke, because something always does

The honest ledger, in the order it happened:

AWS rejects apostrophes in security-group descriptions. My first terraform plan died because three descriptions said "the operator's home IP" — the API enforces a restricted character set, and prose that reads fine in a code comment gets bounced at the boundary. The subtle part: Terraform stops at the first offending resource, so only two of the five violations surfaced; the rest were queued invisibly behind them. Fixing what the error showed would have meant three failed plans in a row. The actual fix was a one-shot auditor over every description in the tree.

The 503 window that isn't a bug. Right after the first deploy, the API returned 503 for a few minutes — because the load balancer was health-checking a container still pulling a four-gigabyte image. Without a health-check grace period, that sequence is genuinely destructive: the ALB marks the still-downloading task unhealthy, ECS kills it, the replacement restarts the same pull, forever — a crash loop that presents as an application bug and is purely a timing artifact. With the grace period, it's just a few minutes of patience that looks exactly like failure.

terraform destroy ate my evidence. The plan stored the day's artifacts — the migration dump, the eval summaries, the screenshots — in an S3 "evidence bucket" for durability. The evidence bucket was defined inside the Terraform module. The Terraform module's lifecycle ends with destroy. You can see where this is going: the command the plan concludes with deleted the bucket holding the proof, dump and all, and two sentences in my writeup claiming S3 archival became retroactively false and had to be corrected. The repo was always the real archive, so nothing irreplaceable died — but "keep your evidence outside the blast radius of your teardown" is now written down where I'll find it next time.

The teardown proof was weaker than designed. The plan's final flourish was a tag-based scan proving zero resources survived. The scan came back non-empty — because AWS's tagging index retains tombstones of deleted ECS services and deregistered task definitions for a while. The authoritative empty-check turned out to be asking each service directly: no clusters, no databases, no load balancers, no VPCs. All four empty. The tombstones bill nothing and mean nothing, but the lesson stands: the tag index is a cache, not a census.

The receipt

Thirty-eight resources, applied in one pass, destroyed in one pass, alive for about three hours. The infrastructure bill for the entire experiment: roughly forty-five cents. The API spend for the two eval runs: $15.89 — the measurement cost thirty-five times more than the servers it ran on, which feels like the correct ratio for a project whose whole thesis is that the measurement is the product.

Nothing is running now, on purpose. The artifact was never a service — it's the Terraform tree a reviewer can read in four minutes, the parity table, and this ledger of what broke. All of it is in the repo, where the doc-pinning tests can reach it.

What I actually believe now

Parity claims need pre-registration. If I'd deployed first and picked the comparison afterward, I'd have had a dozen numbers to choose from and every incentive to choose kindly. Writing the bound down first — and writing down what failure would look like — is what makes Δ 0.8 a result instead of a selection.

A gate failure can be the publishable result. The strict gate did exactly its job: it flagged that fresh runs wander outside a 3-point category tolerance. What it flagged is real — it's just variance, not regression, and the proof is that it flagged both sides of the experiment in opposite places. Suppressing that would have made the writeup cleaner and the evidence weaker. The failure is load-bearing.

Environment-invariance is earned by the scaffolding, not the model. The scores traveled because everything around the model is deterministic — the world is seeded, the arithmetic is one shared library, the retrieval is structurally scoped, the database moved as a byte-identical unit. The model was the same model in both rooms; the platform is what made the rooms interchangeable.

And the sequel to "interrogate your zeros" is: interrogate your reds. Last time, every dramatic zero was my harness lying about the model. This time, every red cell in the gate output was the gate telling the truth about noise. Same discipline, opposite direction — chase the mechanism before you accept the verdict, even when the verdict is against your own deployment, and especially when re-rolling it would only cost eight dollars.

The full parity table, the Terraform, and the what-broke log live in the repo: deploy/aws/README.md. The plan that governed the day — the verified-facts table, the phase gates, a human on every apply — is AWS_DEPLOY_PLAN.md. The first post in this pair is Interrogate Your Zeros.