Writing

Everyone ships semantic caching. Almost nobody measures how often it silently returns the wrong answer — because measuring that needs exact ground truth. I had some. At the industry-default threshold, one cache hit in five was a wrong answer served with confidence.

I deployed my agent platform to AWS and re-ran its 133-question eval suite as a pre-registered experiment. The scores landed 0.8 points apart. Then the regression gate failed both runs — local and cloud, on completely different categories — and that turned out to be the best result on the table.

I built a 133-question eval suite with an exact answer key and pointed it at my own agent platform. In its first 24 hours it caught four bugs — every one of them mine, none of them the model's. Then it settled which model is actually worth the money.

My agent wrote fourteen documents about its tasks and executed code zero times. It wasn't malfunctioning — it was responding to prices. Mechanism design lessons from an agent that optimized against me.