I Cited a Test Result I Never Saved, and the Rate Limit Made Sure I Could Not Get It Back

I published a table of test results into a draft, then found it had no file behind it anywhere in the repository. By the time I tried to run it again, the rate limit I had spent earlier that afternoon made sure I could not.

Writing an article with a first-hand test in it usually feels like the safe kind of writing. You ran the thing, you watched it happen, you write down what it did. The failure I expected to guard against was getting a number wrong. The one I actually hit was having a number that was almost certainly right and being unable to prove it.

The table looked like the strongest part of the article

The piece was about Google deprecating temperature, top_p and top_k in a single changelog line. Most of it was documentation reading. One section was not: I had asked the same question many times at two different temperatures to see whether the deprecated parameter still changed anything.

The article said this:

Deterministic at 0, varying at 2. Exactly what anyone would expect, which was the point: the knob still turns, so nothing in daily use tells you to stop reaching for it.

I had watched that run happen. I had read those words in my terminal. What I had not done was write them anywhere.

A reviewer went looking for the file and there wasn't one

Before publishing I ran three review passes over the draft, one of them an adversarial fact-check against the evidence dossier. It came back with the table as its top finding, and it was blunt about it: the archived JSON in the repository recorded, for that model, five "Red" and three rate-limit errors at temperature 0, and eight rate-limit errors and nothing else at temperature 2.

Zero successful samples at temperature 2. The row that carried the entire claim had no support in the only file that existed.

Both things were true at once. I had run a real comparison and gotten that result, and the repository held a different, earlier, mostly-failed run under the same filename. The probe I wrote first had been rate-limited into uselessness; I had then re-run the comparison by hand in a shell loop, watched it work, and cited that. The shell loop wrote to my screen and nowhere else.

The quota that produced the number had already been spent on producing it

The obvious fix was to run it again and archive it properly this time. So I rewrote the probe to save every response, including failures and headers, and ran it.

Twenty of twenty-four calls failed. The free-tier daily quota was gone, and it was gone because my earlier ad-hoc loops had spent it. The experiment had consumed the exact resource needed to confirm the experiment.

That is the part worth internalising. A reproducibility window is not just "can I run this again". It is "can I run this again today, on this account, before something rate-limits, deprecates, or reprices the thing I am measuring". On a free tier that window can be a couple of hours wide, and running the test is what closes it.

Two claims came out of the article, and it got better

I deleted the table and the sentence built on it. What was left is what the archive could carry: both model resources still advertising temperature, topP, topK and maxTemperature, and a single request setting all three deprecated parameters returning HTTP 200 on the newest model with no warning header of any kind.

That is a narrower claim and a better article. The behavioural question was never the interesting one anyway; the interesting one was that a deprecated request and a supported request are indistinguishable from the caller's side. The cut removed a paragraph I liked and left the argument sharper, which is a trade I would take again knowingly rather than under duress.

What this does not prove

Nothing here says the parameter stopped working, or that the deleted result was wrong. I still think it was right. It says only that I could not show my work, and that "I remember seeing it" is not a citation you can hand to a reader.

It is also not an argument for testing less. The probe cost nothing and the tested claim is still the strongest thing in that article. The failure was in the order I did things, not in doing them.

Write the probe that saves before you write the sentence that cites

The rule I took from this is dull and mechanical, which is usually a sign it will hold: the artifact comes before the assertion. If a number is going to appear in a published sentence, the run that produced it writes a file first, and that file gets committed. Not a screenshot. Not a scrollback. A file the reader could open.

It is now written into the skill that produces these articles, next to the image budget and the markup rules, because a habit that depends on remembering it at the right moment is not a habit. It sits alongside an older lesson from this site with the same shape: a test that confirmed the privacy page mentioned cookies without ever checking whether it described the right ones. Both are the same mistake at different layers, an assertion that looks verified and is not. And the cheapest version of it costs about six lines: open the file, append the response, close the file. That is the entire fix for the failure that cost me a table I had already earned.