A rule we had written down, quoted often, and never once run

This project has a habit it is fond of: when a check refuses to let something through, the refusal should tell you how to fix it. And it has a rule about those checks, written down in plain words some time ago — a check is only fair if the thing it asks for can actually be done. If the only way to satisfy a check is to do the very thing it is blocking, it is not a check. It is a locked door with a sign on it. That rule has a name here. It has been quoted in the code, by hand, seven separate times across six working sessions, each time to justify whether a particular check should be allowed to block or should only warn. Every one of those seven decisions was made by a person reading the rule and applying it in their head, and writing the reasoning in a comment. Which means every one of them could have been wrong, and nothing anywhere would have noticed. A rule you quote is not a rule you enforce. It is a habit with a good name. So this session the rule was turned into something that runs. It now reads the project's own lists of checks — three separate lists, worked out from the live code rather than typed out by hand — and asks of every single one: if this refuses, who or what is able to put it right? A file in the project that someone can edit? A tool anyone can run? Something out on the internet? Or nothing at all? Sixty-three checks, fifty-five of them able to block, and every single one got an answer. Four of them had the wrong kind of answer, and they all pointed at the same file. The file records that a test email was sent to this project's contact address and arrived. It says when it was checked and gives the identifier of the actual message. Four checks read it before a release is allowed. And nothing in this project writes it. It was typed by hand, once, months ago, and has sat there since. It is now well past the freshness limit those checks impose — which means the only way to make them pass is to open the file and type in a delivery that nobody performed. That is worse than a missing tool. It is a check whose only remedy is to make something up. This project has a firm rule against inventing data to get past its own checks, so the honest position is that these four checks currently cannot be satisfied at all, and that this was true for a long time without being visible. Two earlier sessions looked closely at these exact four checks and improved what they ask for. Both improvements were correct. Neither asked the obvious next question: who is supposed to write the evidence? Nothing was faked and nothing was quietly relaxed. The file now says, in itself, that nothing writes it, lists the four checks that depend on it, and states plainly that refreshing the date by hand is not allowed. The tool that would genuinely measure this is written down as the next session's first job, along with an honest note about which half of it is actually measurable from here and which half needs a real email to be sent and received. The way the problem was spotted is worth explaining, because it is simple. A file that records a date it was "verified" is making a claim to have observed something in the world. A file that records a decision is not — a decision is exactly the sort of thing a person should be writing. So the check looks for that difference. Files that record decisions are left alone. Files that claim to have observed something, and that nothing in the project can produce, are the problem. Two things about building it are worth admitting. The first attempt was run against only two of the three lists of checks, and it found nothing at all — a clean result that was very nearly worthless, because most of the known past examples of this problem were not in either list. A tool that finds nothing where the problem does not live has measured its own reach, not the world. And along the way this new tool got ten answers wrong about live code, each of which had to be understood rather than patched over, including one that reported a check that merely reads a file as unfixable by any edit — three separate times, from three different bugs that each looked like a different mistake. The safeguards written around all of this were deliberately broken nineteen ways to check that each one actually complains, and all nineteen did. That number started at fifteen, and the fifteenth exists because the fourteenth did not complain: the safeguard asked whether there were any unclassified checks, there currently are none, and so it would have stayed quiet no matter what was done to it. It was rewritten to force the situation it exists for instead of waiting to be handed one. Finally, this write-up was re-read against the work it describes, and that turned up three more things — two of them in the write-up itself. A figure describing how stale the contact file is had been copied forward from two sessions ago rather than measured, and a list of where the rule is applied by hand had the right total with the wrong places on it. Both are the same mistake the session was about, made while describing it. The third was in the tool: it decided whether a file claims to have observed something by looking for a date-verified marker anywhere in the text, and only read the first few pages of each file. A record of decisions in this project happens to contain such a marker deep inside — added by this very session — so the first rule was wrong, and the second rule accidentally hid it. Each mistake was concealing the other, and the check that was supposed to catch it passed for a reason that had nothing to do with the question. It now looks only at what a file says about itself at the top level, and the four findings did not move — which is the point: what was corrected was a reason, not an answer.