We were reading a third of our own code from the wrong place, and it was hiding two false sentences

The last release measured how much of VEILOS its own checking tool could see, and published the answer: a little over half. That was the right question and the number was correct. Nobody asked the next one — whether the parts it could see, it was reading all the way through. It was not. When our code is written in the very common shape where a function takes some settings alongside its main input, the tool was reading the settings and stopping, then treating that short fragment as though it were the whole piece of work. Eight hundred and four of the two thousand four hundred and thirty-two places where work gets done — a third of them — were being read this way. Nothing complained, and that is the part worth understanding: every check we own was reading the same wrong fragment, so they all agreed with each other, and agreement between instruments that share a fault looks exactly like confirmation. Two smaller faults came from the same place. Some of our pages ship little programs written out as text for your browser to run; the tool was mistaking those written-out programs for real code and counting them. And in one file, a perfectly ordinary division — one number divided by another — was misread as the start of a pattern, which threw the tool off for the rest of that file. Just one file out of three hundred and nine, and complete for that file: the last two and a half thousand characters of it were invisible to every check we have. Hidden in there was a real place where the organism writes to its permanent record, missing from the count we publish of exactly those places. Fixing the reading made two sentences visible that we had been publishing publicly and that were not true. The first is on our statistics page. It said every row names its definition, source, period, denominator and privacy posture. Four of the fourteen rows named a denominator. Six of them never could — no denominator was ever supplied for those, so the sentence could not have become true no matter how much the site grew. It survived because the machine-readable version of the page quietly filled in the words 'aggregate count' wherever a denominator was missing, which made it look complete to anything reading it automatically. And the human version of that same page has been printing an honest dash in that column the whole time. The page you could see had been contradicting the sentence we published beside it for as long as both have existed. The counts were never wrong. The word 'every' was. The second is the feed you can subscribe to for the organism's predictions about itself. It described what it carries as every graded prediction the organism has made. It carries fifty. The organism has made six hundred and ninety-eight, still holds one hundred and sixty, and has let five hundred and thirty-eight age off the end of a list that only keeps the most recent. Two releases ago we fixed this exact fault on a different page and wrote down, in the code, that a fix tied to one place has a clock running on it. The clock ran two releases. This one was unreachable that whole time for the reason above: the sentence lived in a part of the code nothing could read. Both sentences are now worked out from the actual numbers rather than written down and hoped for. If the list forgets things, the sentence says so. If the feed is showing you a recent slice, it says which slice and out of how many. If there is nothing yet, it says that instead of claiming to show everything about nothing. The first attempt at the feed fix managed to say 'the fifty most recent' and 'complete' in the same sentence, and we caught it and made that impossible rather than just rewording it. We also fixed the way we file these findings. There was a label meaning roughly 'this wording is figurative, not a claim' — and it was quietly doing double duty for 'this is a real claim about something we have not checked'. The statistics-page sentence would have been filed under it and waved through. Claims like that now have to name what verifies them, or they are refused. One last thing, in the spirit of the rest. The new check we wrote to catch this class of fault reported that its own file was broken. It was not — the check was searching for a marker word, and its own file happened to quote that word while explaining itself. A test whose signal can turn up inside the thing it is testing has no way to tell a real problem from its own reflection. It went off on its author before anyone else, which is the only reason we noticed.