We checked our own checker, and it could only see half the code

The last release built a tool whose job is to find any page that describes itself as showing everything when it is really showing a recent slice. That tool worked, and it shipped with an admission attached: it only knew how to look inside one kind of code, and nobody had measured how much of VEILOS that left out. This release measured it. There are two thousand four hundred and twenty-nine places in the code where work gets done, and the tool could see one thousand two hundred and eighty-seven of them — a little over half. The rest were invisible to it, including the piece that handles every single request that arrives at the site. It turned out to be blind in two separate ways rather than one, and the second is the part worth telling you about. The first was ordinary: some code is written in a style the tool did not recognise, and one of those pieces reads exactly the kind of forgetful list the tool exists to police. The second was not ordinary. One page was written in the style the tool did recognise — it looked straight at it — and skipped it anyway, because that page reaches its data by a slightly different route than the tool expected. A tool can be blind to something it is looking directly at. That matters, because the easy fix would have been to teach it a few more styles of code, and the easy fix would have left the second problem completely untouched while looking for all the world like it had worked. So the way the tool identifies code was rebuilt rather than merely widened, it now lives in one shared place so the next check we write inherits it instead of reinventing it, and it recognises data being read by any route rather than one particular spelling. Here is the honest result: nothing on any page turned out to be false. All three pages the improved tool newly found were checked by hand, and every one of them was already telling the truth — one publishes no such count at all, one's wording is about a form you fill in, and one is counting something entirely different from the list it reads. What was broken was the reach of the tool, not anything you were shown, and we would rather say that plainly than let three new names imply an alarm we did not find. We have also published what the tool still cannot do — a count assembled in one place and described in another remains outside what it can check, and there are sixty-eight places where that could in principle happen — as a number rather than leaving it unsaid, because a silence reads like an all-clear. Finally, the new check passed on its first run, which is exactly the situation the last release warned us to distrust. So we deliberately put it back into the broken state and confirmed it refuses to release, twice, before believing it.