Building Ranex in the Open: What Gating My Own Code Actually Caught

Your build is green. Do you know what that green light proved?

Not “did the tests pass” — did the thing that says PASS actually run anything, against the code you think it ran against? I couldn’t answer that about my own repository, so I pointed the verifier at itself. Ranex gates Ranex: the kernel evaluates this repository’s own test suite — 738 frozen test IDs, run provisioned, sealed and offline against the real current commit. It didn’t validate the design. It kept catching me. What it caught is below, so you can go looking for the same shapes in your own build.

Why dogfooding a verifier is different

Most dogfooding is a marketing exercise. “We use our own product!” Fine. But a verification tool pointed at itself has a sharper property: every time it catches something, it’s evidence the thing works — and every time it misses something, that’s a defect in the product itself, not just in the code.

Which makes the interesting output the misses, not the passes. I have 17 years in this industry. Four of those were at Pantheon, working platform-level problems on sites where downtime was measured in money. I’ve seen a lot of failures. None of that stopped the kernel from finding how many of them were mine.

The green light that proves nothing

Two states from this build are worth sitting with, because you can check for both of them yourself.

In one, 37 of 47 checks were green while secrets remained readable.

In another, all 47 were green while the gate had actually run none of them.

Read that second one again. A perfect score, from a gate that executed nothing. Now ask what your own gate would report if it were in that state.

Here’s the deal: the question was never “was the AI smart enough.” It was:

Does this green light actually prove the proposition we think it proves?

That problem exists with AI, humans, CI systems, security scanners, tests and auditors. It predates language models by decades — which means you had it before you had an agent. It’s the reason I think the durable idea here is meaningful evidence — not guardrails, not linting, not hallucination detection.

Three that made it into the record

Each of these is a closed slice in the repository, with the failure written down rather than smoothed over. Read them as three shapes to go hunting for in your own pipeline.

A flaky suite that two reviewers approved

A test suite running at roughly 12% instability was reported stable and approved by two independent reviewers. It was caught only because every gate is re-run against the worktree on disk rather than read from the session that produced it. The cause was a fixture swallowing the stderr that would have said so.

Neither reviewer was careless. The report simply wasn’t the artifact.

59 refusals that no test executed

A slice closed on a cleanup control that had never worked on any supported Python — covered by a test that monkeypatched out the very function it was named for. Measuring the general form of that found 59 refusals no test executed at all.

The safety net had holes I’d have sworn weren’t there.

A skipped test reading as a passed test

Exit-code satisfaction let a skipped or vanished test read as success. The measured failure destroyed 27 tests while the remainder stayed green — and the gate said fine.

Absence had to become a first-class outcome.

What you can take from this

A check nobody has tried to break is a check nobody knows works. Every one of those defects survived a review that felt thorough at the time. What found them was mechanical re-measurement — mutation testing, re-running gates against disk, treating absence as failure.

The correction that mattered wasn’t a better prompt or a smarter model. It was structural: read the artifact, not the report. That one costs you nothing to adopt — open the diff instead of the summary, re-run the check yourself instead of reading the log somebody handed you.

Which, if I’m honest, is the same lesson from my Pantheon years. The incident is rarely what the dashboard says it is.

Why I publish the failures

Two reasons.

First, a verification product that hides its own defects is arguing against itself. If I’ll paper over my failures, why would you believe my PASS?

Second, this is the genuinely useful content. Nobody needs another post explaining what CI is. The specific way a green light lied to me — that’s worth your ten minutes.

The kernel is MIT-licensed. Every slice above is in docs/slices/done/ with its own record, including the parts that make me look bad.

Questions people actually ask

What does ‘Ranex gates Ranex’ mean?

The kernel evaluates this repository’s own test suite — 738 frozen test IDs, run provisioned, sealed and offline against the real current commit. Every catch is evidence the thing works; every miss is a defect in the product.

Why publish your own failures?

Because a verification product that hides its own defects argues against itself. If I will paper over my failures, there is no reason to believe my PASS.

What was the worst thing you found?

All 47 checks green while the gate had actually run none of them. A perfect score from a gate that executed nothing.

So here’s your move: take the check you trust most and try to make it lie to you. Delete a test file and see whether the gate notices it’s gone. Point a required check at nothing and see whether it fails or shrugs. Whichever one refuses to go red is the one that was never protecting you.

Try it. Break it. Tell me what broke. If you find a hole in the code that decides pass or fail, that’s a contribution — and I’d rather hear it from you than find it in production.