Six Roads to a False PASS (All One Root Cause)

TL;DR: Signed, subject-bound evidence can still prove the wrong action; bind each claim to the exact command before execution. The parent slice log asks what the credential was actually about.

You have a credential check. It is signed, valid, and attached to the right commit. Before you trust that green light, what was the credential actually about?

In this note

• A valid credential can prove the wrong thing
• The binding held. The boundary around it did not.
• Shapes to hunt for before your next PASS
• Freeze bypasses instead of filing them away
• Make your next credential earn its label

This is where a lot of pipelines quietly fail. They validate the badge and never validate the job the badge claims to represent.

I hit that in Ranex with a record for true. The record could be correctly signed, bound to the subject tree, and accepted as proof that tests had executed. Nothing in that sentence should make you comfortable.

A valid credential can prove the wrong thing

Validating a credential is not validating what the credential is about. A signature can prove who recorded a command and which subject it named; it cannot turn true into a test suite.

Before SLICE-003, evidence satisfaction came down to a claim name, a subject digest, and exit code zero. The evaluator never looked at what ran. A worker could produce a signed, subject-bound record under the claim tests-executed while running true.

That is a clean record of an empty action.

Your equivalent might not be true. It might be a smoke check treated as a release test. A scanner that reports success when it scanned no files. A deployment receipt that names a revision but never proves the artifact reached production. The shape is the same: the system checks that somebody presented a credential, then assumes the credential carries the meaning it wanted.

Do not let a label do the work of evidence.

Ranex changed the claim shape in the committed gate catalog. A claim declares its required argv as a list, not a shell string. The kernel computes a digest of canonical argv bytes and compares it with the signed command digest in evidence. The readable command remains signed too, so a matching digest cannot be paired with a swapped human-readable field.

The result is narrow and useful: a signed record of true no longer satisfies tests-executed. The catalog declares the command that can satisfy the claim.

That is a kernel decision, not a convention in a build script. The check does not need to trust the worker summary.

The binding held. The boundary around it did not.

Six independent audits failed to break the claim-command binding itself. The same audits found six ways to get a false PASS around it because observation still happened inside state the observed party controlled.

This is the uncomfortable part. A strong control can be real and still be surrounded by weak seams. The right response is to name precisely what it does, then follow the attacker to the next boundary.

The slice also re-proved criteria one through nine by mutation. Each safeguard was deleted from src/ in turn, and its covering test had to go red. That work established that the binding was enforced where it claimed to be enforced.

Then the audit moved outward.

One bypass inherited an environment that retargeted the bound binary. A PYTHONPATH setting and a sitecustomize.py outside the repository could make the bound command exit zero without running. The argv, digest, absolute interpreter, tree, and signature remained truthful.

Another let a file Git was told to exclude decide the outcome. Another used a clean filter that made a modified tracked file hash as if it were the committed blob. Others targeted the trust-root blob, the git executable selected from an editable path, or an untracked empty directory that Git cannot represent.

Six roads. One root cause: the thing being measured still controlled the place and tools used to measure it.

The slice did not pretend those paths were closed. They were reproduced, recorded, and assigned to the next slice. That is a better status than a clean report that has never tried to drive around the guardrail.

Shapes to hunt for before your next PASS

Look for the seam between a declared command and the process that actually runs. Attackers do not need to defeat a digest when they can influence the interpreter, environment, filesystem, or tool that feeds it.

• A claim name that has no pinned argv, test-plan digest, or other executable definition.
• A command comparison performed as a shell string rather than a structured argument vector.
• A signed digest that does not cover the human-readable command or executable path.
• Environment variables inherited by the measured command.
• Tools resolved from ambient PATH or from a directory the observed party can write.
• Ignored, excluded, or untracked files that can affect the measured result.
• Git configuration, filters, replacement refs, or other local state that can change what “committed” appears to mean.
• A passing test that removes a safeguard only in a mock instead of from the running path.

For each shape, write down the proposition you need to prove. “The record is signed” is not enough. “The catalog-bound argv ran against the subject in a measurement environment chosen outside the observed party” is a proposition you can test.

That sentence is longer because the boundary is larger. Good. Short security claims often hide missing nouns.

Freeze bypasses instead of filing them away

When you reproduce a bypass, freeze it as a strict expected-failure test with a green control beside it. That keeps the hole visible without pretending it is fixed.

SLICE-003 did exactly that for the false-PASS paths around the binding. Each one is a strict xfail. Each has a control that proves the test can pass when the vulnerable condition is absent. The expected failure will fail loudly on the day the path is actually closed.

This matters because a plain issue can go stale, lose its reproduction, or be mistaken for a theoretical concern. A strict expected failure runs in the suite. It carries the attack shape forward with the code.

There is a trap here: an expected failure without a green control can pass vacuously. The test might not run the relevant code at all. You have traded one invisible gap for another. Put the control beside the bypass. Make both explain themselves.

Also state the boundary plainly. SLICE-003 did not claim output digests, authentic approver identity, or key containment. A command matching the bound digest can still be trusted to have done what its name suggests. That limitation was recorded rather than decorated with an unchecked field.

Ranex is pre-release. The claim-command binding is built; the surrounding observation hardening was next-slice work in this record. “Designed, not built” is a useful sentence when a control stops at a real boundary.

Questions people actually ask

These questions help you bind a CI claim to the command that produced it.

How do you bind a CI claim to the command that produced it?

The committed catalog declares the argv that satisfies a claim, and the kernel compares a digest of that argv with the signed evidence.

Why does a signed record of true not prove tests executed?

A signature and subject digest establish who recorded an observation and which tree it names. They do not establish that the command was a test run.

What should happen to bypasses found around a control?

A team should reproduce bypasses, keep a green control beside each one, and freeze them as strict expected-failure tests until the next slice closes the root cause.

Make your next credential earn its label

Pick one important claim in your pipeline: tests executed, artifact scanned, release approved, migration applied. Find the credential that satisfies it. Now write the exact action that credential is supposed to represent.

Bind that action before the worker runs. Verify the binding after it returns. Then try to go around it through the environment, the executable path, the working tree, and every ignored input you can name.

The reproductions live in the Ranex repository, under docs/slices/done/. Take the shape, not the marketing.

Try it. Break it. Tell me what broke. If this gave you a useful test, star the repository and send an honest critique. A false PASS found before release is a gift.

Disclosure: this post was drafted with AI assistance. Every factual claim traces to the repository’s README or slice records, the same fact gate the product enforces on code. It ships only after Anthony’s own review.