Measure Before Learning: Stop Calling One Run a Win

TL;DR: You cannot call a change better because you watched it work once; freeze the comparison and its rules before you look. The Ranex slice log records the same distinction between claims and evidence.

You change a prompt, a handbook, a workflow, or a model setting. One run looks better. The temptation is immediate: ship the winner and tell yourself you learned something.

You learned that one run happened. You did not learn that the change caused the result.

In this note

• One run is an observation
• Freeze the ruler first
• Inconclusive is a valid answer
• A checklist before you call it better
• Promotion is not self-approval
• Where Ranex stands

One run is an observation

An observation records what happened on one run. A grade answers a stricter question: did a candidate beat a control under a comparison you defined before the result arrived?

That difference is the center of ADR-016. It covers future evidence for revisions to handbooks, rules, processes, workflows, models, skills, and their harnesses. The ADR rejects treating every run as a grade because monitoring has no counterfactual and lets people change the treatment after they have seen the outcome. That’s painting the bullseye with extra steps — moving the treatment once you already know where the dart landed.

A counterfactual is simply the missing comparison. If the candidate passed, what would the unchanged version have done on that same case, under the same conditions? Without that answer, your explanation has a hole right where causation should be.

Ranex’s design prevents the shortcut structurally. An ExperimentObservation belongs in a separate experiment-record namespace and binds the experiment, pair, arm, subject, target, treatment, outcome digests, and fault classification. An ordinary observation cannot become a grade just because someone likes its number.

Freeze the ruler first

You should freeze the material and the rules before the candidate exists. If the candidate can shape the ruler, a favorable result tells you far less than it appears to.

The design freezes a TargetCorpus and resolves a TreatmentManifest before dispatch. The treatment includes the relevant dependencies, model and provider metadata, harness, handbook, rules, process, workflow, prompt, skills, tools, and budgets. An alias without a provider-issued immutable version is ineligible unless the owner approves that concrete version.

That sounds strict because reproducibility has a strict requirement. You cannot rerun “whatever model that name meant last week” and call it the same treatment. You also cannot let a proposed change alter the corpus that judges it. The candidate is untrusted input, never authorization.

For the first scope, ADR-016 fixes one candidate and one binary PASS/FAIL primary endpoint. Before the trial, the design preregisters corpus size, alpha, practical-effect threshold, minimum discordant count, and fault policy. It uses a paired A/B trial on the same frozen corpus, with deterministic counterbalanced order and a fresh worktree or session.

Inconclusive is a valid answer

An inconclusive result means the comparison did not earn a stronger claim. It is not a soft pass waiting for a launch date.

The first design uses exact McNemar on discordant pairs and reports an exact Clopper-Pearson interval for discordant-win probability. SUPERIOR requires more candidate wins than losses, the preregistered minimum number of disagreements, a two-sided result at or below alpha, and the registered practical margin. INFERIOR is symmetric. A grader or registered-fault-limit failure is FAULT. Every other complete result is INCONCLUSIVE.

Those names are not branding. They stop a familiar move: seeing a secondary metric improve, ignoring a failed primary endpoint, and declaring progress anyway. ADR-016 makes secondary metrics descriptive and ineligible. It also states an important limit: exact power and sample-size design for this paired test is UNVERIFIED because the research did not find mature permissive exact McNemar power code.

The design says what it can grade, what it cannot yet establish, and what happens when the evidence is thin.

A checklist before you call it better

You can take this process to your own change, whether or not your work involves AI. The point is to stop the result from choosing the rules that justify it.

• Name the primary outcome. Choose one result that decides the comparison.
• Freeze the cases. Keep the candidate from editing the target after it is proposed.
• Resolve the treatment. Record the exact version, dependencies, tools, and settings you actually ran.
• Run a paired control. Compare old and new on the same material rather than comparing two unrelated good days.
• Set failure policy first. Decide which infrastructure faults void a pair and which outcomes count against the candidate.
• Keep promotion separate. A grade is evidence; an accountable owner still decides whether to promote.
• Allow inconclusive. If your process cannot return that answer, it is built to confirm a belief.

Promotion is not self-approval

A comparative grade does not authorize promotion. ADR-016 keeps promotion separate, requiring authenticated owner approval, stale-base checking, monotonic versioning, and a new version for rollback.

This protects the boundary between a proposal and a decision. A candidate generator can recommend a revision. It cannot declare itself superior, approve itself, or convert a one-off run into authority. Unknown fault classification blocks and returns INCONCLUSIVE; absent evidence never promotes.

That separation matters for people too. You can ask an assistant for a draft, a review, or a proposed fix. You still need a process that tells you whether the evidence supports changing the thing people depend on.

Where Ranex stands

Ranex measurement is not implemented. ADR-016 is an accepted source-backed design only; it does not claim a measurement feature in the working product.

Its measurement-local M0 is a disposable two-week prototype that opens only after ADR-017 P0 permits it. M0 must produce a digest-bound green exit record before future F1 and later production work can proceed. A red record followed by dropping or superseding the design is an allowed result.

Ranex overall is pre-release. The README says the product has a working verdict path and little else; this measurement path remains future work. Treat that status as a constraint on every claim in this note.

Questions people actually ask

What is the difference between an observation and a grade?

ADR-016 defines an observation as a recorded run and a grade as the result of a preregistered controlled paired comparison; ordinary observations structurally cannot grade.

What must be fixed before a paired trial starts?

ADR-016 requires the frozen corpus, resolved treatment, primary endpoint, sample size, alpha, practical threshold, minimum discordant count, and fault policy to be fixed before the trial.

Is Ranex measurement live today?

Ranex measurement is not live today: ADR-016 is an accepted design, its M0 prototype is future work behind ADR-017 P0, and the product remains pre-release.

Before you call a change a win, write down the old version, the cases, the threshold, and the answer you will accept if the result stays unclear. Try it. Break it. Tell me what broke.

Disclosure: this post was drafted with AI assistance. Every factual claim traces to the repository’s README or slice records — the same fact gate the product enforces on code. It ships only after Anthony’s own review.