Green That Measures Nothing: Dead Test Switches, Silent Skips and Typechecked Demos That Never Run

Pitch

A skip is the same colour as a pass, so tests gated behind a switch no CI job sets, a toolchain no CI job installs, or a demo that is only typechecked report green while measuring nothing. We classify the gate shapes, measure what executing them costs and finds, and give a guard that makes each shape a build failure.

Abstract

Test suites fail silently in a way dashboards do not show: a correct test is gated behind an environment variable, a filesystem probe or a toolchain that continuous integration never provides, so every test reports skipped, and skipped is green. We report an experience from a multi-backend compiler repository where this happened three times in one quarter. One job ran pytest tests/ without cordis-py installed, so 398 tests across 81 modules executed nowhere. A cross-tier suite had its slow probes gated behind a variable set in no workflow, so the Rust and Java halves never ran and two Java correctness defects sat under assertions that already covered them. A TypeScript demo asserted an impossible answer for months because CI typechecked it but never ran it. We describe four gate shapes (unset variable, filesystem probe, right variable in the wrong job, typechecked-only file), the audit that found them, and a static guard that reads the workflow files and fails when a switch, a toolchain or a test file is not exercised by a job that can fail the build. Executing the skipped tests of the cross-tier file on one machine moved it from 30 s with 93 passed and 94 skipped to 6 min 31 s with 180 passed and 5 skipped: the pass count rose by 87, and the totals at the two ends differ by two tests that the source does not explain. We state what the guard does not catch, and which instances remain open.

Problem and claim

A test that is skipped reports no failure. In a repository that supports six code-generation tiers, each needing its own toolchain, "skip when the toolchain is absent" is the right local behaviour and the wrong CI behaviour: a job that installed the toolchain and still skips has a broken provisioning step, and nothing says so. The same holds for slow tests gated behind an opt-in variable that nobody opts into.

The claim: this is a class with a small number of recognisable shapes, most of its documented instances were found by tripping over a defect the dead suite already covered rather than by looking for the class, and it can be closed by two cheap mechanisms. One is a switch that turns "toolchain absent" into a failure for the tiers a job has provisioned. The other is a static check that every gate and every tier test is wired to a job that can fail the build.

The claim is not that the guard is complete. The guard's own docstring lists what it does not catch.

Contributions

1. A taxonomy of four gate shapes with an instance of each from one repository: an environment variable set nowhere; a filesystem probe (node_modules absent) in place of a variable; the right variable set in the wrong job; and a file that is typechecked but never executed.
2. A static guard, tests/test_env_gated_skips_run_somewhere.py, that reads the CI workflow files and fails when a variable that tests read is set nowhere, set only in a job that does not run the reading test, or when a tier test is not run by a step requiring its tier; plus the guard's own check that it cannot skip.
3. A runtime switch, REVL_REQUIRE_TIERS, and a marker type (Absent) that turn only the "toolchain not installed" kind of skip into a failure while leaving by-design refusals as skips (src/revl/test.py:175-215).
4. A measured cost and yield of executing the skipped tests: one file, 30 s to 6 min 31 s, 93 to 180 passes, 87 more passes, one failure found (Java structural equality) and fixed before the flip.
5. A negative result on conformance validators: they stop at compile depth, so a cross-tier value question cannot be answered by the corpus, which is why a typechecked demo could be wrong for months.

Related work

Evidence on main

Tree: revl main at commit 45bec431f. The relevant roadmap entries are 430, 433, 446 and 453 in docs/v2.0-roadmap.md. Item 430 has no heading of its own; it appears only in the prose of 446.

Numbers that exist

• 30 s, 93 passed, 94 skipped; 3m53, 128 passed (node_modules); 4m53, 156 passed (plus slow switch); 6m31, 180 passed, 5 skipped (plus JDK). 87 tests newly run. Source docs/v2.0-roadmap.md:1872. n = 1 file, 1 machine, 1 run per configuration; no variance reported. The totals disagree: 93+94=187 at baseline, 180+5=185 at the flip; the source does not explain the two missing tests.
• 398 tests across 81 modules not executed in CI before item 430. Source docs/v2.0-roadmap.md:1864. Count as stated in the roadmap; not recomputed.
• 17 skipif decorators on REVL_CROSS_TIER_SLOW, in 2 files. Source docs/v2.0-roadmap.md:1866.
• 1 failing probe out of the full five-tier matrix once everything ran. Source docs/v2.0-roadmap.md:1874.
• 2 Java correctness defects found under already-written assertions (item 433, R1 and R2). Source docs/v2.0-roadmap.md:1864, docs/v2.0-roadmap.md:1729.
• Of the documented instances, the unset-variable cases (430, 433) and the typecheck-only case (453) were found by tripping over a defect; the filesystem probe was found while measuring; the wrong-job case was found by audit (446).
• Not measured: how many silently skipped tests exist in comparable repositories, the false-positive rate of the guard, time-to-detection before versus after the guard. None yet.

Paper outline

1. Introduction. The incident, why skip equals pass, summary of the three occurrences and the measured yield.
2. Setting. A compiler with six backends, per-tier toolchains, cross-tier "one source, compare values" tests.
3. Four gate shapes, each with its instance and how it was found.
4. The audit. Separating real gates from overrides ("one dead switch plus three overrides").
5. The mechanisms: the require-tiers switch with the Absent marker; the static guard; the guard's self-check and the non-vacuity check on the demo step.
6. Measurement. Cost and yield of executing the skipped tests; the one failure; the wrong "no JDK" premise.
7. What conformance validators cannot answer, and the demo case.
8. Limits and open instances. Related work.

Threats to validity and limits

• The measurements are one laptop, one file, one run per configuration; the source labels them as measured on an arm64 laptop. They are not CI timings.
• All instances are from one repository and were reported by the people who fixed them. No external prevalence data.
• The guard does not catch a toolchain-probe skip for a tier outside the revl.test runners, pytest.importorskip elsewhere, or anything it cannot see in the workflows; its docstring says so (tests/test_env_gated_skips_run_somewhere.py:1-60).
• The rule "conformance CI fails if a tier is unavailable" holds only for tiers named in REVL_REQUIRE_TIERS for that step (py,ts,go,rust). Java and wasm are not required yet, so a missing JDK or wasm substrate still reads green there.
• One right-variable-wrong-job instance (REVL_CORDIS4J_CLASSES) and the wasm demo are explicitly left open on main.
• The Java defects were first verified by emitted-code inspection and later by execution on a real JVM; the earlier "no JDK on this machine" statement was a wrong premise.
• The 398 / 81 figure comes from roadmap prose and has not been independently recomputed.
• The two-test gap between the baseline and final totals is unexplained, so the figure of 87 more passes is quoted as a pass-count difference only.

Fact-check

Claims in this draft were checked against revl main at commit 45bec431f: 46 checked, 9 corrected.

Cite this draft

@misc{revlp70,
  author       = {Thomas Moussajee},
  title        = {Green That Measures Nothing: Dead Test Switches, Silent Skips and Typechecked Demos That Never Run},
  year         = {2026},
  howpublished = {\url{https://inso1337.github.io/revl/research/P70.html}},
  note         = {Research draft, not peer reviewed. Written against revl commit 45bec431f. Licensed CC BY 4.0.}
}

Text licensed CC BY 4.0, author Thomas Moussajee.