Counterfactual Incident Replay: Asking What a Different Policy or Action Would Have Changed, With Zero Live Effects by Import Shape

Pitch

Because revl's admission rules and action classifier are pure functions, a logged incident can be re-graded offline under another policy, or with one agent action swapped, and the policy reader imports no runtime tier, so nothing in it can fire an effect.

Abstract

Post-incident questions of the form "would the new policy have stopped this" are usually answered by re-running the incident or by reading a policy simulator that never saw the incident. Both are unsatisfying for agents that act on the world: a re-run repeats the effects, and a simulator answers about a hypothetical rather than about the logged run. We describe two counterfactual readers in revl, a language whose compiler and gate mediate an agent's capability use. The first, revl replay WAL --under POLICY, regrades the boundary crossings logged in a write-ahead log under an alternate policy and a candidate composition, names the earliest crossing the new policy would have refused, and reports whether the candidate is admitted. The second, revl_counterfactual, takes a session's action log, replaces, inserts or drops one action, decides both arms with the gate's own pure classifier and ticket rules, and reports the first divergence and the downstream steps it changes. Neither module imports the runtime tier. For the policy reader, a test runs a full counterfactual in a fresh interpreter and asserts the runtime never enters the module table, and another pins the WAL digest equal before and after. The action reader is pinned only by a test that makes runtime calls raise, and eight of its nine tests are skipped unless the cordis package is installed. We state what the approach cannot say: the WAL holds no admit decision and no host response, so the first reader recomputes reach rather than re-deciding the crossing, and the gap for a free host-extern emission depends on the tier that wrote the log: on the Python tier such an emission is now recorded (backends/python/runtime.py:1388-1414), while the bound the reader prints still says it leaves no record, so for those logs the printed text is stale; other tiers were not checked. The implementation is covered by 14 tests for the policy reader, 9 for the action reader and 26 for the related policy-diff.

Problem and claim

Incident review asks counterfactual questions: would policy B have refused the crossing that hurt us, would this candidate component have been admitted, would the gate have decided differently had the agent done something else at step k. Existing answers either re-execute (unsafe for an agent whose actions are real) or evaluate the policy on synthetic inputs (not about the incident).

The claim is narrow and checkable: when the decision procedure is a pure function of logged data and static compiler output, a counterfactual can be answered with no effect authority, and the absence of authority can be a property of the code (no import path to a runtime) rather than a discipline. The claim is paired with an equally precise statement of where the answer is only a recomputation. The design doc says so itself: the crossing report recomputes the alternate policy over the reach the log names rather than re-deciding the crossing that was logged (docs/design/467-counterfactual-replay.md, section "What this surface cannot answer").

On main, the WAL reader recomputes reach; it does not re-decide the crossing and it does not compute a causal cone. The session-log reader does re-decide, with the gate's own classifier, and reports downstream steps changed by an approval shifting, but over the in-memory session log, not a WAL.

Contributions

1. A WAL reader that regrades every logged emission crossing, in logged order, as permitted, refused or not-in-candidate under an alternate policy, names the earliest refused crossing, and separately reports the whole-policy admission verdict for a compiled candidate (src/revl/counterfactual.py:375 replay_under; CLI src/revl/cli/parser.py:1951-1965).
2. A structural zero-effect property for the policy reader: no runtime import, WAL digest equal before and after, and a report field stating liveEffects: 0, each pinned by a test (tests/test_467_counterfactual_replay.py:499).
3. A catalogue of printed bounds that the answer cannot cross (authority absent from the log, free host-extern emissions on a tier that does not record them, reach-leg only per crossing, torn WAL, realm rules without a candidate), carried on every report rather than in prose.
4. A single-action substitution reader over a session's action log (replace, insert, drop) that decides both arms with the gate's pure parts, shifts approvals with the surrounding calls, and reports per step whether the original arm reproduces the receipts, so a divergence is not read off an arm that did not reproduce (src/revl/mcp/counterfactual_act.py:52-226, tests/test_counterfactual_act_1752.py).
5. A policy-diff preview over a WAL with a bounded blast radius and a refusal to report "unchanged" on any leg the WAL cannot decide (src/revl/policy_diff.py, tests/test_468_policy_diff.py).

Related work

None of the above was checked against a specific paper; every row needs a proper citation pass.

Evidence on main

Tree: revl main at commit 45bec431f.

Numbers that exist

None as measurements. Test counts only, from counting test functions in the files (not a run): 14 in tests/test_467_counterfactual_replay.py, 9 in tests/test_counterfactual_act_1752.py, 26 in tests/test_468_policy_diff.py. No latency, accuracy or case-study data exists.

Paper outline

1. Introduction: the incident question and why re-running is not acceptable for acting agents.
2. Background: gate, WAL, capability tokens, boundary policy (docs/boundary-policy.md).
3. Policy counterfactual over a WAL: crossings, candidate, verdicts, ordering.
4. Action counterfactual over a session log: classifier, tickets, approval shifting, reproduction check.
5. Zero effects by shape: import discipline, digest pinning, report field, the test with teeth.
6. What the log cannot say: the printed bounds, and why not logging the decision is deliberate.
7. Evaluation: incident corpus, fidelity of recompute, comparison with a generic policy engine.
8. Related work and limits.

Threats to validity and limits

• The docs name the largest caveat: a crossing reported permitted may have been refused at the time, and one reported refused may have been covered by an approval edge the log does not hold. The design deliberately keeps the WAL free of decisions (src/revl/recovery.py holds the same line). This is a design trade, not a defect to be hidden.
• Per-crossing verdicts read only the capability reach leg; taint, register floors, evidence bundles and tenant partitions are heard only in the whole-candidate admission verdict.
• A crossing is identified by (component, label), so a candidate that moves a crossing between scopes under the same label is not distinguishable.
• "Admitted" means the boundary policy admits the candidate, not that the gate would boot it.
• The action reader's reach is the static closure, so a listed crossing may not fire for this data; no result value or data-dependent path is computed; ttl is not simulated; a log spanning a swap is refused.
• The zero-effect test checks the Python module's imports; it says nothing about other tiers, and nothing about the action reader beyond a test that makes runtime calls raise.
• The live half (re-executing arms) is not built.
• Eight of the nine action-reader tests carry the needs_cordis skip, so a run without cordis does not exercise them.
• Policy-diff on WALs written today yields an empty action set per its own docstring; it is not a working end-to-end preview without a scoped WAL.

Fact-check

Claims in this draft were checked against revl main at commit 45bec431f: 60 checked, 6 corrected.

Cite this draft

@misc{revlp19,
  author       = {Thomas Moussajee},
  title        = {Counterfactual Incident Replay: Asking What a Different Policy or Action Would Have Changed, With Zero Live Effects by Import Shape},
  year         = {2026},
  howpublished = {\url{https://inso1337.github.io/revl/research/P19.html}},
  note         = {Research draft, not peer reviewed. Written against revl commit 45bec431f. Licensed CC BY 4.0.}
}

Text licensed CC BY 4.0, author Thomas Moussajee.