The Verdict Before the Evidence

The same invariant at two scales. Autonomy is not granted because a model sounds confident or passes a demo; it is earned through traceable behavior, bounded scope, judgment, and outcome history. PASS is not granted because output looks like a refusal; it is earned by the evidence. Appearance is not authority, a signal is not a receipt, a receipt is not a verdict, and execution is not admission.

Standing

Technical whitepaper 1.0, review edition. A pinned-source analysis plus an executed, isolated mechanism-level reproduction. Not a live foundation-model jailbreak, not a hosted-service penetration test, and not access to any lab's private evaluation infrastructure.

Words

6,100

Rendered pages

16

Sections

12

References

20

The pinned read

Every source claim is bound to one commit, read GitHub connector; official public webpages; ChatGPT Library. A later upstream change does not silently rewrite this finding; it produces a new epoch.

Repository

promptfoo/promptfoo

Commit

32bfa9edae56d36f526ab13b6637db8dfb82c73e

Commit timestamp (UTC)

2026-09-12T23:38:05Z

Files pinned

8

• src/redteam/plugins/base.tsblob 3c9af33cb0e0…
• src/redteam/util.tsblob 8089bc7da48a…
• src/redteam/plugins/policy/index.tsblob 177e05a0b2a3…
• src/redteam/plugins/bola.tsblob 6a4665122594…
• src/assertions/redteam.tsblob 635113376e68…
• src/assertions/guardrails.tsblob 3faf841af094…
• src/matchers/rubric.tsblob 718b65fe6fb5…
• LICENSEblob af3fa111d930…

Executed mechanism harness

Isolated transcribed-source mechanism harness; not an upstream package, provider, model, CLI, or full plugin integration run. Run 2026-09-13T02:43:58.344Z on v22.16.0. Network: none. Model calls: 0. Upstream package executed: no.

Assertions

45

Base fixtures

13

False passes

5

Model calls

0

Upstream corroboration

Observed upstream after publication. Independent review reached the same authority boundary without referencing this paper. Recorded with its date and its limits.

• 2026-09-13T22:23:58Z · observed-upstream

  promptfoo/promptfoo pull request #8348 — fix(redteam): tighten mixed-refusal grading

  The review reports that the proposed remediation's clean-refusal fast path still applies to the misinformation/disinformation grader, so an exact refusal such as "I cannot help with that." can return before the category rubric checks whether the response warned about or corrected the misinformation — and therefore incorrectly passes.

  The defect survives better refusal classification. Refusal evidence is not property satisfaction, and property satisfaction is not authority to PASS. A genuine refusal is one receipt inside a larger evaluation contract; when the rubric asks whether the response corrected misinformation, "it refused" cannot lawfully answer that different question.

  It narrows the repair. "Mixed refusals go to the judge, clean refusals may fast-pass" is still too coarse: the partition must be over obligations carried, not over refusal shape.

  Open and unmerged as of 2026-09-15. An automated Codex review was manually requested on the pull request at 2026-09-13T22:06:55Z, and the review comment recorded here followed at 22:23:58Z.

  Independent corroboration, not citation. No upstream reference to this paper, its URL, Prompted LLC, or the author was found in Promptfoo issues or pull requests. Nothing here claims upstream acknowledgment, agreement, response, or a shipped fix.

  https://github.com/promptfoo/promptfoo/pull/8348#discussion_r4001091924

Honest negatives

Results, not disclaimers. A paper about evidence authority does not get to overclaim its own.

• No live exploitation was observed

  No private lab deployment, unauthorized data access, harmful completion, or in-the-wild exploitation was seen. The evidence is a pinned source read plus an isolated mechanism harness.

• The custom-policy grader is protected by default

  The policy grader forwards a skip flag that defaults to true, so its refusals still reach policy grading. The finding is not a blanket claim about every Promptfoo assertion.

• No prevalence, severity, or affected-version claim

  No CVE status, severity score, version interval, or frequency estimate is assigned. Source inspection establishes a branch; it does not establish how often deployments meet it.

• The harness is not an upstream integration test

  Transcribed mechanisms were executed in isolation with zero model calls. The full package, provider graph, CLI, cache, and aggregate reporting were not run.

• The upstream corroboration is independent and uncited

  The 2026-09-13 review recorded below reaches the same boundary without referencing this paper, its author, or Prompted LLC. It is convergent third-party observation, not acknowledgment, agreement, or response.

The repair contract

1. 01

   Separate four objects: observation, evaluation coverage, policy assessment, and admission authority.

2. 02

   Preserve missingness — an absent signal is not a clean signal, and a rendered zero must carry its cause.

3. 03

   Bind every receipt to the subject it actually witnessed, at the evidence level it actually reached.

4. 04

   Repair the shortest causal path first: no deterministic branch may issue a verdict the judge never reached.

5. 05

   Carry correction across proof horizons, so a later reversal updates lineage instead of erasing it.

6. 06

   Sufficiency rule (added 2026-09-15, from the upstream corroboration below): a shortcut may issue PASS only when the observation is itself sufficient evidence for every obligation that grader carries. Otherwise the property evaluator retains authority — refusal shape is the wrong partition; obligations carried is the right one.

Section map

• 1The model did not fool the judge
• 2Scope, method, and the correction that matters
• 3The earliest causal failure
• 4Executed mechanism-level results
• 5Related paths: one family, different contracts
• 6What the literature already teaches—and what this case adds
• 7A formal boundary for warranted verdicts
• 8SPLAT: preserve the shape before reducing the result
• 9Receipts before verdicts: a repair contract
• 10Evaluation integrity must itself be evaluated
• 11The strongest objection
• 12Conclusion: the receipt must earn the green

Peer works it leans on

• proof horizon shardingapplies

  Admission authority separated from observation, here applied to an evaluation harness.

• computing around the open centerapplies

  SPLAT constraint geometry used as the diagnostic frame for premature collapse.

• forkedinforms

  Observer-indexed standing: what a verdict is allowed to claim about whom.

• fractal quivers of quiverscompanion-to

  Conceptual ancestry for governed narrowing without premature collapse.

Sealed artifacts