One evidence-first platform, proven on two hard tasks. Every result is pass, fail, or unknown — lack of evidence can never become success. All numbers below come from real runs, not self-report.
Lower pass-rate here is higher honesty: stubs that point elsewhere are labelled, not faked.
Before the audit, confidence was a flat ~0.958 for everything. Now:
The answer to “how do you ensure status is trustworthy?” — not an LLM judging itself, but SEC’s own structured numbers. A real Item 8 must contain the reported revenue / net income / assets; a wrapper stub cannot.
Same search task on two site versions. v2 drifts the UI; the agent repairs itself via the accessibility tree and the verifier confirms real results.
The verifier itself is measured against labelled corruptions, impossible tasks probe silent failure, and mutation sites bend the site along three axes.
Ground truth by construction: known-good runs vs four corruption
classes. Positive = verdict pass; unknown never counts as success.
Impossible-task probes, pass@k repeats, trajectory hygiene and the opt-in false-success triage — all measured, none self-reported.
Teal = final success, grey = mid-task checkpoint (query reached the real field). A red stub marks success 0. Checkpoint ≠ success localises where the failure happened.
A third extraction engine triangulates every item, SEC’s own CYD iXBRL block-tags anchor Item 1C, char-offset F1 locks span boundaries, and landmine cases are executable regressions.
Coverage = share of the officially tagged span our segment covers. The two disagreements are wrapper 10-Ks whose 1C is an honest reference stub.
Where the pipeline is proven vs honestly unsupported — pre-2001 plain-text SGML fails cleanly (0 fabricated items).