bayesianCalibration
Do the probabilities mean what they say? We matched the engine's verdicts against 1,577 labelled claims: AVeriTeC and FEVER benchmark claims, our own test inputs, multilingual documents and the verified media sample. We then scored them by Brier score, expected calibration error (ECE), abstention and label buckets.
Read these numbers with their caveats.
- The document labels are provisional. A model assigned them from primary sources, and there is no human-labelled sample yet.
- Fast mode decides little without research. In the baseline, 551 of 609 matched rows came out “insufficient evidence” (p = 0.5, u = 1). Most of the calibration error comes from that vacuous 0.5 on document facts that are 81–98% true, not from wrong answers. All 28 decided rows were right.
- The verified-mode numbers are a small sample. They rest on 13 benchmark rows, which proves the path, not the number.
- The archival reference runs come from earlier builds. For decided media claims, the labeller started from the engine's own research evidence.
- Uncertainty ranks errors only weakly among decided verdicts (AUROC 0.57–0.65). The wrong verdicts are confident ones.
Loading the reports…
Sources: eval/calib/reports/*.json|md|svg, generated by eval/calib/score_calib.py. The same summary comes back from GET /api/v1/stats under calibration.