On hard problems the most confident answers are the wrong ones, and confidence-based selectors pick exactly those.
GPT-OSS-120B · LiveCodeBench v6
Correct completionsIncorrect completions
Data table
The idea
Not how confident. When.
Late confidence means the reasoning converged. Early confidence means the model committed before exploring.
A model that sees several routes splits its probability between them, so good reasoning starts uncertain and ends certain. That is consilience: a conclusion is trustworthy when several lines of reasoning arrive at it.
Ends confident
Ends uncertain
Starts confident
Mean-confidence favours
Starts confident · ends confident
Premature convergence
Commits instantly. Confidently wrong on hard problems.
Starts confident · ends uncertain
Degeneration
Starts sure, then breaks down.
Starts uncertain
Consilience targets
Starts uncertain · ends confident
Consilience
Explores, then converges. The trace to pick.
Starts uncertain · ends uncertain
Total confusion
Never finds a path.
Sketches. Grey ticks mark the scoring windows.
The score
S = Cfinal − α·Cinitial
Reward certainty earned at the end. Penalise certainty claimed at the start. Keep the highest-scoring trace.
ctToken confidence: −mean of the top-5 log-probs. High when the alternatives are ruled out.
CinitialMean ct over the first 20% of reasoning, after skipping 5%.
CfinalMean ct over the last 20% of reasoning.
αPenalty weight. Default 3; anything in 2–5 works.
Measured: correct traces gain confidence as they go
Correct minus incorrect confidence along the trace (normalised), hard problems, mean ± SEM. Own y-scale per panel.
The gap rises in every setting. On hard problems consilience separates correct from wrong (AUROC 0.61–0.62); mean confidence cannot (0.47–0.50).
Data table
Playground
Turn the knob yourself.
α = 0 picks the trace that jumped to an answer. A moderate α picks the one that explored. Too large an α picks one that never converged.
Confidence of each trace in the pool
simulated, for intuition
Mean-confidence pickConsilience pickOther traces
Simulated traces. Shaded: skipped prefix, then the Cinitial and Cfinal windows.
0 = last-window confidence only · 2–5 recommended
The same knob on real benchmarks
Best-of-64 accuracy vs α. Grey line: pass@1 (random pick). The slider above moves the cursor.
α = 0 often lands below random. The default α = 3 beats it in every setting.
Data table
Results
Gains where verifiers and voting can't help.
Code and agent actions have nothing to vote on and no cheap verifier. Consilience still works.
LiveCodeBench v6 GPT-OSS-120B
69.7%
+4.0 over pass@1 (65.7%), best of 64
HMMT 2025 GPT-OSS-20B
80.7%
+4.2 over pass@1 (76.5%); mean confidence falls to 68.0%
SWE-bench Verified GPT-OSS-120B · mini-SWE-agent
26.9%
+3.9 over the plain agent (23.0%)
LiveCodeBench, medium tier GPT-OSS-120B
88.6%
+13.7 over mean-confidence selection (74.9%)
As n grows, confidence maximisers fall below random. Consilience doesn't.
A Claude Code skill that adds consilience to any LLM system, single-turn or agentic, on any provider. It checks the model exposes top-K log-probs, finds the generation call, wraps it without changing its interface, and verifies the scorer against known-answer tests. It asks you when a choice is yours to make.
# install for Claude Code
curl -LO https://consilience-tts.github.io/skill/consilify.zip
unzip consilify.zip -d ~/.claude/skills/
# then, inside your project
claude "consilify this agent"
Read it first: SKILL.md (the method, defaults, CONSILIENCE_* settings and integration steps) and reference.py (reference scorer with known-answer tests).
Case study
The confident trace is wrong.
Same model, same hard problem: Qwen3-80B on a LiveCodeBench tree problem that 4 of 128 samples solve. One trace commits early; the other keeps exploring.
Confidence over the first 2,048 reasoning tokens
Click a numbered marker or excerpt to link the two.
…When visiting a node: If its value is already in the path, we cannot extend further (since values must be unique). Otherwise, we can extend the path…
Never revisited. It becomes a return that prunes valid subtrees.
2Token ~766: starts writing code
## Implementation
```python
class Solution:
def longestSpecialPath(self, edges, nums):
Commits early, stays confident, fails the tests.
Consilience pick✓ Correctinitial confidence 5.56
3Token ~1,989: still questioning the framing, and finds the key
…we cannot have any path from any ancestor before the previous occurrence of this value to u… But we can have paths from ancestors after the previous occurrence of nums[u] to u.
Wait, that's key!
…So the only valid paths ending at u are those that start from an ancestor after the last occurrence of nums[u]…
Still exploring at token 2,000. Low confidence here is the model keeping alternatives alive.
The problem (LiveCodeBench #3, pass@1 = 0.03)
Given an undirected tree rooted at node 0 with n nodes, where each node has a value nums[i] and each edge has an integer length. A special path is a downward path (ancestor → descendant) in which all node values are distinct. Return [max_length, min_nodes]: the length of the longest special path, and the minimum number of nodes among all such longest paths. (n ≤ 5·10⁴, nums[i] ≤ 5·10⁴.)
Cite
BibTeX
@article{kong2026consilience,
title = {Consilience for Verifier-Free Test-Time Scaling},
author = {Kong, Lecheng and Hui, Like and Mao, Haitao and Huan, Jun},
journal = {arXiv preprint arXiv:2608.09898},
year = {2026},
url = {https://arxiv.org/abs/2608.09898}
}