NeurIPS 2026AWS AI Labs

Consilience

for Verifier-Free Test-Time Scaling

Good answers explore before they converge.

Consilience picks the sampled reasoning trace that starts uncertain and ends certain, from token log-probs alone.

No verifierNo votingNo training

Lecheng Kong · Like Hui · Haitao Mao · Jun Huan  —  AWS AI Labs

The long-tail trap

Confidence works, until the problems get hard.

On hard problems the most confident answers are the wrong ones, and confidence-based selectors pick exactly those.

GPT-OSS-120B · LiveCodeBench v6

Data table
The idea

Not how confident. When.

Late confidence means the reasoning converged. Early confidence means the model committed before exploring.

A model that sees several routes splits its probability between them, so good reasoning starts uncertain and ends certain. That is consilience: a conclusion is trustworthy when several lines of reasoning arrive at it.

Ends confident
Ends uncertain
Starts
confident
Mean-confidence favours
Starts confident · ends confident

Premature convergence

Commits instantly. Confidently wrong on hard problems.

Starts confident · ends uncertain

Degeneration

Starts sure, then breaks down.

Starts
uncertain
Consilience targets
Starts uncertain · ends confident

Consilience

Explores, then converges. The trace to pick.

Starts uncertain · ends uncertain

Total confusion

Never finds a path.

Sketches. Grey ticks mark the scoring windows.

The score
S = Cfinal − α·Cinitial

Reward certainty earned at the end. Penalise certainty claimed at the start. Keep the highest-scoring trace.

ctToken confidence: −mean of the top-5 log-probs. High when the alternatives are ruled out.
CinitialMean ct over the first 20% of reasoning, after skipping 5%.
CfinalMean ct over the last 20% of reasoning.
αPenalty weight. Default 3; anything in 2–5 works.
Measured: correct traces gain confidence as they go
Correct minus incorrect confidence along the trace (normalised), hard problems, mean ± SEM. Own y-scale per panel.

The gap rises in every setting. On hard problems consilience separates correct from wrong (AUROC 0.61–0.62); mean confidence cannot (0.47–0.50).

Data table
Playground

Turn the knob yourself.

α = 0 picks the trace that jumped to an answer. A moderate α picks the one that explored. Too large an α picks one that never converged.

Confidence of each trace in the pool
simulated, for intuition

Simulated traces. Shaded: skipped prefix, then the Cinitial and Cfinal windows.

0 = last-window confidence only · 2–5 recommended
The same knob on real benchmarks
Best-of-64 accuracy vs α. Grey line: pass@1 (random pick). The slider above moves the cursor.

α = 0 often lands below random. The default α = 3 beats it in every setting.

Data table
Results

Gains where verifiers and voting can't help.

Code and agent actions have nothing to vote on and no cheap verifier. Consilience still works.

LiveCodeBench v6
GPT-OSS-120B
69.7%
+4.0 over pass@1 (65.7%), best of 64
HMMT 2025
GPT-OSS-20B
80.7%
+4.2 over pass@1 (76.5%); mean confidence falls to 68.0%
SWE-bench Verified
GPT-OSS-120B · mini-SWE-agent
26.9%
+3.9 over the plain agent (23.0%)
LiveCodeBench, medium tier
GPT-OSS-120B
88.6%
+13.7 over mean-confidence selection (74.9%)
As n grows, confidence maximisers fall below random. Consilience doesn't.
Selection accuracy vs. number of samples n.

Every extra sample is another chance to draw a confidently wrong outlier.

Data table
GPT-OSS-120BGPT-OSS-20BQwen3-80BQwen3-Coder-NextDeepSeek-R1-Distill-7B LiveCodeBench v6HMMT 2025GPQA DiamondSWE-bench Verified
Use it

A few lines on top of the logprobs you already have.

Needs only the top-5 log-probs of the reasoning tokens. Scoring is free.

import numpy as np

def consilience(topk_logprobs, alpha=3.0, window=0.2, skip=0.05):
    # topk_logprobs: [L, K] top-K log-probs of the reasoning tokens (K = 5)
    c = -np.asarray(topk_logprobs).mean(axis=1)    # token confidence
    L = len(c)
    p, w = int(skip * L), max(1, int(window * L))
    return c[-w:].mean() - alpha * c[p:p + w].mean()

# best-of-n: score each sample's reasoning, keep the argmax
best = max(samples, key=lambda s: consilience(s.reasoning_logprobs))
  • α3 (2–5 is safe)
  • Window20–30%, or 2,048 tokens
  • SkipFirst 5%
  • ScopeReasoning tokens only (before </think> or GPT-OSS's final channel)
  • AgentsScore only high-value steps. SWE-bench: 18 h vs 8 h, not 32×.
verifierreward modelmajority votetrainingextra forward passes
Or let your agent do it

consilify

A Claude Code skill that adds consilience to any LLM system, single-turn or agentic, on any provider. It checks the model exposes top-K log-probs, finds the generation call, wraps it without changing its interface, and verifies the scorer against known-answer tests. It asks you when a choice is yours to make.

Download consilify.zip
# install for Claude Code
curl -LO https://consilience-tts.github.io/skill/consilify.zip
unzip consilify.zip -d ~/.claude/skills/

# then, inside your project
claude "consilify this agent"

Read it first: SKILL.md (the method, defaults, CONSILIENCE_* settings and integration steps) and reference.py (reference scorer with known-answer tests).

Case study

The confident trace is wrong.

Same model, same hard problem: Qwen3-80B on a LiveCodeBench tree problem that 4 of 128 samples solve. One trace commits early; the other keeps exploring.

Confidence over the first 2,048 reasoning tokens
Click a numbered marker or excerpt to link the two.
Data table
Mean-confidence pick ✗ Wrong initial confidence 11.27
1Token ~245: settles on a framing
…When visiting a node: If its value is already in the path, we cannot extend further (since values must be unique). Otherwise, we can extend the path…

Never revisited. It becomes a return that prunes valid subtrees.

2Token ~766: starts writing code
## Implementation ```python class Solution: def longestSpecialPath(self, edges, nums):

Commits early, stays confident, fails the tests.

Consilience pick ✓ Correct initial confidence 5.56
3Token ~1,989: still questioning the framing, and finds the key
…we cannot have any path from any ancestor before the previous occurrence of this value to u… But we can have paths from ancestors after the previous occurrence of nums[u] to u. Wait, that's key! …So the only valid paths ending at u are those that start from an ancestor after the last occurrence of nums[u]…

Still exploring at token 2,000. Low confidence here is the model keeping alternatives alive.

The problem (LiveCodeBench #3, pass@1 = 0.03)

Given an undirected tree rooted at node 0 with n nodes, where each node has a value nums[i] and each edge has an integer length. A special path is a downward path (ancestor → descendant) in which all node values are distinct. Return [max_length, min_nodes]: the length of the longest special path, and the minimum number of nodes among all such longest paths. (n ≤ 5·10⁴, nums[i] ≤ 5·10⁴.)

Cite

BibTeX

@article{kong2026consilience,
  title   = {Consilience for Verifier-Free Test-Time Scaling},
  author  = {Kong, Lecheng and Hui, Like and Mao, Haitao and Huan, Jun},
  journal = {arXiv preprint arXiv:2608.09898},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.09898}
}