Investigate and maybe implement "Deep Think with Confidence"

[hamannju]

Hello, first let me clarify that I am no expert on this subject so I apologize in advance if I get something wrong or accidently wasted your time.

I just saw a post on Twitter by an AI researcher from Meta making these claims:

"Introducing DeepConf: Deep Think with Confidence

🚀 First method to achieve 99.9% on AIME 2025 with open-source models! Using GPT-OSS-120B even without tools, we reached this almost-perfect accuracy while saving up to 85% generated tokens.

It also delivers many strong advantages for parallel thinking:
🔥 Performance boost: ~10% accuracy across models & datasets
⚡ Ultra-efficient: Up to 85% fewer tokens generated
🔧 Plug & play: Works with ANY existing model - zero training needed (no hyperparameter tuning as well!)
⭐ Easy to deploy: Just ~50 lines of code in vLLM (see PR below)"

Here is the link to it: https://x.com/jiawzhao/status/1958982524333678877

To me it sounds pretty much too good to be true, but maybe someone who understands the llamacpp project better could take a look at it? I would be up to testing an implementation or contribute in any other way that I can because unfortunately I do not have C++ programming experience.

[hamannju]

Oh someone has already opened an issue about this, too: #15518

[ggerganov]

Quickly skimming the paper, few impressions:

• Feels like rediscovered beam search with extra hoops
• Not sure how it reduces generated tokens if you need to generate 100s of traces?
• AIME25 99.9% (from baseline 97%) feels a bit weak given that it depends on a free parameter η which is unclear how to determine. For example using η = 10% improves GPT-OSS AIME25 but worsens some other benchmarks

Could be missing something, but does not feel very relevant for local/edge use cases where we usually don't have the resources to generate ensembles. But either way, I don't think it requires any changes in llama.cpp to be able to implement this approach - it's entirely client-side filtering and weighting of tokens/traces.