Don't ask an LLM for a confidence score: 78% of answers cluster on three values.

Models don't treat 0-100 as continuous. Verbalized confidence is badly overconfident (Xiong et al.) and only calibrates after task-specific fine-tuning (Lin et al.). The recursion problem is the core of it: if you can't trust the answer, you can't trust the model's confidence in its confidence. The recommended alternative is semantic entropy (Farquhar et al.). Note this post is a synthesis, not original experiments. (justinflick.com)