Article URL: https://justinflick.com/2026/07/27/llm-confidence-scores.html Comments URL: https://news.ycombinator.com/item?id=49077443 Points: 29 # Comments: 1

If someone sent you this post, you have probably tried to extrude a confidence score out of an LLM. I’ve now had this conversation at multiple companies with multiple people, and I’ve had it enough times that it seems like there’s a broader misunderstanding at work here. So my hope is that I can just send people this post instead of relitigating it in a thread every six months. The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless. I want to be upfront that I’d love to be wrong about this. There are people much, much smarter than me working in this space (and I work with many of them!). If you can rebut any of the arguments below, or point me at research I haven’t read, I would genuinely enjoy that conversation. But absent that, these are my current thoughts as I understand them. The pattern shows up everywhere. Chat-based outputs, structured outputs, agentic task results. Someone wants the model to hand back a JSON object with a response key and a confidence key. And more often than not, that confidence key is a continuous score from 0 to 100. That continuous scale is what really grinds my gears, and I’ll come back to why. But the broader problem is that there’s no scientific validity behind any of it. What you have built is a psychological safety trick. It makes the output feel more trustworthy without making it more trustworthy, and I’d argue the people shipping this experience are mostly lying to themselves about what they’ve shipped. I recognize that Anthropic and others have published research arguing that models maintain some kind of latent internal state while they generate. Anthropic’s interpretability work on tracing the thoughts of a language model found that Claude plans several words ahead when writing a rhyming couplet, which is not the behavior you’d predict from a naive next-token story. And their more recent work on emergent introspective awareness found that models can sometimes notice concepts injected into their activations and report on them accurately. That’s real, and it’s interesting. But read the caveat the researchers themselves put on it: the capability is highly unreliable and highly context-dependent. We do not currently have a strong enough understanding of that internal state to assert that models have any usable ability to assess their own correctness. Noticing that something was injected into your activations under laboratory conditions is a very long way from quantifying how likely your paragraph about a customer’s refund policy is to be right.