Of course an LLM does have a latent concept of its own confidence that is internally consistent, and is perfectly able to articulate it, the author just chose the worst possible way to ask. Using a 0-100 scale for anything is a bad idea with both LLMs and people. Instead of rating it 0-100, split your problem into many dimensions/axes and give each a small and meaningful enum or just a less granular numerical scale (ideally just a binary choice). Then apply weighted scoring to the result. You'll be surprised how much better it becomes at meaningfully expressing its confidence this way.
YouTube removed the 1-5 scale for the same reason, people have no idea what to do with it.
I would also like a source for this, I do not believe it is true and founded that "an LLM has a latent concept of its own confidence that is internally consistent".
LLMs generally cannot report the outputs of their own activation functions. Taking the output of softmax as if it were a probability score is considered a first-year PhD student misconception.
They can generally do a decent enough job with a likert scale and a rubric. Which isn’t quite the same thing as a confidence score and won’t be calibrated at all, but may be preferable to users.
Peoper confidence scores on a continuous scale, or even a fine grained discrete scale, are arguably skeuomorphic. It’s worth introspecting on whether that’s what you really needed in the first place, or if it’s just what you’re used to from past experience working with regression-type models.
Exactly my experience: given a set of 3-5 scores a and a sentence or two for each describing what earns that rating is quite effective in getting LLMs to score accurately. Failing to give a rubric doesn't give enough context, and using many values (like 1-100) produces more arbitrary results.
"LLM" literally converse/compute via statistical inference.
It is given sets of paramaters/data and infers a response based on it's acceptable confidence level. The bottleneck for this is compute power and model/prompt/rails/integrations.
The reason LLM stays consistent/can be somewhat trusted to stay on rails is that it's aware of both it's context/scope and it's confidence about the context.
Anything else would be....original thought and exposition? I guess. You could say that our brains do the same, IE; unconcious statistical analysis leading to inference based on confidence.
You converse through electrical signals and muscle contractions, but that will not make you an expert in either.
Similarly, a language model trained to emulate speech, follow instructions, and solve problems will not implicitly have an understanding of its own statistics. The research doesn't back that up and it just doesn't make any sense if you think about it.
The only thing LLM produce that could be considered its "confidence" would be its token logit values, but even that is a sketchy connection.
> It is given sets of paramaters/data and infers a response based on it's acceptable confidence level. The bottleneck for this is compute power and model/prompt/rails/integrations.
Does the output of the model have insight into those details? Just because our brains function based upon electrical potential doesn't mean we have innate insight into it.
Yes, this is what is user-level marketed as context memory.
Certain models of course force this context memory, some don't, some allow you to toggle it's "trigger level".
For example Anthropic, use haiku for low memory context queries, research. Then Opus/Fable for compiling/analysis/design, which relies on the LLM inference constantly going back into context memory.
Another example is OpenAI/GPT. The "free" tier forces context memory using the "highest available model" per your subscription (or lack thereof), and upon hitting its limit, drops you down to a low context model, and then ultimatley blocking that chat entirely.
(I appreciate this discussion and am not trying to inform you, I'm sure you already know, but typing these things out helps me understand my holes in understanding.)
Excuse my ignorance, but how does context memory provide an insight into a model's inner workings? I mean this:
> It is given sets of paramaters/data and infers a response based on it's acceptable confidence level. The bottleneck for this is compute power and model/prompt/rails/integrations.
or whatever the LLM claims its confidence is on any given output?
This is an incorrect conclusion. Your statement about statistical inference is correct but it doesn't explain why your initial statement, especially "is perfectly able to articulate it".
A good litmus test for this is, typically any artificial neural network, perceptron or even a simple linear regression model do statistical inference based on weights and selection based on confidence level. It's a generic characteristic, which doesn't explain anything specific to LLMs.
> The reason LLM stays consistent/can be somewhat trusted to stay on rails is that it's aware of both it's context/scope and it's confidence about the context.
Post hoc ergo propter hoc.
> I guess. You could say that our brains do the same
I understand that it "feels" the same from a certain perspective, but we don't know that.
You are only talking about what you're feeling. Care to actually make a statement about what you think is true, besides implicitly denigrating someone else?
LLMs have trouble with open labeling tasks, this was studied.
But, what also was studied is that if you instruct it how to score something, it can do it not so far than a human. If you leave out the instructions, I believe it can be considered an open codebook and therefor it probably wont work well.
If it is nominally a probability then make sure it’s well calibrated.
If it is something like cosine similarity or SVM scores, which aren’t probabilities or even percentages, then users will misinterpret it if you present it as a number from 0-100 or 0.0-1.0.
Lets be honest with Youtube's system: it's not about what people really want; it's about whats enough to make them _feel_ in control.
What many people would want it just a complete block on select youtube channels and providers. Just whole hog, rip them out of any of the potential recommendations.
But that's not whats best for Youtube. The point system still stands in ratings, once exposed to the viewer, are gamified. They would however still work if it was a secret vote, but then, again, the user wouldn't be compelled to use it as people want immediate visual cues that they're doing what benefits themselves.
That's very well put. I noticed the issue myself when building my own evals and naturally started drifting into the direction you are describing, but you describe it so succinctly that now I am seeing how it should actually be done. Thank you
Yeah, I agree this approach can produce usable results. The more narrow and well scoped you can make each decision the model has to make (e.g. binary vs 0 to 100) the better the results you'll get (and the easier it is to troubleshoot and tune).
>>YouTube removed the 1-5 scale for the same reason, people have no idea what to do with it.
As always, I'm in a minority - I have a VERY clear idea what to do with it :-).
(1/2 - never ever show me crap like this again. 3 - decent mainstream stuff. 4 - This meets my interests, is good, and I'd like more of it please. 5 - this is endorphin analog candy for me. Scott Pilgrim, Hunt for Red October, Neuromancer, Dune grade content :)
The scale is almost never useful. You cannot quantify most things. Even with humans you get inconsistent results and confusion. As the parent said, a binary (ok/not ok) or ternary (bad/ok/good) choice is what’s appropriate, most of the time. If this is not granular enough, the vast majority of the time it means that you need more axes.
OP says that LLMs were never scored against correctness. But that's an assumption about the optimization process.
This is sort of right:
- pretraining scores on next token, and says nothing about confidence
- RLHF and RLVR scores are binary (1=preferred/correct, 0=worse/incorrect) and the feedback doesn't do anything to push the weights in a direction that improves calibration/confidence
BUT you can ask the model to output its confidence, and then train it to be more correct. RLCR (mentioned in the article I linked above) adds a Brier penalty to the reward, so the model's performance isn't just measured on % correctness. (Presumably TypeSafeAI's RLCD is similar to RLCR, but they haven't published anything about it.)
This eliminates the "always say 30%" loophole OP mentioned. The Brier penalty, (q-c)^2, punishes a forecaster who ignores instance-level information. For example if we have a model that's right 60% of the time, it could use one of these strategies to give its confidence:
Strategy A, always say 0.6:
When right (60% of the time): (0.6-1)^2 = 0.16
When wrong (40% of the time): (0.6-0)^2 = 0.36
Expected penalty: 0.6×0.16 + 0.4×0.36 = 0.24
Strategy B: the model can tell its answers apart. Half are 'easy' (right 90% of the time) and half are 'hard' (right 30% of the time), which still averages to 60%. It says 0.9 on easy ones and 0.3 on hard ones:
Easy: 0.9×(0.1)^2 + 0.1×(0.9)^2 = 0.09
Hard: 0.3×(0.7)^2 + 0.7×(0.3)^2 = 0.21
Expected penalty: (0.09+0.21)/2 = 0.15
Both strategies are perfectly calibrated (60% is both the average confidence and thje average % of time it's right), but B gets a lower penalty. Of course these are not actual strategies used by LLMs. But I'm just trying to say why training can push the model toward optimizing instance-specific confidence.
I got a degree in statistics, and what I remember from those times is that we spent maybe 10% of our time learning how to build the model and 90% of the time correctly interpreting the model output. Of course, in the time I learned statistics it was almost exclusively applied in hard science, finance, and actuary. The wild west of machine learning hadn't gotten started. I wonder if they teach it differently now.
When we start using big datasets where n went into the tens of millions we kinda stopped bothering because margins were so small it was effectively certain you found the mean of some range. However I think in that time we forget what confidence scoring actually is and how quickly multiplied alpha can balloon on you.
From my time building ML models (2016-2023) evaluation and interpretation was a large chunk of it. I often used "brute force" interpretation methods but there were others like SHAP [1] and LIME [2]. The other large chunk of ML work focused on feature engineering and data prep. Today in the LLM agentics era this is out the window. There are "evals" but no one does it seriously, and interpretation is often propagated to the consumer.
I am happy to hear high quality evaluations are out of fashion. That will make the competition so much less hard in the EU where evaluation is a regulatory requirement.
On the other hand, it’s kind of expected that the hard part of the job in AI/ML engineering gets neglected over that part people can brag on social media about. Deeply concerning but sadly understandable.
Its embarrassing how much people are larping as ML folks and just building things that manifestly have no evidence that they work. I have seen it the last 3 places I have worked and most of the folks seem to have no interest in learning how to get consistently good results, just ship it and focus on the next thing.
It has always seemed to me crazy (and maybe a bit lazy) that people will just blindly use self-generated confidence scores from LLMs without having some sort of validation mechanism.
Also, as another poster said: decomposing your rating into as many dimension as possible is better way to generate some confidence score grounded in fact.
Confidence scoring is fine when you provide the right context, like everything else LLM related. I use grading scripts for things that can be measured directly so the LLM just operates on it. For things less well-defined, I use a grading matrix with explicit criteria on what constitutes an A to F grade, usually with an assumption of an A grade that drops depending on specific findings.
If you are asking about confidence and somehow it can’t be defined either mathematically or in a grading rubric, then you the implementer need to start asking yourself the questions, because then you got a target that you don’t even know how to measure. The LLM isn’t even the problem at this point because another human would also just be going off gut feeling and vibing a number that may or may not be accurate.
Yeah I agree - if you're very clear and consistent with how you ask it score something it will do well. Not fundamentally different than giving it clear instructions when you code.
One thing that has helped me make the LLM more consistent on scoring is deduction based rubrics. Start everything at perfect and deduct points for specific things (ie style, missed facts, etc). It gives much tighter final scores under these conditions (for me) than giving rubrics + samples.
In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.
I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.
or put the classical ML model after an LLM-based model that includes a prompt to give a "yes"/"no" answer represented with specific tokens and then retrain the last few layers of that model on your specific data and freeze the rest.
For classification. You do need training data for this to work, because (a) the "zero-shot" model of "write a prompt for ChatGPT" doesn't work that well compared to less powerful models that are trained with, say, 5000 samples and (b) the probability estimates depend on the probability distribution of the data you are showing it. That is, the data you are classifying might have more or less things in the class and that "prior distribution" affects how confident the model should be. If you train on data that is very different from the data that you're working on your calibration will be poor.
Anyhow if you want it in a box see "Decision Models"
I was hoping the writer would follow the interesting framing of the quality of an LLM's confidence score with some data on how good it is, rather than just proclaiming it's bad because of sycophancy tendencies. If someone collected the data on something measurable, I would be willing to bet the LLM's confidence number isn't too far off.
I’ve run those experiments. Commercially, so I can’t share data. But I can say what the article describes roughly matches what I’ve seen. How well calibrated the scores are varies by model, application, and subtle wording changes. I’m a little more optimistic than the author in that I have gotten it to work reasonably well. But I would not recommend productizing it with a SaaS LLM because it will break every time your vendor sunsets the model you were using, and getting it working again on a new model will be an unholy PITA.
From that paper, model accuracy is not perfect, but it's overall pretty favourable. There are some encouraging findings in there, too, including that accuracy of model confidence scales with reasoning effort.
I would guess it tends towards more likely because it can't not think of something mentioned.
But I agree, I would8 much rather have some data than reckons.
It's with noting that people do this too. They tend to vastly overetimate a low probability event that they know is possible. Often by more than one order of magnitude.
It is something worth keeping in mind when you are people throwing around personal p doom scores.
Interestingly when humans have to make a decision based upon the probability they feel, they tend to act far closer to the actual probability than what they would say the probability is.
I think you're making the wrong takeaway here. LLMs aren't bad because of sycophancy tendencies. Any prediction at all is bad when it isn't based on real data, whether or not it ends up being correct, whether or not it is made by an LLM, a human, or some other thing capable of spitting out a number.
Analogize to basketball here. How confident are you that you can hit a free throw? If you ask someone who has played a thousand hours a year for the past ten years, has recorded their own statistics, and has averaged 70% success rate oer those past ten years, and they tell you they're 70% confident they can hit a free throw, that's a good prediction. If you ask me and I give you any prediction at all, I'm full of shit. I haven't played basketball or taken a free throw in 30 years. I can pull a number out of my ass, I can reason through it, I can do a Fermi, I can assume I'll be the average of all people who have ever tried and do my best to collect whatever data might exist on brand new players with no experience trying to take a free throw for the first time. I can narrow that to people my height, my age. Now we're getting better, but it's still not as good as having a real record of my actual past performance and basing a prediction off of that.
This is just science. Hierarchy of evidence. Epistemology. Sometimes answering a question requires collecting data that doesn't exist yet and waiting a while, in the meantime admitting you don't have an answer. It was already sorely lacking in the world, but in this age of oracles, it's only getting worse.
Isn’t this what jev training is supposed to improve? They don’t train on predicting the next token but predicting the distribution of next tokens, or something like that.
I think it's good to continually remind oneself that these are next-word/token-predictors...you can wrap them up in a harness, you can feed them into each other, you can influence the likelyhood of a given word through weights and system prompts...but that's literally all they can do. They can't reason, but they can predict what reasoning looks like.
None of this means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, and be honest with ourselves about what they could ultimately be capable of.
we don't know how the brain works yet, and I don't see a thread on hackernews "new paper describes how brain works by mannytv" with 1000 comments so don't talk out of your ass
It’s equally useful to continually remind yourself that a Turing machine is just a spool of tape with a magnetic head, and that a human brain is just fatty lipids in a chemical bath.
The majority of humans is not delusional all the time. But every human is delusional some times.
Same as the majority of LLMs hallucinating some times but not all the time.
So if the bar for proper reasoning is never ever being delusional then humans and LLMs sit in the same bucket.
They are delusional in ways that don't overlap, though. An LLM will not be delusional in the way Van Gogh or Howard Hughes was, but humans (sane ones anyways) will not hallucinate the exact kind of odd, unpredictable LLM output that makes you go "huh? why the heck did it do this in this way or write this." Things which are extremely obvious well within the limits of even elementary human reasoning are completely outside the purview of model capability; it's much easier for a model to "reason" when it comes to systems and ideas that are very discrete (which is why it's so useful when writing code, esp. in higher-level languages -- little uncertainty there and a round-robin verification process for workflows like mathematics. However once you ask it to intuit anything that falls apart quickly; its main utility lies in being able to compensate for these deficiencies with sheer volume and burning through tokens)
Even on the frontier models these blips still happen fairly often and it's intrusive to a lot of workflows.
Calling it delusion is too severe. Our reasoning fails us all the time. For example, if you get a problem wrong on a math test, you’ve simply made a mistake, and with a little more “prompting” (even of your own volition) you could often easily arrive at the correct answer. Now if you stubbornly persist with the wrong answer even in the face of evidence to the contrary, then you could call it delusion. But my point is, there’s a spectrum of mistakenness.
Well, the human is just a big bag of neurons. So in what sense is its "understanding" more real?
It is perhaps sad, but with our current knowledge of neuroscience, we can't really argue that humans are somehow special. All we can say is that in practice we are still more intelligent/understanding/meaningful/conscious/whatever than LLMs. But for how long?
When you use "next word predictor", it sounds like slow process, like each word take few seconds to minute to be generated. Why not call it "sentence predictor", or "paragraph predictor" ? i would argue it's the same in principle.
It's like calling a video clip a "picture sequencer".
It's literally what a LLM is, and it's not even a word it's a token which is about 2/3'rds of a word. It has nothing to do with speed of producing tokens. Others are not responsible for correcting your bad understanding of a true statement.
We've found that you can fine-tune confidence scores for your domain into existing models. I think the way this works is that you basically help the model understand what types of inputs it tends to get right and which ones it tends to get wrong. This seems like a reasonable thing you'd expect a model to be able to learn
I recently did an experiment with LLMs where I asked them to rate the odds of a given outcome explaining an observation, and I found they are systematically biased much like humans [1].
Even if they can do great maths I think bayesian statistical fundamentals are not reliable at this point.
I know that you can use LLMs to write ascii numbers that represent a "confidence" as in TFA. This is fraught and strange and hard to trust.
But can't you also use the next token likelihood itself, so long as you rearchitect to use the input classes as the only possible output tokens? Something like stripping away speech and only working with the input labels as vocabulary...
I guess I thought that's what decision-like models were doing to output confidence scoring. (or something like that)
That is the opposite of dumb, you can use the next token likelihood to macgyver together a jev in a pinch exactly that way, you just would also want to take some steps to scramble up the choices to smooth over bias and tokenization artifacts. llms have favorite letters for example. Qwen likes A and D.
Recently joined a company that does just this. They are using the LLM to review text and produce a grade and a confidence interval.
I'm no expert in the field but it is clear that the model outputs are not reliable. To my frustration this was designed by "experts" so I originally trusted that this approach was sound.
How does this compare with descriptive confidence? For example, major v.s. minor? Intuition tells me it's a different flavor of the same illusion.
I have Claude score threads on socials related to my SaaS.
Claude found this thread and wrote something how his own scoring sucks and he just marks any relevant thread 7/10. Thought it was funny how he just roasted my product while also being the cause of that product sucking.
Items 1-5 at the end of the article basically describe a supervised learning process. Data + labelling, updating weights, calibration.
Yes, using an LLM for confidence scoring while generating prose is a bad idea. I expected a mention of decision models here, as that exactly what they are trained for.
But also... isn't the entire point of "confidence scoring" to tell you how confident the model in use is? I mean, I kind of agree, Claude still tells me things it does in like 5 to 15 minutes will take many months (which for a human, maybe a few weeks).
I think JEV has shown the industry you don't need LLMs for everything, and the point of confidence scoring is to know pretty well. I think an LLM could mess up confident scoring, but for the most part, for very specialized tasks with small context windows, I could see it getting it mostly right. I do wonder if having "criteria" defined for how it should determine its own confidence score would work, but then at that point, why not just ask it to validate the criteria and you measure the confidence of the models answer, each specific answer would bear some of the weight into the confidence score anyway no?
I'm not a guru, just thinking aloud. I think there's right to be concerned, the other day a buddy showed me a screenshot (last week) of google suggesting if he has RAM problems to buy new RAM, because RAM is cheap. Models can be stupid.
An LLM call is just text until you act upon it, so why not send many requests for the exact same prompt and then externally measure the likelihood of a specific tool choice being made?
I'm pretty sure this is how reasoning models work at the big providers. Averaging 10 lunas together likely provides more stable outcomes than one big sol invoke. In any case, you could use the statistical distribution to decide on subsequent inference steps.
Whilst general self-reporting of confidence will be hard to achieve (we can't even do it as humans), I'm sceptical of the notion that we can't train models to give some confidence scores accurately. I think over the past year that with the correct RL, models can showcase some pretty insane capabilities. If reporting the correct confidence was a RL training target say for predicting weather events or a game outcome that is supported by real RL environment or simulated data, then I suspect LLMs will develop a strong intuition for confidence similar to how the recent JEV and decision models are trained.
So can you not do structured output and get the log probs across all your allowed output classes? Then use that relative measure of how confident it is?
While I agree the autoregressive training is poor at calibrating a model, it's clear that LLMs encode a kind of world model to be able to reason, do mathematics, programming, and logic. They are certainly imperfect, but the emergent internal representations go beyond mere "what text comes next" even though LLMs typically simply use tokens.
By analogy to your 'Measuring confidence' section: a function estimator (neural nets can be good at this) and a bunch of past observations could be pretty calibrated at making weather predictions. An LLM has a wild mix of past observations and random collated opinions from previous text (it might mention months, recency of rain etc.. etc...) so lands somewhere between 'text completion' and 'estimator' with a model of unknown fidelity. It is not 'merely' generating a plausible sentence.
I would never use an LLM for accurate forecasting, and certainly agree they are not calibrated, but I would not dismiss them out of hand.
The author has used weather prediction confidence as an example. I think it is not analogous to LLM's next-token prediction. LLM's next-token is predicted from a finite statistical model. And for the weather prediction model, there is no finite data-based statistical model.
I was surprised that the author hasn't mentioned Jev-like models. They're quite good for general purpose confidence rating.
LLMs are bad for rating or scoring anything because they have the sensibilities of the average person. Unfortunately, the average person has poor taste.
This is a tough one to explain because some people think the average person has good taste by definition. I used to think that way but over the years I've realized that a lot of what the average person likes is derivative 'slop' - This was true before AI and still true with AI.
Once you start deducting points for lack of originality, you will realize that people have really poor taste... Very narrow taste. Most people literally consume trashy variations of the same thing over and over. We have to start deducting points for this because the more intelligent among us can't take it anymore! We're tired of being exposed to thousands of variants of the same slop... It might be sufficiently different to fool the average pleb's simple mind but to us it's like eating the exact same dish every meal of every day with different amounts of salt and pepper.
This comes through when you ask an LLM to rate/score something. It's frustratingly conventional and not based on first principles thinking and it ADDS extra points for being derivative instead of SUBTRACTING as it should. We can see it play out in the workplace. LLMs have poor judgement. It's the one area where they lag behind.
YouTube removed the 1-5 scale for the same reason, people have no idea what to do with it.
LLMs generally cannot report the outputs of their own activation functions. Taking the output of softmax as if it were a probability score is considered a first-year PhD student misconception.
Peoper confidence scores on a continuous scale, or even a fine grained discrete scale, are arguably skeuomorphic. It’s worth introspecting on whether that’s what you really needed in the first place, or if it’s just what you’re used to from past experience working with regression-type models.
Can you provide credible reference for this?
It is given sets of paramaters/data and infers a response based on it's acceptable confidence level. The bottleneck for this is compute power and model/prompt/rails/integrations.
The reason LLM stays consistent/can be somewhat trusted to stay on rails is that it's aware of both it's context/scope and it's confidence about the context.
Anything else would be....original thought and exposition? I guess. You could say that our brains do the same, IE; unconcious statistical analysis leading to inference based on confidence.
Similarly, a language model trained to emulate speech, follow instructions, and solve problems will not implicitly have an understanding of its own statistics. The research doesn't back that up and it just doesn't make any sense if you think about it.
The only thing LLM produce that could be considered its "confidence" would be its token logit values, but even that is a sketchy connection.
Does the output of the model have insight into those details? Just because our brains function based upon electrical potential doesn't mean we have innate insight into it.
Certain models of course force this context memory, some don't, some allow you to toggle it's "trigger level".
For example Anthropic, use haiku for low memory context queries, research. Then Opus/Fable for compiling/analysis/design, which relies on the LLM inference constantly going back into context memory.
Another example is OpenAI/GPT. The "free" tier forces context memory using the "highest available model" per your subscription (or lack thereof), and upon hitting its limit, drops you down to a low context model, and then ultimatley blocking that chat entirely.
(I appreciate this discussion and am not trying to inform you, I'm sure you already know, but typing these things out helps me understand my holes in understanding.)
> It is given sets of paramaters/data and infers a response based on it's acceptable confidence level. The bottleneck for this is compute power and model/prompt/rails/integrations.
or whatever the LLM claims its confidence is on any given output?
A good litmus test for this is, typically any artificial neural network, perceptron or even a simple linear regression model do statistical inference based on weights and selection based on confidence level. It's a generic characteristic, which doesn't explain anything specific to LLMs.
> The reason LLM stays consistent/can be somewhat trusted to stay on rails is that it's aware of both it's context/scope and it's confidence about the context.
Post hoc ergo propter hoc.
> I guess. You could say that our brains do the same
I understand that it "feels" the same from a certain perspective, but we don't know that.
"It's aware" is a big leap and bold original thought, as awareness is not something it has full stop, try different terminology?
Even a probability?
If it is something like cosine similarity or SVM scores, which aren’t probabilities or even percentages, then users will misinterpret it if you present it as a number from 0-100 or 0.0-1.0.
What many people would want it just a complete block on select youtube channels and providers. Just whole hog, rip them out of any of the potential recommendations.
But that's not whats best for Youtube. The point system still stands in ratings, once exposed to the viewer, are gamified. They would however still work if it was a secret vote, but then, again, the user wouldn't be compelled to use it as people want immediate visual cues that they're doing what benefits themselves.
As always, I'm in a minority - I have a VERY clear idea what to do with it :-).
(1/2 - never ever show me crap like this again. 3 - decent mainstream stuff. 4 - This meets my interests, is good, and I'd like more of it please. 5 - this is endorphin analog candy for me. Scott Pilgrim, Hunt for Red October, Neuromancer, Dune grade content :)
OP says that LLMs were never scored against correctness. But that's an assumption about the optimization process.
This is sort of right:
- pretraining scores on next token, and says nothing about confidence
- RLHF and RLVR scores are binary (1=preferred/correct, 0=worse/incorrect) and the feedback doesn't do anything to push the weights in a direction that improves calibration/confidence
BUT you can ask the model to output its confidence, and then train it to be more correct. RLCR (mentioned in the article I linked above) adds a Brier penalty to the reward, so the model's performance isn't just measured on % correctness. (Presumably TypeSafeAI's RLCD is similar to RLCR, but they haven't published anything about it.)
This eliminates the "always say 30%" loophole OP mentioned. The Brier penalty, (q-c)^2, punishes a forecaster who ignores instance-level information. For example if we have a model that's right 60% of the time, it could use one of these strategies to give its confidence:
Both strategies are perfectly calibrated (60% is both the average confidence and thje average % of time it's right), but B gets a lower penalty. Of course these are not actual strategies used by LLMs. But I'm just trying to say why training can push the model toward optimizing instance-specific confidence.When we start using big datasets where n went into the tens of millions we kinda stopped bothering because margins were so small it was effectively certain you found the mean of some range. However I think in that time we forget what confidence scoring actually is and how quickly multiplied alpha can balloon on you.
[1] https://shap.readthedocs.io/en/latest/
[2] https://christophm.github.io/interpretable-ml-book/lime.html
On the other hand, it’s kind of expected that the hard part of the job in AI/ML engineering gets neglected over that part people can brag on social media about. Deeply concerning but sadly understandable.
Also, as another poster said: decomposing your rating into as many dimension as possible is better way to generate some confidence score grounded in fact.
If you are asking about confidence and somehow it can’t be defined either mathematically or in a grading rubric, then you the implementer need to start asking yourself the questions, because then you got a target that you don’t even know how to measure. The LLM isn’t even the problem at this point because another human would also just be going off gut feeling and vibing a number that may or may not be accurate.
One thing that has helped me make the LLM more consistent on scoring is deduction based rubrics. Start everything at perfect and deduct points for specific things (ie style, missed facts, etc). It gives much tighter final scores under these conditions (for me) than giving rubrics + samples.
I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.
https://scikit-learn.org/stable/modules/calibration.html
I've had very good luck with
but you can use simpler models like or put the classical ML model after an LLM-based model that includes a prompt to give a "yes"/"no" answer represented with specific tokens and then retrain the last few layers of that model on your specific data and freeze the rest.For classification. You do need training data for this to work, because (a) the "zero-shot" model of "write a prompt for ChatGPT" doesn't work that well compared to less powerful models that are trained with, say, 5000 samples and (b) the probability estimates depend on the probability distribution of the data you are showing it. That is, the data you are classifying might have more or less things in the class and that "prior distribution" affects how confident the model should be. If you train on data that is very different from the data that you're working on your calibration will be poor.
Anyhow if you want it in a box see "Decision Models"
https://docs.liquid.ai/lfm/models/decision-models
https://arxiv.org/html/2607.20526v1
From that paper, model accuracy is not perfect, but it's overall pretty favourable. There are some encouraging findings in there, too, including that accuracy of model confidence scales with reasoning effort.
But I agree, I would8 much rather have some data than reckons.
It's with noting that people do this too. They tend to vastly overetimate a low probability event that they know is possible. Often by more than one order of magnitude.
It is something worth keeping in mind when you are people throwing around personal p doom scores.
Interestingly when humans have to make a decision based upon the probability they feel, they tend to act far closer to the actual probability than what they would say the probability is.
Analogize to basketball here. How confident are you that you can hit a free throw? If you ask someone who has played a thousand hours a year for the past ten years, has recorded their own statistics, and has averaged 70% success rate oer those past ten years, and they tell you they're 70% confident they can hit a free throw, that's a good prediction. If you ask me and I give you any prediction at all, I'm full of shit. I haven't played basketball or taken a free throw in 30 years. I can pull a number out of my ass, I can reason through it, I can do a Fermi, I can assume I'll be the average of all people who have ever tried and do my best to collect whatever data might exist on brand new players with no experience trying to take a free throw for the first time. I can narrow that to people my height, my age. Now we're getting better, but it's still not as good as having a real record of my actual past performance and basing a prediction off of that.
This is just science. Hierarchy of evidence. Epistemology. Sometimes answering a question requires collecting data that doesn't exist yet and waiting a while, in the meantime admitting you don't have an answer. It was already sorely lacking in the world, but in this age of oracles, it's only getting worse.
Asking random experts their confidence is already pretty bad most of the time.
If you want a confidence score you need a way to consistently model the system you are trying to estimate the outcome of.
Anything else is just numerology.
None of this means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, and be honest with ourselves about what they could ultimately be capable of.
None of that means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, etc etc.
They don't always match up.
Sometimes the map IS the territory, or at least superior to it.
As far as I'm aware, when reasoning does not meet reality, it is called a delusion, and I'm fairly certain that majority of humans are not delusional.
So if the bar for proper reasoning is never ever being delusional then humans and LLMs sit in the same bucket.
Even on the frontier models these blips still happen fairly often and it's intrusive to a lot of workflows.
Stallman vehemently states llms are bullshit generators with no true understanding.
But sometimes you can approximate your problem and unfold an llm interlocator to producing the strings you need for your task
It is perhaps sad, but with our current knowledge of neuroscience, we can't really argue that humans are somehow special. All we can say is that in practice we are still more intelligent/understanding/meaningful/conscious/whatever than LLMs. But for how long?
It's like calling a video clip a "picture sequencer".
Even if they can do great maths I think bayesian statistical fundamentals are not reliable at this point.
[1] https://think-twice.me/is-ai-biased-or-sexist-making-chatgpt...
I know that you can use LLMs to write ascii numbers that represent a "confidence" as in TFA. This is fraught and strange and hard to trust.
But can't you also use the next token likelihood itself, so long as you rearchitect to use the input classes as the only possible output tokens? Something like stripping away speech and only working with the input labels as vocabulary...
I guess I thought that's what decision-like models were doing to output confidence scoring. (or something like that)
It's thought to work "better" but score calibration is still a problem.
I'm no expert in the field but it is clear that the model outputs are not reliable. To my frustration this was designed by "experts" so I originally trusted that this approach was sound.
How does this compare with descriptive confidence? For example, major v.s. minor? Intuition tells me it's a different flavor of the same illusion.
Claude found this thread and wrote something how his own scoring sucks and he just marks any relevant thread 7/10. Thought it was funny how he just roasted my product while also being the cause of that product sucking.
Yes, using an LLM for confidence scoring while generating prose is a bad idea. I expected a mention of decision models here, as that exactly what they are trained for.
In fact, LLMs are good at forecasting now: https://www.metaculus.com/questions/43900/bot-superiority-in...
I think JEV has shown the industry you don't need LLMs for everything, and the point of confidence scoring is to know pretty well. I think an LLM could mess up confident scoring, but for the most part, for very specialized tasks with small context windows, I could see it getting it mostly right. I do wonder if having "criteria" defined for how it should determine its own confidence score would work, but then at that point, why not just ask it to validate the criteria and you measure the confidence of the models answer, each specific answer would bear some of the weight into the confidence score anyway no?
I'm not a guru, just thinking aloud. I think there's right to be concerned, the other day a buddy showed me a screenshot (last week) of google suggesting if he has RAM problems to buy new RAM, because RAM is cheap. Models can be stupid.
I'm pretty sure this is how reasoning models work at the big providers. Averaging 10 lunas together likely provides more stable outcomes than one big sol invoke. In any case, you could use the statistical distribution to decide on subsequent inference steps.
If it's deciding between two phrasings, I imagine the confidence you get will drop, even if the phrasings are equivalent.
This approach with more data for calibration is basically the beginning of making a decision model like Jev.
By analogy to your 'Measuring confidence' section: a function estimator (neural nets can be good at this) and a bunch of past observations could be pretty calibrated at making weather predictions. An LLM has a wild mix of past observations and random collated opinions from previous text (it might mention months, recency of rain etc.. etc...) so lands somewhere between 'text completion' and 'estimator' with a model of unknown fidelity. It is not 'merely' generating a plausible sentence.
I would never use an LLM for accurate forecasting, and certainly agree they are not calibrated, but I would not dismiss them out of hand.
I was surprised that the author hasn't mentioned Jev-like models. They're quite good for general purpose confidence rating.
It's basically setting the reasoning mode to 'pro' on OpenAI's API.
This is a tough one to explain because some people think the average person has good taste by definition. I used to think that way but over the years I've realized that a lot of what the average person likes is derivative 'slop' - This was true before AI and still true with AI.
Once you start deducting points for lack of originality, you will realize that people have really poor taste... Very narrow taste. Most people literally consume trashy variations of the same thing over and over. We have to start deducting points for this because the more intelligent among us can't take it anymore! We're tired of being exposed to thousands of variants of the same slop... It might be sufficiently different to fool the average pleb's simple mind but to us it's like eating the exact same dish every meal of every day with different amounts of salt and pepper.
This comes through when you ask an LLM to rate/score something. It's frustratingly conventional and not based on first principles thinking and it ADDS extra points for being derivative instead of SUBTRACTING as it should. We can see it play out in the workplace. LLMs have poor judgement. It's the one area where they lag behind.