In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset. Some tasks look similar, but change one word and it turns from easy to very hard. I don't think a single step classifier can predict it without calibration work.
I think it's good to continually remind oneself that these are next-word/token-predictors...you can wrap them up in a harness, you can feed them into each other, you can influence the likelyhood of a given word through weights and system prompts...but that's literally all they can do. They can't reason, but they can predict what reasoning looks like.
None of this means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, and be honest with ourselves about what they could ultimately be capable of.
LLMs are bad for rating or scoring anything because they have the sensibilities of the average person... And unfortunately, the average person has poor taste.
This is a tough one to explain because some people think the average person has good taste by definition. I used to think that way but over the years I've realized that a lot of what the average person likes is derivative 'slop' - This was true before AI and still true with AI.
Once you start deducting points for lack of originality, you will realize that people have really poor taste... Very narrow taste. Most people literally consume trashy variations of the same thing over and over. We have to start deducting points for this because the more intelligent among us can't take it anymore! We're tired of being exposed to thousands of variants of the same slop... It might be sufficiently different to fool the average pleb's simple mind but to us it's like eating the exact same dish every day with different amounts of salt and pepper.
This really comes through when you ask an LLM to rate something. It's frustratingly conventional and not based on first principles thinking. We can see it play out in the workplace. LLMs have poor judgement. It's the one area where they are way behind.
None of this means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, and be honest with ourselves about what they could ultimately be capable of.
They don't always match up.
This is a tough one to explain because some people think the average person has good taste by definition. I used to think that way but over the years I've realized that a lot of what the average person likes is derivative 'slop' - This was true before AI and still true with AI.
Once you start deducting points for lack of originality, you will realize that people have really poor taste... Very narrow taste. Most people literally consume trashy variations of the same thing over and over. We have to start deducting points for this because the more intelligent among us can't take it anymore! We're tired of being exposed to thousands of variants of the same slop... It might be sufficiently different to fool the average pleb's simple mind but to us it's like eating the exact same dish every day with different amounts of salt and pepper.
This really comes through when you ask an LLM to rate something. It's frustratingly conventional and not based on first principles thinking. We can see it play out in the workplace. LLMs have poor judgement. It's the one area where they are way behind.