Alex Molas argues Jev's 'calibrated probabilities' can't be calibrated, citing a 0.92 heads prediction for a fair coin
alexmolas.com / Hacker News (49pts, 58 comments)·low signal
Molas argues that calibration depends on each user's data distribution, so a model that returns the same probability for everyone can be calibrated for one company and badly off for another. He cites Jev giving 0.92 for a fair coin landing heads when the prompt itself implied 0.5. He recommends treating Jev scores as rankings and recalibrating on a few hundred labeled examples of your own. This is a technical critique, unlike the parody and clone repos from earlier in the Jev wave.