A Very Good Guess

Matt Birchler links to a piece from 404 Media about chatbots being terrible doctors:

I think there's something to be understood here about the difference between memorizing every bit of information about something, with the ability to deploy that information effectively.

This is the bit that gets lost in every breathless announcement about an LLM passing a bar exam or acing a medical test. Passing a test and being a doctor are two completely different things. One requires pattern matching against a dataset. The other requires judgement, context, and the ability to sit across from another human and figure out what's actually going on.

LLMs are not alive. They don't "know" things. They are prediction machines that guess the most logical output based on the data they've been trained on. As much as the companies might lead you to believe otherwise, and I genuinely think Anthropic are under the impression Claude is alive, all LLM output is a guess. The more powerful they get, the better the guess. It's still a guess.

When Google or OpenAI show off their latest model acing a medical licensing exam, they're showing you a party trick. A doctor doesn't just know what's in the textbook. They read body language. They weigh up your history against what you're telling them right now. They factor in that you might be lying about how much you drink. They make calls with incomplete information, knowing the stakes. An LLM can only predict the next most likely token in a sequence.

We keep conflating information with intelligence and it's making us all dumber. A library contains more knowledge than any single person, but it doesn't write books. Knowing that a medication treats a condition is not the same as knowing when to prescribe it, or when to try something else entirely.

Memorising every medical textbook doesn't make you a doctor. Training on the entire internet doesn't make you a writer. It makes you a very fast, very confident parrot.