This came up back on Reddit in the “what’s that book?” sub. Someone posted asking about a book; they had 6 lines of preamble about why they wanted to reread this, and 55 lines that were an excerpt from the book. This is normally a slam dunk–it’s pretty trivial to search for literal text. But nobody could find it. Discussion with the user led to the revelation that they’d asked AI about a book and described a scene they remembered, and the AI had responded with a title and “excerpt” matching what the user described. But then the user still couldn’t find the book, so they posted the “excerpt” to the subreddit. I.e., they were asking for a book containing an AI hallucination, rather than just describing the book to us the way they had to the AI. In fact the story they were searching for was “Harrison Bergeron,” one of the most commonly asked-about things on the subreddit, but nobody recognized it until we understood that the excerpt was slop made up in response to a scene they misremembered as part of the story.
It’s a frustrating time to be a human on the internet.
Man, that sucks. It’s really annoying how sycophantic these companies have made their chatbots, which makes this worse.
But, I think by the very nature of how chatbots work, you can’t fix that problem. Chatbots simulate what text is likely to follow other text.
If someone online has definitely read a book that really exists, and they describe a scene they remember from that book, the clear pattern online is that someone will say “Oh yeah, that’s the book X by author Y”. Some people might say “I’ve never heard of that book”, but it’s likely they might just not say anything. If we’re looking at a kind of forum with upvotes and downvotes, the person who found the real book will be the most upvoted.
So, that’s the pattern the LLM trains on. It knows that when someone posts a question like that, the best answer is one where someone names the book and types out the passage in the book.
It would be nice if the chatbot would instead say “I don’t know that book” or “That book doesn’t exist”.
But, it can’t do that, because it doesn’t “know” anything. Everything the chatbots put out is a hallucination. It’s just that some hallucinations are more useful. It has been trained millions of times on patterns when the passage someone remembers is “It was the best of times, it was the blurst of times”, so when it hallucinates about that sequence of words, its hallucination is bound to be more helpful. But, if what someone remembers doesn’t exist, or is obscure enough that it’s not in the training data, things break. Then it follows the general pattern used for every other book, and comes up with a “useful” answer that doesn’t match any actual book.
It’s definitely in the nature of the current-gen systems. It’s possible to imagine other ways to train chatbots that wouldn’t favor bullshitting in the same way, though. The way they currently are isn’t a way they necessarily have to be.
Anyway, it’s very clearly a case where it would have been better to cut out the middleman and just share the prompt, which was my point in sharing it.
This came up back on Reddit in the “what’s that book?” sub. Someone posted asking about a book; they had 6 lines of preamble about why they wanted to reread this, and 55 lines that were an excerpt from the book. This is normally a slam dunk–it’s pretty trivial to search for literal text. But nobody could find it. Discussion with the user led to the revelation that they’d asked AI about a book and described a scene they remembered, and the AI had responded with a title and “excerpt” matching what the user described. But then the user still couldn’t find the book, so they posted the “excerpt” to the subreddit. I.e., they were asking for a book containing an AI hallucination, rather than just describing the book to us the way they had to the AI. In fact the story they were searching for was “Harrison Bergeron,” one of the most commonly asked-about things on the subreddit, but nobody recognized it until we understood that the excerpt was slop made up in response to a scene they misremembered as part of the story.
It’s a frustrating time to be a human on the internet.
Man, that sucks. It’s really annoying how sycophantic these companies have made their chatbots, which makes this worse.
But, I think by the very nature of how chatbots work, you can’t fix that problem. Chatbots simulate what text is likely to follow other text.
If someone online has definitely read a book that really exists, and they describe a scene they remember from that book, the clear pattern online is that someone will say “Oh yeah, that’s the book X by author Y”. Some people might say “I’ve never heard of that book”, but it’s likely they might just not say anything. If we’re looking at a kind of forum with upvotes and downvotes, the person who found the real book will be the most upvoted.
So, that’s the pattern the LLM trains on. It knows that when someone posts a question like that, the best answer is one where someone names the book and types out the passage in the book.
It would be nice if the chatbot would instead say “I don’t know that book” or “That book doesn’t exist”.
But, it can’t do that, because it doesn’t “know” anything. Everything the chatbots put out is a hallucination. It’s just that some hallucinations are more useful. It has been trained millions of times on patterns when the passage someone remembers is “It was the best of times, it was the blurst of times”, so when it hallucinates about that sequence of words, its hallucination is bound to be more helpful. But, if what someone remembers doesn’t exist, or is obscure enough that it’s not in the training data, things break. Then it follows the general pattern used for every other book, and comes up with a “useful” answer that doesn’t match any actual book.
It’s definitely in the nature of the current-gen systems. It’s possible to imagine other ways to train chatbots that wouldn’t favor bullshitting in the same way, though. The way they currently are isn’t a way they necessarily have to be.
Anyway, it’s very clearly a case where it would have been better to cut out the middleman and just share the prompt, which was my point in sharing it.