AI-powered chatbots are everywhere, from classrooms to offices to our daily lives. However, these tools have a frustrating problem: They can sometimes generate completely false, fabricated information. This phenomenon is called “hallucination.” OpenAI believes it has solved this problem and found a solution that will make future AI tools much more reliable. Here are the details.
The Problem Lies in the AI’s Evaluation Method
OpenAI, in collaboration with Santosh Vempala of Georgia Tech and other researchers, delved into this issue in depth in a 36-page paper. The researchers suggest that hallucinations are not a flaw in the model design, but rather the way AI systems are tested and ranked.
Current evaluation methods reward a chatbot for answering every question, while penalizing models that remain silent when unsure. This can be likened to a multiple-choice exam that rewards a student for randomly answering questions rather than leaving them blank.
To address this issue, the statement proposes reversing the scoring system: Models should be heavily penalized for confident answers that are incorrect. Furthermore, models that exercise caution, express uncertainty, or say “I don’t know” should be rewarded.
The first examples demonstrating the difference of this new approach are quite revealing. In one test, a cautious model answers only half the questions but gets 74 percent of them correct. Another model answers almost all the questions but hallucinates three out of every four.
If this approach is adopted, the daily behavior of AI assistants could be completely transformed. Instead of fabricating sources or statistics, they would be more likely to give answers like “I’m not sure about this” or “I don’t know.”
While this might seem less impressive at first glance, it could provide a much more reliable experience by eliminating the need for users to constantly check their answers. For OpenAI, this research is a significant step toward an AI that values accuracy and trust, rather than grandiose but unreliable self-confidence.
