The news and AI don’t mix, research indicates

New data from the BBC won’t help anyone with trust issues.

The British news org recently gave four prominent AI chatbots – OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini, and Perplexity – access to its site, then asked the bots questions about the news, prompting them to use BBC articles as sources where possible.

Its findings:

• 19% of AI answers that cited BBC content included factual errors.
• 13% of quotes sourced from BBC articles were either altered or didn’t actually exist in that article.
• In total, 51% of all AI answers to questions about the news were judged to have significant issues of some form.

AI inaccuracy is a persistent problem

Even though it’s impossible for them to consume LSD, generative AI models have a tendency to hallucinate – or present incorrect or misleading results with the confidence of Neil deGrasse Tyson.

And research indicates this may be more of a feature than a bug. A July 2024 study from Cornell concluded that no matter how advanced language models become, hallucinations are inevitable.

🤔 But…Solutions may exist. One such potential fix is called retrieval augmented generation, which the Wall Street Journal compares to looking through a library of photos from the past year before writing a holiday letter instead of writing it all from memory.

Another potential fix? Teaching the models to say: “I don’t know.”