A ‘hallucinating’ research assistant?
I’m bullish on the potential for generative AI. I think newsrooms and academic researchers alike should be aggressively exploring the transformative potential of large language models (LLMs). So, for the past three weeks I’ve been experimenting with various applications built on OpenAI’s GPT models to see whether it can speed up my research. My research topic is misinformation — specifically looking into how journalists can contribute to at-scale AI systems that detect misinformation. Like many people with more expertise than me, I’m concerned that the ability to generate authoritative-seeming nonsense at scale could help bad actors spread mistrust and confusion.
This week, I jumped on the latest shiny new thing in generative AI: using an autonomous agent to independently achieve a multi-step goal. AgentGPT is a browser-based tool that anyone with an OpenAI API key can make use of at minimal cost (try it yourself with these instructions). You give your AI agent a high-level goal and it autonomously breaks this goal down into steps, then executes these steps one-by-one while showing its homework.
Creating a reading list of research on AI and misinformation seemed like a perfect task for my new research assistant. GPT 3.5 (the model I had access to) was trained at the end of 2021, so the list would be missing new research, but I figured it would likely turn up some relevant readings, as both AI and misinformation are fast-growing fields.
I entered the following prompt into AgentGPT: ‘Create a list of academic papers to support a literature review about how journalists and other human experts can contribute to at-scale AI systems that detect misinformation and disinformation. Ideally these academic papers have high numbers of citations, but very specialized papers dealing with this topic are also acceptable.’
It immediately broke this prompt down into three sub-tasks:
- Identify academic databases likely to have relevant papers.
- Develop a search strategy to identify papers relevant to the research question.
- Analyse the selected papers to determine relevance, reliability and quality.
So far, so impressive — after transparently running through these steps and ‘conducting extensive research’, AgentGPT presented me with a list of five papers that were directly relevant to my topic. So relevant, in fact, that I was both excited, and a little worried that I may not be producing any original research.
Using Google Scholar, I immediately searched for the first reference: ‘The Need for Human-Centered AI in Misinformation Detection’ by S. T. Gao, E. M. Bender, and H. Wallach (2020). There was no record of it there, or elsewhere, I searched each database AgentGPT had identified, and our own UTS Library website, but to no avail.
So, I turned my attention to the author’s names. ‘E.M. Bender’ is Professor Emily M. Bender, a linguistics expert who writes extensively on AI. She was the co-author of a paper ‘On the Dangers of Stochastic Parrots’, which was quoted in a widely shared letter that called for a pause in all AI research and was signed by Elon Musk and AI pioneer Yoshua Bengio, among others. Professor Bender’s publications were listed on her website — ‘The Need for Human-Centred AI in Misinformation Detection ’ was not among them.
My automated research assistant was ‘hallucinating’ — in other words, the AI made it up.
The Guardian describes the same problem in this fascinating article, where GPT created a fake reference to a Guardian article, but it was convincing enough that the author of the piece felt they might have forgotten writing it and therefore decided to dig it up.
What lesson can we take from this? Warnings not to blindly trust GPT’s output should be taken seriously. LLMs are not truth-producing machines, but rather incredible language guessing machines. These references were immediately convincing when I saw them as the authors were actual academics, and the titles were written in the descriptive, scientific tone associated with journal articles. I thought I’d hit the jackpot.
More than 100 million users are already registered with ChatGPT. If generative AI driven by LLMs becomes a core information-discovery activity among billions of people, two actions seem vital: companies that create these models must aggressively reduce hallucinations, and that the public is provided with education and transparency about the nature of this technology and its relationship with the truth.
It’s important to resist the urge to focus only on the negatives, and instead think about solutions and opportunities. Microsoft’s Bing now has a chatbot that fuses GPT with its vetted search database. When I tried the same prompt in Bing using its ‘Precise’ mode, I received only genuine (and far less exciting) references with links out to each source. By combining GPT and other LLMs with trustworthy data, the generative power of this new technology can potentially be harnessed with more respect for the truth. What role could media companies and their (generally) reliable, fact-checked content play in this new world? Is there revenue opportunity, or even the chance to build new products, such as AI chatbots that answer questions based only on authoritative data? For now, though, let’s stick to research assistants who only hallucinate when they’re off-the-clock.
Shaun Davies — FASS Masters student
