Insider Brief
- A University of Kansas study funded by internal academic support and published in PLOS One used psycholinguistic methods to test how ChatGPT processes nonwords, revealing differences in pattern recognition between human and AI language processing.
- The research found ChatGPT correctly defined 36 of 52 extinct English words, identified 11 as unfamiliar, hallucinated 2 definitions, and in some cases drew from foreign languages, underscoring how AI can both mimic and deviate from human understanding.
- In additional experiments, ChatGPT struggled with phonological similarity tasks and coined novel terms by combining familiar English words, highlighting both its linguistic creativity and limitations in replicating nuanced human responses.
PRESS RELEASE -— A new study appearing in the peer-reviewed journal PLOS One from a psycholinguist at the University of Kansas explores how ChatGPT, the popular artificial-intelligence chatbot, responded to nonwords.
Nonwords are meaningless letters and sounds used in cognitive psychology to explore how humans learn, remember and process language.
Author Michael Vitevitch, professor in the Speech-Language-Hearing Department at KU, wanted to know how the “black box” of ChatGPT might react to similar experimental prompts.
“As a psycholinguist, one of the things I’ve done in the past is to give people nonsense to see how they respond to it — nonsense that’s specially designed to get an understanding of what they know,” he said. “I’ve tried to use methods we use with people to appreciate how they’re doing what they’re doing — and to do the same thing with AI to see how it’s doing what it’s doing.”
By feeding nonsense to ChatGPT, Vitevitch found it was quite good at discovering relationships between things.
“It finds patterns, but not necessarily the same patterns that a human would use to do the same task,” he said. “We do things very differently from how AI does things. That’s an important point. It’s okay that we do things differently. And the things that we need help with, that’s where we should engineer AI to give us a safety net — to help us on those things.”
In one part of the study, ChatGPT showed it could provide correct definitions to many actual English words that are no longer in use. One example is “upknocking,” an occupation from the 1800s that involved waking people up, now made obsolete by alarm clocks.
As these words are no longer part of English, researchers consider them “nonwords.”
The results are illustrative of the chatbot’s internal workings. Of 52 carefully chosen extinct words, ChatGPT responded with 36 correct definitions. For 11 words, ChatGPT responded that it couldn’t recognize the word. For three words, the chatbot gave definitions gleaned from words in foreign languages.
For two words, ChatGPT appears to have made up answers, giving so-called “hallucinations” in response to the word-definition requests.
“It did hallucinate on a couple of things,” Vitevitch said. “We asked it to define these extinct words. It got a good number of them right. On another bunch, it said, ‘Yeah, I don’t know what this is. This is an odd word or very rare word that’s not used anymore.’ But then, on a couple, it made stuff up. I guess it was trying to be helpful.”
In another part of the experiment, ChatGPT was given a set of Spanish words and asked to provide similar-sounding words from English. The task is much like the “phonological associate task” used with English speakers to gauge aspects of phonological similarity.
Vitevitch said the experiment highlighted the difference in how humans and AI process language.
“If I give you a Spanish word and tell you to give me a word that sounds like it, you, as an English speaker, would give me an English word that sounds like that thing,” he explained. “You wouldn’t switch languages on me and just kind of give me something from a completely different language, which is what ChatGPT did.
“The only time people switch languages like that is when multilingual parents who didn’t teach their kids any language besides English want to talk about something without the kids understanding.”
In a third aspect of the study, nonwords were designed to resemble real words in English. Chat GPT then was asked to rate the words on a scale from one for a “Bad English word” up to seven for a “Good English word” — its assessment of how much the nonword sounded like a word in English. ChatGPT’s ratings then were compared against those given by human English speakers.
Lastly, the KU researcher prompted Chat GPT to create new English words that refer to a novel concept — much like the “sniglets” popularized in the 1980s by comedian Rich Hall on HBO’s “Not Necessarily the News.”
“He used to do ‘sniglets,’ which were words that don’t exist,” said Vitevitch. “Like with a vacuum cleaner, when there’s a thread on the floor and you go over it and it doesn’t get sucked up. So, you go over it — again and again. What is that thread called? ‘Carperpetuation.’ He came up with a name for that thread that doesn’t get sucked up. So, we have many things that should have a word, but we don’t have one in English.”
According to Vitevitch, the AI chat bot did “kind of an interesting job there.” After prompting ChatGPT for new words that matched certain concepts, he found it often relied on a predictable method of combining two words.
“My favorite was ‘rousrage,’ for anger expressed upon being woken,” said Vitevitch.
Other ChatGPT word coinages from the research include:
“Prideify:” taking pride in someone else’s achievement
“Lexinize:” the process by which a nonword begins to take on meaning
“Stumblop:” to trip over one’s own feet
By talking nonsense to ChatGPT, Vitevitch hopes to better understand when AI might be able to help humans in certain language tasks rather than simply duplicate what humans can already do quite well.
