Insider Brief
- A study published in Nature Human Behaviour shows that the way large language models represent speech aligns closely with neural activity in the human brain during conversation.
- Researchers found that the internal “embeddings” used by OpenAI’s Whisper model to represent words correspond to brain activity patterns recorded from epilepsy patients during natural dialogue.
- Despite architectural differences, both human brains and language models process speech by sequentially transforming sound into meaning, suggesting shared computational principles.
Neural activity in the human brain during conversation mirrors the way large language models represent speech, according to a new study by Google Research.
Published in Nature Human Behaviour, the study found that internal “embeddings” — numerical representations used by AI models to process language — closely align with brain activity recorded during everyday conversations. The research suggests that artificial language models trained to predict the next word in a sentence may use similar computational principles to those used by the human brain.
In experiments conducted with researchers from Princeton University, NYU, and the Hebrew University of Jerusalem, the team recorded neural activity from people engaged in spontaneous conversations using intracranial electrodes. They then compared this brain activity to the internal representations generated by Whisper, an open-source speech-to-text model developed by OpenAI. These internal representations, or embeddings, reflect the model’s understanding of each word based on the surrounding context.
“All in all, our findings suggest that the speech-to-text model embeddings provide a cohesive framework for understanding the neural basis of processing language during natural conversations,” the Google Research team wrote in the post accompanying the study.
The researchers emphasized that Whisper was not designed with human brain function in mind — yet its internal workings map surprisingly well onto human neural responses.
Speech Comprehension Process
The team found that during speech comprehension, the brain first shows activity in regions responsible for processing sound — such as the superior temporal gyrus (STG) — which aligns with the model’s speech embeddings. A few hundred milliseconds later, activity shifts to Broca’s area — a region known for language processing — which aligns with the model’s word-level embeddings. This sequencing mirrors how the brain goes from hearing a sound to understanding its meaning.
The pattern reverses during speech production. When a person prepares to speak, Broca’s area activates first, aligning with the model’s language embeddings. This is followed by motor cortex activity — responsible for articulation — and then STG activity as the speaker hears themselves talk.
For each word in the conversation, the researchers used a simple linear transformation to map the model’s embeddings onto the brain’s electrical signals, both before and after the word was spoken. The correlation held across subjects and multiple electrodes, providing strong evidence of alignment between machine learning representations and human cognitive processes.
“This alignment was not guaranteed — a negative result would have shown little to no correspondence between the embeddings and neural signals, indicating that the model’s representations did not capture the brain’s language processing mechanisms,” the researchers write.
Builds on Earlier Work
The study builds on earlier work from the same team, including findings published in Nature Neuroscience and Nature Communications. In those papers, the researchers showed that the human brain also tries to predict the next word before it is heard, similar to how AI models work. The level of surprise or error after hearing the word is also similar in both systems, depending on how confident the prediction was.
These shared behaviors point to what the researchers call a “soft hierarchy” in the brain’s language processing regions. While higher-order areas like Broca’s region prioritize word meanings, they also show some sensitivity to sound-level information. Conversely, speech-processing areas like the STG — typically focused on the sound of speech — also encode some word-level meaning.
Despite the strong parallels, the researchers emphasized that human and artificial systems differ in key ways. Unlike Transformer-based language models, which can process thousands of words in parallel, the human brain processes language one word at a time, in a more sequential and context-sensitive manner.
“For example, in a follow-up study, we investigated how information is processed across layers in Transformer-based LLMs compared to the human brain,” the team writes. “The team found that while the non-linear transformations across layers are similar in LLMs and language areas in the human brain, the implementations differ significantly. Unlike the Transformer architecture, which processes hundreds to thousands of words simultaneously, the language areas appear to analyze language serially, word by word, recurrently, and temporally.”
Human brains also acquire language through years of lived experience and social interaction, unlike AI systems trained on massive text datasets in isolated environments. These differences raise questions about how closely artificial models can truly mimic biological cognition.
Potential For Improved AI Models?
Still, the shared architecture offers a new way to study how language is encoded in the brain — and potentially to improve AI models by drawing on biology.
The team writes: “Moving forward, our goal is to create innovative, biologically inspired artificial neural networks that have improved capabilities for processing information and functioning in the real world. We plan to achieve this by adapting neural architecture, learning protocols and training data that better match human experiences.”
Direct Recordings of Brain Activity
The study’s methods relied on rare and direct recordings of brain activity from epilepsy patients who had electrodes implanted for clinical reasons. While powerful, this type of data is difficult to collect and limited in scope, which constrains how broadly the findings can be generalized.
Future work will likely expand these studies to include more diverse subjects and natural settings. Researchers are also exploring whether adjusting AI model architectures and training data to better match human experience can improve both the models’ accuracy and their alignment with the brain.
The findings reflect a five-year collaboration between Google Research and multiple academic institutions, including the Hasson Lab at Princeton’s Neuroscience Institute and Psychology Department, the DeepCognitionLab at the Hebrew University, and the NYU Langone Comprehensive Epilepsy Center.
