While large language models like GPT-3 have made remarkable advances, they still lack critical capabilities needed for true autonomy according to Yann LeCun, Meta’s Chief AI Scientist. In a recent Forbes interview, LeCun advocated for open-sourcing these massive models while outlining his vision for imbuing them with real-world understanding.

LeCun made it clear that current large language models (LLMs) have severe limitations.

“They don’t understand the world, they don’t have persistent memory, they can’t really reason, they can’t really plan either,” said LeCun, believing fundamentally new architectures are required to create AI systems that can truly comprehend the physical world, retain memories, reason from first principles, and plan actionable sequences.

The missing ingredient, according to LeCun, is enabling AI to learn from observations and experiences just as humans do.

“The missing parts is how do we get AI systems to understand the world by watching it a little bit like baby animals and humans,” he said. “It takes a very long time for baby humans to really understand how the world works, like the whole idea of the fact that an object that is not supported falls because of gravity, takes nine months for human babies to learn this.”

LeCun’s proposed solution is an approach called Vision-Joined Embedding Predictive Architecture (VJEA), which aims to train systems to build predictive “mental world models” by observing videos of the physical world.

“VJEA is a particular instance of this kind of thing and the results are very promising,” he began to explain. “I think ultimately we’re going to be able to build or train systems that basically have mental world models, have some notion of intuitive physics, have some possibility of predicting what’s going to happen in the world as a result of taking an action for example.”

Combining massive scale, predictive world models, and the open-sourcing philosophy could lead to autonomous AI according to LeCun. However, he cautioned that we are still missing critical components beyond just scale: “It’s clear that we’re missing a number of things. The problem is that we don’t exactly know what and we can see the first obstacle really but like where is that going afterward is not clear.”

While arduous hurdles remain, LeCun expressed optimism that this approach could eventually produce systems with common sense reasoning abilities akin to animals, a pivotal milestone on the path to artificial general intelligence (AGI).

Featured image: Credit: Forbes