Intelligence May Arise Naturally from Thermodynamics
Agentic AI Edition #5 - Part 2
I stumbled across an interesting rabbit hole in the waning hours of 2023: the idea that intelligence arises naturally as a result of a generalization of the 2nd law of thermodynamics.
I wouldn't have paid much attention to this idea, except that I noticed it popping-up in multiple lines of research: Verses AI and Extropic. Also...the success of GPT-based LLMs is validation of the concept.
Extropic
In a recent appearance on the Lex Fridman podcast, Guillaume Verdon, founder of Extropic, explains his theory for how intelligence evolved on earth.
He proposes that biological systems are pockets of complexity that develop because they are more efficient at consuming free energy and increasing the overall entropy in the universe. We consume sources of energy voraciously, converting them into heat, accelerating the universe's trajectory toward overall entropy. So intelligence is complexity at the local scale but generates more entropy at the larger scale, conforming with the 2nd law of thermodynamics.
He founded Extropic to create other forms of intelligence based on thermodynamic principles, stating that they can achieve efficiency somewhere between classical and quantum computers (and easier to build than quantum computers). He doesn't get into any more detail, but it's an interesting concept.
Verses.ai
Verses AI is building AGI based on a similar concept--neuroscientist Karl Friston's idea that intelligence develops naturally in a system that seeks to consume free energy and produce equilibrium.
In the brain, achieving "cognitive equilibrium" can be thought of as "minimizing surprise". "Surprise" can be defined in this context as prediction error with respect to the system's next sensory inputs, so intelligence arises in systems that learn to accurately predict their own sensory inputs. No need for arbitrary objective functions like "pleasure" or "meaning". Since "surprise" is negatively correlated with survival, it makes sense that biological systems would minimize it, since those systems develop via natural selection.
Intelligence in GPTs
Friston's theory is validated by the recent success of GPT-based LLMs. LLMs learn by predicting the next word/token in a sequence of text, which is their "next sensory input". Learning to minimize "surprise" in this input-prediction task gives the model the ability to answer human questions in an intelligent way.
Many suggest that next-word prediction is a "dumb" objective function and that we only use it because it works on GPUs. But if these theories are correct, they get us closer to a "first principles" explanation of why predicting the next token might be exactly the correct objective function for creating intelligence.
The theories also suggest that we should be able to increase the intelligence in AI systems by giving them more sensory input and training them to predict those sensory inputs accurately (e.g. video as input, predicting the next frame of video).
Further Research
I look forward to exploring these concepts further. If you’re interesting in reading on your own, here are some scientists behind the ideas:
Karl Friston
Jeremy England
Nick Lane
Harold Morowitz