Brain-IT: Israeli AI Model Reconstructs Images from Human Brain Activity
Developed at Israel’s Weizmann Institute of Science, Brain-IT is an artificial intelligence model capable of reconstructing images seen by a person based solely on their brain activity. The breakthrough significantly improves the accuracy of brain decoding and opens up new possibilities for neuroscience research.
A baseball stadium packed with spectators, a ball flying through the air and a player getting ready to hit it. Based solely on the brain activity of someone viewing this scene, an artificial intelligence system can generate a visual representation of what they are seeing. The system, known as Brain-IT, was developed by a team led by Professor Michal Irani in the Department of Computer Science and Applied Mathematics at the Weizmann Institute of Science.
Unlike models that require many hours of training using an individual's brain data, Brain-IT relies on patterns of brain activity shared across different people. As a result, it can learn to decode images seen by a new individual using just one hour of brain imaging data, according to its developers.
More Accurate Image Reconstruction
In recent years, advances in artificial intelligence have made it possible to decode certain brain signals and translate them into images or language. Experimental technologies have also helped people with severe paralysis communicate by converting their brain activity into speech.
However, these systems are generally tailored to individual users and require large amounts of data to function effectively.
Brain-IT aims to overcome this limitation. Rather than simply identifying the general meaning of an image, the model also improves the reconstruction of its visual features, including composition and colors.
"The new model we have developed outperforms other systems in reconstructing image content and details," explains Michal Irani.
Turning a Lack of Data into an Advantage
Training an artificial intelligence system to reconstruct images from brain signals requires thousands of images paired with functional magnetic resonance imaging (fMRI) data. This technique indirectly measures brain activity by tracking changes in blood flow.
However, collecting such data is both time-consuming and expensive. Participants must spend extended periods inside MRI scanners, meaning that the datasets available in this field remain relatively limited.
Michal Irani's team developed an innovative approach to address this challenge by creating two complementary models: a decoder and an encoder.
The decoder learns to reconstruct an image from a brain signal. The encoder performs the opposite task, predicting the brain activity that a particular image would generate.
The researchers used this system to generate estimated brain data from images that had never been shown to participants inside an MRI scanner. These synthetic data were then used to train the decoder to reconstruct the original images.
This method significantly expands the amount of training data available without requiring additional participants to undergo lengthy brain-imaging sessions.
An Unprecedented Map of the Human Brain
One of the most intriguing findings of the research concerns the understanding of how the human brain works.
To account for differences between individuals, the researchers divided the brain into approximately 40,000 voxels, small three-dimensional units of volume. They examined how these units responded to different image features, including colors, object locations and semantic meaning.
By combining these measurements with the analytical capabilities of an artificial intelligence model, the team identified recurring patterns of brain activity shared across different individuals.
During this process, the encoder identified 128 shared functional regions associated with specific roles in image processing. Some correspond to areas already known to neuroscientists, while others reveal functional distinctions that have received little attention until now.
Among their findings, the researchers observed a division of labor within the parahippocampal place area (PPA), a brain region involved in processing scenes and locations. One part responds more strongly to indoor environments, while another is more responsive to outdoor scenes.
The approach could also provide new insights into the mechanisms of human perception. By comparing, for example, the predicted brain response to the same image in color and in black and white, scientists can investigate which regions are involved in processing color information.
Could We One Day Decode Dreams?
Selected for presentation at the International Conference on Learning Representations (ICLR), one of the leading scientific conferences in machine learning, Brain-IT highlights Israel's growing contribution to research at the intersection of artificial intelligence and neuroscience.
The team is now working to extend its methods to the decoding of auditory information. Video represents another major scientific frontier, but it presents a significant challenge: images change dozens of times per second, whereas an fMRI acquisition takes approximately two seconds.
This gap between the speed of brain activity and the speed of measurement makes reconstructing dynamic sequences particularly difficult.
For Michal Irani, overcoming these obstacles could eventually pave the way for even more ambitious applications, including studying brain activity during sleep.
Reading dreams, however, remains a distant prospect rather than a current capability of Brain-IT. At present, the model reconstructs static images from brain activity measured under experimental conditions. It cannot freely read a person's thoughts or reproduce their memories with perfect accuracy.
The next step will be to understand how brain signals evolve over time and how different regions of the brain work together to represent complex scenes. In the longer term, this research could deepen our understanding of human perception and open up new possibilities for assistive technologies designed to help people who have lost the ability to communicate.
Caroline Haïat
Comments