A group at Berkeley has just published (£) the first successful attempt to reconstruct colour video imagery from the mind using an fMRI brain scanner. The results are startling, encouraging and a little bit scary.
The method used, called fMRI, is known for its high spatial resolution (3D imaging ability) but notoriously low temporal resolution (measurements with respect to time – effectively a slow shutter speed). In the past, this has been a barrier to research on the visual cortex because of the incredibly high rate of information processing in the visual system. However, scientists have recently developed a new MRI encoding method that allows for the modelling of brain activity in the visual system at a faster rate. In the experiment conducted by the Berkeley group, participants were shown 7,200 seconds of random colour video clips one time only, while their brains were scanned using the novel fMRI sequence. From these scans, researchers were able to create a “dictionary” of brain activity in the visual system.
After a dictionary of brain activity in the visual system was created, the participants watched a fresh unseen video from YouTube while undergoing a brain scan. This resulted in video outputs that resembled the new YouTube video shown to participants. The output appears as a collage of flickering pixels that reminds me of a cross between the paintings of prosopagnosia sufferer Chuck Close and the imagery in A Scanner Darkly.
The correlation between the videos shown to the participants and the output imagery in the collage-like videos (below) is pretty astounding especially when considering there is zero overlap between the clips used for calibration and the clips used to test the system.
The study authors suggest that the method used in this paper could eventually lead to the generation of video output from participants experiencing dreams or hallucinations. Watch this space! What was once a field reserved firmly for science fiction may fast be becoming a reality.
Listen to an NPR interview with the researchers here:
Nishimoto S, Vu AT, Naselaris T, Benjamini Y, Yu B, & Gallant JL (2011). Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies. Current biology : CB PMID: 21945275Follow Simon on Twitter, Facebook, Google+, RSS, or join the mailing list.
So, theres been a lot of chatter over recent weeks about Brain Computer Interfacing since Tan Le’s TED talk on the Emotiv which resulted in the emotiv site being overloaded with hits and the letters “B, C and I” becoming familiar to fellow geeks the world over. The talk is mostly a marketing pitch so as I’m currently (amateur that I may be) involved in some BCI research myself I figured I’d give a little explanation of whats going on here…
Brain Computer Interfacing is not a new discovery but is currently emerging as a practical option for clinical treatment of “locked in syndrome”. The claims made by Emotiv however that the company has made a discovery that allows BCI to be released on the consumer market for gaming purposes are therefore intriguing. A major problem for decoding the EEG signals that the BCI relies upon is that the electrical signal the brain gives off is very weak and must travel through both skull and tissue. Emotiv claims it’s “breakthrough was to create an algorithm that unfolds the cortex so that we can map the signals closer to it’s source and therefore making it capable of working across a mass population“.
This currently hasn’t really been achieved by the army of BCI researchers working in institutions around the world so if true it is a genuine breakthrough that could have positive implications for sufferers of many degenerate brain disease patients and paralysis patients as well of course as the shock and awe of gamers everywhere. This finding however hasn’t been published in any journals to date, this should ring a number of alarm bells.
The second eye brow raiser is that even relatively old EEG equipment costs tens of thousands of dollars and requires close contact between fine metal electrodes which must be carefully placed exactly on top of very precise areas of the scalp and the signal to be conducted with conductive gel in an intricate and time consuming process. Emotiv claim that their system is able to perform what appear to be better results than the state of the art in BCI with only dry electrodes and without any real care in the electrode placement.
Another problem with the portrayal of the emotive device is that it seems to operate very differently from traditional cursor control BCI systems. Cursor control generally requires motor imagery. This means imagining a movement in a part of your body which “lights up” a part of your brain and programming a computer to recognise when you are imagining this movement. This requires pinpointing signals from a part of the brain called the motor cortex…
Besides being relatively small, this problem is made worse by the fact that the nerves that send and receive motor information around the body are not evenly distributed. This if you have not seen it before is called a homunculus and is the widely regarded and long held model of how psychologists believe the motor system is distrbuted proportionally around the body (this is why contemporary BCIs revolve around imagined hand movement)…
And just for fun recent research has even suggested that a sensory homunculus may even look a bit more like this…
My sincere apologies for inflicting that last image on you. No comment on the implications of that with regard to BCI though, just no, final answer.
My point is that BCI systems as they are known today require us to imagine things that can be very hard to identify alone let alone program a computer to decode so the process as it stands is considerably more complicated than imagining a computer doing something and “poof” it happening. Unfortunately the signals given off when we think are far more complex than any TV or radio signal and can’t be decoded in the same way. I’m not saying it will forever be impossible just that development will take time, progress will be made in baby steps and we may well be far further behind the point in the developmental process than this video suggests we are. It would be a great shame for the importance of future research to be diminished by a popular perception that “someone else did all that years ago”. I really don’t want to throw sand on anyones bonfire I just believe you have the right to be well informed.
These three factors that I mentioned, in combination with a lack of any published work make myself and all of the experts I have spoken to, very reluctant to believe that this device will prove as effective as it is billed. The main concern of the critics to date is that the emotiv may actually use facial muscle movements rather than brain waves to provide the minimal control it allows. This would make the headset a rather obscure and ineffective way of picking up these signals (which would be EMG signals). One professor of Engineering working within the BCI field who I won’t name has described such devices as “Head Computer Interfaces rather than Brain Computer Interfaces”. We won’t know either way until someone does a controlled study and makes the results public. If the critics are wrong however this will come to be seen as a commercial breakthrough of epic proportions.
Regardless of whether the discoveries made by Emotiv are as grand as they sound or not the team at Emotiv certainly deserve credit for their work so far, their product is a step towards the hopes and dreams of all that have stepped foot in the field and we are all looking forward to seeing some published results by the company and hearing how this research develops.
Until then however, I won’t be splashing out the the $500 (just slashed from $2500) for the research edition of the product and nor will I be buying the somewhat cheaper consumer model just yet. I will however be keeping a close eye on the progress of this product as it does seem to be the most advanced product in a field just gagging for it’s bubble to burst.
For the record, I am only just beginning to dip my toe in to the deep dark lake that is this field so BIG respect to Emotiv for what they have achieved and all the other people working on mind bending projects like this. If I have learned one thing from seeing inside this field of research it is that it is going to be a very hard nut to crack.
If you are interested in this product this is their site and there is an interesting though now somewhat dead public google wave for users and developers of the emotiv product here.Follow Simon on Twitter, Facebook, Google+, RSS, or join the mailing list.
Cookie ComplianceThis site contains cookies. If you have ever used the internet before then you probably knew that already and ate them long before you arrived here. If you are allergic to cookies please leave now.