One of my all time favourite bloggers, Oxford Neuroscientist Prof. Dorothy Bishop, or DeevyBee as she is known on Twitter has performed an amazing open access lecture focusing largely on the misunderstanding of neuroscience (click down to the “Emanuel Miller Lecture” to play the video). The talk is incredibly informative and digestible, even those with no understanding of neuroscience or psychology whatsoever will take a great deal away. The problem of the poor understanding of neuroscience is one of the main reasons why I started this blog, so if you like this blog then you’ll love this lecture.
The talk begins with a reasoned explanation of how and when we should be sceptical of neuroscience research, Bishop goes on to cite 4 key reasons why certain kinds of scientific research will inevitbaly result in false-positives:
“The four horseman of the apocalypse”
1. Maturation – People develop naturally over time.
“There seems to be an implicit assumption that the brain, because it is a physical organ is somehow not going to change unless you give it some intervention – that it is there as a static thing. This is completely untrue… as evidenced by this series of images.”
2. Practice effects – when people keep doing the same test again and again, they get better at it.
“…purely to do with the fact that you have got better at doing the test and nothing to do with your abilities… People forget that this can apply to language tests and thing like that. It also applies to some extent to the brain, often we don’t know how important this is because brain imaging is so new.. clearly if you get brain responses to novelty, that means if you do something twice – the first time round you will get different responses to the second time round when it is no longer novel”.
3. Regression to the mean – a statistical artefact of longitudinal studies that is exacerbated if you select participants on the basis of a low score on a test (for example participants with developmental difficulties). Bishop does an outstanding job of explaining the problem at about 18 minutes in to the talk.
“Regression to the mean is as inevitable as death and taxes”
Campbell and Kenny (1999) A primer on regression artefacts
4. The placebo effect. This is the obvious consideration that continues to impact poorly designed research but according to Bishop, the three issues listed above could actually be having an even greater impact than the placebo effect.
Bishop explains that a control group is vital in order to achieve valid findings, but a control group alone is not enough, we should also be asking questions such as:
- Are the groups randomly assigned – or is there some other factor at play?
- Is the control group given an alternative treatment? If not, why not?
- What causes drop out? People don’t tend to drop out at random and this can have a very big effect on results.
Weisberg, Deena Skolnick. (2008). The Seductive Allure of Neuroscience Explanations. Journal of Cognitive Neuroscience, 18 (3), 229-477 DOI: 10.1162/jocn.2008.20040
Campbell and Kenny (1999) A primer on regression artefactsFollow Neurobonkers on Twitter, Facebook, Google+, RSS, or join the mailing list.
A wise man named Karl Popper once disagreed with the orthodox view that scientific activity starts with observation noting that “observation is always selective”. It is often possible to propose a theory and find results that support or verify it but Popper proposed that it is the negative results that are of crucial value. He used the following simple thought experiment to prove it.
Europeans for thousands of years had observed millions of white swans. Using inductive evidence, we could come up with the theory that all swans are white.
However exploration of Australasia introduced Europeans to black swans. No matter how many observations are made which confirm the theory that swans are white there is always the possibility that a future observation could refute it. Induction cannot yield certainty.
This simple principle revolutionised science and the world lived happily ever after.
…Of course that was not how the story ended. Today there is still massive pressure on scientists by industry, funding bodies, the media and universities to chase after verifying positive findings and in effect supporting established knowledge. This is all very well but when a negative finding is discovered it is all too often brushed under the carpet, almost as if it is an embarrassed (an ugly duckling perhaps). For example consider for a second how truly horrifying the following statistic actually is:
Of that 5.9%, an astounding 75% give positive results – suggesting negative findings are simply not published. This is not the only problem for academics searching for negative results to support a proposition. Because of the way boolean search algorithms work you have to have a little bit of boolean-know-how to actually search for a negative result. Simply adding “not” in to an english language proposition will still yield positive findings. Now a group of scientists have created a database called BioNot that uses data mining and intelligent machine learning methods to systematically search for negative results in Pubmed and Elsevier.
You may wish to write down the URL http://snake.ims.uwm.edu/bionot/ because in an ironic twist the boolean wizards who created this programme have made the URL a little trixy for google to find, presumably because this website is google’s arch nemesis. Perhaps not, but either way if you hit “BioNot” in to google you end up with something to do with nuts.
Anyway, I’m off to have a play with it, if you need some search ideas, check out the fabulous results of my very first search!
Via Neuroskeptic (where you can pop on over and read a little more explanation of the potential applications of this tool)
Agarwal S, Yu H, & Kohane I (2011). BioNOT: A searchable database of biomedical negated sentences. BMC bioinformatics, 12 (1) PMID: 22032181Follow Neurobonkers on Twitter, Facebook, Google+, RSS, or join the mailing list.
UPDATE 06/12/11: THE INDEPENDENT HAVE NOW AMMENDED THEIR ARTICLE FOLLOWING A PCC COMPLAINT (THEY STILL FAIL TO REFERENCE THE FACT THAT THE DRUG IN QUESTION IS ALMOST CERTAINLY “VALIUM”.)
UPDATE 05/12/11: THE METRO HAVE NOW REMOVED THEIR ARTICLE FOLLOWING A PCC COMPLAINT.
Africa America Bad Science BCI Brain Computer Interfacing breaking news Cannabis Censorship Cocaine Copyright Counterfeit Drugs Daily Fail DailyFail daily mail Daily Mail Demolition Squad Drugs EEG Emotiv Fake Drugs FMRI Health Hoax Independent Misinformation Music Neuroscience Open Science Procrastination Psychology Rat Brain Robot Review Satire Science sex Skepticism Statistics Student Loans Crisis Susan Greenfield Synaesthesia Technology The confederacy of dunces Video walking War on Drugs Wikileaks
Cookie ComplianceThis site contains cookies dropped by Facebook, Twitter, Google Plus and Google Analytics. If you have ever used the internet before then you probably knew that already and ate them long before you arrived here.