Over at the Chronicle of Higher Education, Tom Bartlett offers up a rather interesting tale of the travails of two psychologists and a replication study:

Back in January, a paper published in the Journal of Personality and Social Psychology appeared to prove that ESP is real, that in certain circumstances (involving, as it happens, erotic pictures) people really can predict the future. Naturally, this got more attention than your average academic publication. At the time I talked to the author of the paper, Daryl J. Bem, who was reeling from all the media attention.

To summarise, 12 months on from the publication of Bem’s rather notorious paper, Bartlett went looking for replication studies for, one presumes, a follow-up story and unearthed an illuminating example of the shortcomings of scientific publishing, courtesy of Stewart Ritchie, a PhD student at the University of Edinburgh:

Here’s the story: we sent the paper to the journal that Bem published his paper in, and they said ‘no, we don’t ever accept straight replication attempts’.

Ever?

Once upon a time, back in the days before teh Interwebs, that might just about have been a defensible position. After all, there’s a cost involved in committing articles to print.

But today? When articles can be readily be stored and distibuted via the internet at a fraction of a penny/cent per page view or download?

Even if a journal wishes to stick steadfastly to the ‘novel studies only’ rule for its print edition, there is nothing to prevent them accomodating replication studies in a web-only edition or supplement, so this whole ‘we don’t ever accept straight replication attempts’ just doesn’t fly.

We then tried another couple of journals, who said the same thing.

Really?

Maybe that’s their policy – or maybe the real issue here lies in the fact that Ritchie and his colleagues replicated Bem’s study but failed to replicate his positive findings – given the amount of interest that Bem’s original paper attracted, one can’t help but wonder whether these journals might well have taken a very different view of their ‘no replication studies’ rule had Ritchie’s study provided evidence to support Bem’s original findings.

That said, our intrepid researchers did find one journal that was willing to step up to the plate and, at the very least, accept their paper for review:

We then sent it to the British Journal of Psychology, who sent it out for review. For whatever reason (and they have apologised, to their credit), it was quite badly delayed in their review process, and they took many months to get back to us.

Despite the delays, that’s a yay for the BJP, at least to begin with.

When they did get back to us, there were two reviews, one very positive, urging publication, and one quite negative. This latter review didn’t find any problems in our methodology or writeup itself, but suggested that, since the three of us (Richard Wiseman, Chris French and I) are all skeptical of ESP, we might have unconsciously influenced the results using our own psychic powers.

Oh dear. Someone’s not happy because the researchers forgot to wear their tinfoil hats while conducting the experiment.

Seriously, if a reviewer comes back with a critique based entire on an unfalsifiable hypothesis then its time to bin the review and get someone who can actually provide a scientific critique.

Actually, there is a little bit more to this, in terms of a back story which provides a pretty useful insight into the general state of psi research.

The story behind this is that Richard has co-authored two papers where he and a believer in psi both did the same experiment, and the believer found positive results but he didn’t. However, the most recent time they did this – which was the best-controlled and largest size – neither found results. This doesn’t exactly give hugely compelling evidence for an ‘experimenter effect’ in psi research, in our opinion. Here’s that last paper, if you’re interested.

Richard’s paper is well worth a read but to summarise it tells a very familiar story in which a psi effect is apparently found by the researcher who believes in psychic powers when small scale, statistically underpowered, studies are conducted, only for the effect to evaporate once you include enough people in the study to reduce the possibility of getting false positive by chance to a statistically reasonable level:

The basic experimental procedure that has evolved involves the participant and experimenter being located in two separate, sensorially-isolated rooms. A closed-circuit television system feeds a live image of the participant to a monitor in the experimenter’s room and, at randomly determined times, the experimenter either stares at this image with the intention of physiologically arousing the participant (‘stare’ trials) or looks away from the monitor and disengages his/her intention (‘no-stare’ trials). The participant’s EDA is continuously recorded during the experiment and any significant differential effects observed in EDA between ‘stare’ and ‘no-stare’ trials is inferred to reflect the existence of psychic functioning. A recent meta-analytic review of 15 experiments using these types of procedures revealed a small, but statistically significant, overall effect (Schmidt, Schneider, Utts, & Walach, 2004).

EDA, here, means ‘Electrodermal Activity’ or, more commonly, skin conductivity, so the ‘psychic ability’ that’s being tested here is the experimenter’s ability to stare at test subjects on a monitor and remotely induce them to sweat… a little bit. Professor Xavier, it ain’t!

Wolverine: So kid, what’s your mutant name…?

Kid: Perspira.

When you’re reduced to looking for evidence of scientifically implausible abilities in trivial and extremely marginal effects then maybe its time to conclude that you’re just chasing shadows and you should go find yourself a real phenomenon to study, but even amongst scientists, belief can still retain its unmatched ability to blind some people to the facts that are staring them in the face.

Conventional parapsychology, i.e. looking for the existence of psychic abilities, is a busted flush. What’s interesting here, and well worth studying, is not whether psychic abilities exist – they don’t – but why people cling to their belief in psychic powers, etc. As an adjunct to the wider study of cognitive biases and heuristics there is still quite a bit of mileage left in the field of parapsychology. The study of human irrationality remains worth pursuing, not least because there’s so much of it around to look at and understand.

Sadly, the editor of the BJP didn’t quite see things that way:

Anyway, the BJP editor agreed with the second reviewer, and said that he’d only accept our paper if we ran a fourth experiment where we got a believer to run all the participants, to control for these experimenter effects. We thought that was a bit silly, and said that to the editor, but he didn’t change his mind.

We don’t think doing another replication with a believer at the helm is the right thing to do, for the reason above, and for the reason that Bem had stated in his original paper that his experimental paradigms were designed so that most of the work is done by a computer and the experimenter has very little to do (this was explicitly because of his concerns about possible experimenter effects). So, after this very long and unproductive delay, we’re off to another journal to try again. How frustrating.

Quite. Bem’s original study was designed specifically to eliminate the possibility of experimenter effects influencing the outcome, so what the editor of the BJP has, in effect, fallen for is the old canard, and far too convenient, about psychic powers not working when there’s a skeptic in the room, as if to suggest that there is something about skeptics which causes them to unconsciously emit a psychic damping field which magically interferes with the so called psychic’s special abilities.

Nowhere in any of this, of course, is there the merest hint of a plausible hypothesis to account for the underlying mechanism behind any of these supposed phenomena. Most so-called psychics rely on wholly unscientific and unfalsifiable ‘spiritual’ explanations for their claimed abilities although, if pushed, some will beat a hasty retreat to the ‘I don’t know why it works, I just know it does’ fall back position, as if that explains anything.

Yes, there are indeed some things that we know work very effectively despite the fact that we don’t have the foggiest idea about how they actually work – anaesthesia is a very good example. But even if we don’t yet understand the biochemical mechanism behind anaesthesia, one thing we can be sure of that such a mechanism does indeed exist. Anaesthesia works by introducing foreign chemicals into the body, either intravenously or by inhalation, so we can be sure that the effect we’re looking for is ultimately a matter of chemistry, or rather biochemistry, and we can even take a reasonable punt at suggesting that whatever the actual mechanism it seems likely to bear some relationship to the normal sleep mechanism or, perhaps even more likely, to a primal hibernation mechanism inherited from a very distant ancestral species.

How distant?

Well, anaesthesia has been found to work in invertebrates, including crustaceans, arthropods, arachnids and even nematodes, so we’re looking for an extremely primal mechanism, one which is likely to one of life’s oldest and most basic survival mechanisms, given its apparent ubiquity.

For anaesthesia we may not have the actual mechanism but we know that the mechanism must exist and can even construct a plausible evolutionary hypothesis for its origins, giving us something to work with. For psychic phenomenon we have neither and although we can speculate that certain psychic abilities would prove to be advantageous in evolutionary terms, and idea that Phillip K Dick explored in a typically idiosyncratic manner in his 1954 novelette ‘The Golden Man‘, its almost impossible to see where the evolutionary benefits of some of the ‘abilities’ studied by parapsychologists, like remote sweating, might lie.

There really is no good reason to consider psychic phenomenon to be scientifically plausible at any level in the absence of solid, reliable, empirical evidence for the existence of such phenomena, evidence that parapsychologist have signally failed to produce despite their best efforts over the last 120 years, or so*.

*History buffs might like to know that the term ‘parapsychology’ was coined in 1889 by the philosopher Max Dessoir, although it didn’t formally become part of the scientific lexicon until it was adopted by JB Rhine in the 1930’s as a replacement for the term ‘psychical research’.

If you are looking for any kind of attempt to account for psychic phenomenon is ‘scientific’ terms then, unfortunately, you’ll have to enter the realm of quantum woo, a field in which the Dunning-Kruger is rife and the primary hypothesis for which all other hypotheses are derived begins with ‘I don’t understand the Heisenberg Uncertainty Principle at all, therefore…”.

What some of the more imaginative fans of psychic superpowers have hit on is the apparent observer effect in the classic double-slit experiment which demonstrates the existence of wave-particle duality and, in particular, John von Neumann’s attempt to account for the process of wave function collapse by postulating the idea that its consciousness, itself, that causes the collapse.

This is one of those points at which quantum mechanics bleeds into metaphysics and, unfortunately, re-opens the door to dualism and as result its been cheerfully leapt on by a wide range of cranks, including Deepak Chopra, as a means of lending an umerited veneer of quasi-scientific respectability to their batshit insane wibblings even though, if you follow the reasoning, it ultimately leads you inexorably back to the old philosophical chestnut about whether trees falling in a forest miles away from the nearest observer actually make a sound.

All this presupposes that the Von Neumann interpretation is correct when it is only one of many extant interpretations and, equally, one that finds little support amongst physicists.

Even if you buy into the idea that consciousness causes collapse, none of this explains why it should be the case that a particular type of consciousness, i.e. non-skeptical vs skeptical, should necessarily cause things to ‘collapse’ in a particular direction, i.e. one which either produces or fails to produce evidence of the existence of a psychic ability. No matter how you try and dress it up in pseudoscience, there is still no concrete, reliable, explanation for the common, arse-covering, claim that psychic abilities somehow don’t work with a skeptic in the room.

Let’s suppose, for a moment, that you were to run Wiseman’s stare test again, only this time with two experimenters operating at the same time, one believer and one skeptic, each with their own independent monitor and measuring equipment set to record and report back the results of the test only to the relevant experimenter.

What would be the outcome, assuming that the quantum woo interpretation of the inability of psychics to generate positive results under rigorous test conditions is correct?

Would the two experimenters simply cancel each other out, giving a null outcome?

Would each generate a different set of results, one positive and one negative?

Would the believer still register an effect, albeit a smaller one than might have been the cause without a skeptic riding on the experiment – and if so, might the skeptic not also register a similarly small effect that they might not otherwise have seen?

Or, would the result just depend entirely on whether The Force is stronger with one experimenter than the other?

Who knows for sure, but even so, such an experiment would not provide any kind of plausible explanation for the inability of self-styled psychics to produce positive results under rigorous, scientific, test conditions.

Maybe we’re looking at this from the wrong direction?

Instead of trying to account for inability of psychics to get it up in the lab, so to speak, maybe we should be asking just exactly what it is about skeptics that’s blocking their efforts to perform?

That possibility has already been raised in terms of the idea that skeptics may be unconsciously influencing experiments with their own psychic powers, but as skeptics don’t belive in psychic powers anyway, that’s just not a satisfying explanation.

So what about something a little more scientific by way of a hypothesis?

For the sake of shits and giggles, lets assume that there could be something to the quantum woo interpretation and that skeptics are unwittingly interfering with psychic experiments at a sub-atomic level?

The question is how? What’s the mechanism and what do skeptics have that tips the balance towards the negative, that believers don’t.

Let’s imagine that there is an as yet unknown and undiscovered sub-atomic particle behind this phenomenon, one that skeptics emit without realising it, and that its this that directly interferes with and block psychic abilities at the sub-atomic level.

For the sake of brevity, lets call this particle the ‘Skeptrino’.

So how do we test for the existence, or otherwise, of Skeptrinos?

Well, to begin with, we need a test subject – say ‘Psychic’ Sally Morgan. She seems very popular and a lot of people seem to be sufficiently convinced of her abilities that makes a pretty decent living as a psychic.

Next, we need a good, reliable, source of Skeptrinos – how about Professor Brian Cox, he seems game and he knows his way around the sub atomic realm pretty well as much as anyone else who understands quantum mechanics and particle physics.

Still, just to be on the safe side, we need to sure of building up a good supply of Skeptrino’s for the experiment, a critical mass so to speak – or maybe not as Skeptrino’s are pretty likely to be massless particles.

We also need a venue that’s set up for detecting sub atomic particle like, say, the Large Hadron Collider and so, in order to ensure that Brian is lit up with Skeptrinos like a proverbial Christmas Tree, how about we put on a bike and get him to do a couple of laps of the main ring while listening to Stephen Fry reading carefully selected from ‘The God Delusion’ and ‘The Portable Atheist’ on an iPod – a Nano, naturally.

Then its off to the test chamber, turn the particle detectors up to their maximum resolution, and on with the test:

So, Sally, tell me everything you can about my great-grandfather…

What do you reckon?

Would that work – and if so would we not just have created the world’s first 100% reliable, scientifically proven, bullshit detector?

Surely that must be worth a paper or two in the British Journal of Psychology.