And we should help. Here we go: Daryl Bem Proved ESP Is Real. Money quote:
If you bought into those results, you’d be admitting that much of what you understood about the universe was wrong. If you rejected them, you’d be admitting something almost as momentous: that the standard methods of psychology cannot be trusted, and that much of what gets published in the field—and thus, much of what we think we understand about the mind—could be total bunk.
Of course, anyone sane and conversant in the scientific method would find the idea that “much of what you understood about the universe was wrong” to be nearly a truism, and “that the standard methods of psychology cannot be trusted, and that much of what … we think we understand about the mind—could be total bunk” to be a revelation on the order of “the sun rises in the East.”
These two ideas go together. While it is important – very important – to recognize that science, especially applied science, has produced a large number of very useful and valuable insights into how the world works, that’s not the same as thinking science has gotten to the bottom, or nearly to the bottom, of How Things Are. It is yet another truism that each scientific answer generates an unlimited supply of additional questions. This state of an ever-receding bottom is in addition to the metaphysical questions the answers to which are both essential to the very concept of science and outside the methods of science to answer.
Science should be a humbling exercise, the thrill of discovery balanced with the inescapable reality that there’s more to figure out than will ever be understood. While egomaniacs can be found in all areas of study, it seems there’s an overall bias: the softer the science, the more play there is for ego, the more ready people are to blow their own horn and take offense at legitimate questions.
Back to the article. There’s a useful recap of what happened in parapsychology in the late ’80s. James Randi had made a name for himself by showing, essentially, that parapsychologist are gullible rubes, or, more generously, that scientists are not trained to expect Nature to try to pull one over on them, leaving them vulnerable to frauds. With careers to consider and funding money on the table, this state of affairs must be addressed.
A raft of reforms were proposed and implemented. Experimenters were advised to be wary of the classic test for “statistical significance,” for example, since it could often be misleading. They should avail themselves of larger groups of subjects, so they’d have sufficient power to detect a real effect. They should also attempt to replicate their work, ideally in adversarial collaborations with skeptics of the paranormal, and they should analyze the data from lots of different studies all at once, including those that had never gotten published. In short, the field of parapsychology decided to adopt the principles of solid scientific practice that had long been ignored by their mainstream academic peers.
“the principles of solid scientific practice that had long been ignored by their mainstream academic peers.” Let that sink in. Psychology is a field where Freud remains among the top handful of most cited sources. For those who have not had the pleasure of reading ol’ Siggy, he perfected and took to new extremes the approach of answering critics *of* his theories from *inside* his theories – typically, any attempt to point out flaws in his theorizing (and they are patent and legion) was answered by the accusation that the critic was obviously repressed. Jung counts on the same dynamic – reflexive dismissal of critics as simple unenlightened – but has vaguer, less vulgar theories and so appears nicer about it. And so, down the lineage of ‘great’ psychologists to this day.
Success in such an environment hinged more on titillating the undergrads and keeping a straight face than on anything remotely related to science. All serious and fundamental criticism was summarily dismissed – it had to be, or we’d have never heard of these jokers, who, based on the merit of their theories alone, would hold the same intellectual position as Rosicrucians. Instead, they got paying gigs on the public teat at our great universities, and positions of influence over our young.
Not that things never changed. After Skinner and all the rat running (1), it became popular to use cook-book level statistical analysis in studies. To do this, one needs to assign numerical values to data, ignoring that much, maybe most, things that count as data in psychology do not admit of valid numerical values (on a scale of 1 to 5, how happy are you right now?). Low p-values became the ultimate validation that what you were doing was real, just like the real scientists.
Further, in order to get those p-values, it became common practice to follow many paths, ignore the ones that didn’t ‘work’ and report on those that did. This is an example of an old-style scam (not that the researchers were always aware that scamming was what they were doing – could be enthusiasm + ignorance): send a prediction to 1000 people on who will win that week’s big game – 500 predicting the home team, 500 predicting the away team. Next week, discard the 500 you got wrong, and send out 500 more to those you got right on this week’s big game – 250 predicting the home team, 250 predicting the away team. Repeat a few more times. Then send a note to the remaining people who have received an amazing string of predictions that proved right, saying you’ll send them predictions for the upcoming week for a mere $1,000. How could they resist? You’ve never been wrong before!
To Slate’s credit, this is all explained fairly well in the article.
Bern submitted a paper for publication to the Journal of Personality and Social Psychology, the most prestigious and rigorous journal in his field. An E. J. Wagenmakers read it.
Wagenmakers finally managed to get through Bem’s paper. “I was shocked,” he says. “The paper made it clear that just by doing things the regular way, you could find just about anything.”
“Clearly by the normal rules that we [used] in evaluating research, we would accept this paper,” said Lee Ross, a noted social psychologist at Stanford who served as one of Bem’s peer reviewers. “The level of proof here was ordinary. I mean that positively as well as negatively. I mean it was exactly the kind of conventional psychology analysis that [one often sees], with the same failings and concerns that most research has.”
This was all happening way back in 2010. As a result, there is a movement to tighten up research practices. The article neither mentions nor have I read elsewhere any movement to disavow all findings under the previous method, after the manner in which companies recall batches of product that have poison in them. Calling this a ‘replication crisis’ is dramatically underselling the problem: we have a ‘this is a stinking pile and needs to be shoveled out of here’ crisis. But no one in the field will say that. Instead they will say limp-wristed things like ‘these issue call some earlier findings into question.’ Right. (2)
The article, which is in general commendable and full of useful information, still attempts early on the standard ‘science is hard’ spin I’ve found so often in places like fivethirtyeight: any inclination you might have toward dismissing the entire field of psychology must be resisted, because science is hard!
The replication crisis as it’s understood today may yet prove to be a passing worry or else a mild problem calling for a soft corrective. It might also grow and spread in years to come, flaring from the social sciences into other disciplines, burning trails of cinder through medicine, neuroscience, and chemistry. It’s hard to see into the future. But here’s one thing we can say about the past: The final research project of Bem’s career landed like an ember in the underbrush and set his field ablaze.
Note the not so subtle inclusion of medicine, neuroscience and chemistry as other fields that might be affected by these methodological problems. These three fields do not stand in the same relationship to scientific method as the the social “sciences”. If by neuroscience the author means the wild approaches that lead to MRI studies of dead salmon, then, yes, neuroscience is in exactly the position of psychology. Medicine, on the other hand, has always been a combination of art and science, and has always had a lunatic fringe very similar to mainstream psychology in its approaches and conclusions. But medicine also has results – epidemics prevented, successful surgeries, recoveries from formerly fatal conditions – much more measurable and important. Finally, chemistry is wonderful in that it either works or it doesn’t, so that if you make a claim with any real-world implications, incompetence and fraud will soon out.
No, Slate, there’s no chance this is “a passing worry or else a mild problem calling for a soft corrective.” Nor is it likely to have much effect on fields where hard, objective results are routinely demanded.
There is no replication crisis. There is a this is utter BS crisis, to be resolved once people in general conclude: the social sciences are purveyors of utter BS.
Why, yes, I am a little grumpy today. Why do you ask?
- Long quote from Feynman’s famous and oft quoted Cal Tech commencement speech:All experiments in psychology are not of this type, however. For example, there have been many experiments running rats through all kinds of mazes, and so on–with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.
The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the
corridor, and still the rats could tell.
He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell.
Now, from a scientific standpoint, that is an A-number-one experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers the clues that the rat is really using–not what you think it’s using. And that is the
experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running.
I looked into the subsequent history of this research. The next experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn’t discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic of cargo cult science.
- There are, of course, people who were thrilled at Bern’s results, and accepted them with unfiltered enthusiasm: “But for Bem’s fellow members of the Parapsychological Association, the publication marked a great success. “He brought a lot of attention to the possibility that this research can be done, and that it can be done in a mainstream establishment,” says Marilyn Schlitz, a sociolinguist who studies psi phenomena and has an appointment at the Institute of Noetic Sciences in Petaluma, California.“