I really don’t know what to make of this. Yesterday morning, via the Google science news feed, I found an article on an effort to reproduce results from psychology studies published in 3 leading journals. Later that day, I went back to it, only to find that it had been heavily edited. I thought the original had gone down the memory hole, and so despaired of seeing exactly what had been changed. However, turns out I had not closed the tab with the original article from yesterday morning, and so was able to capture it. I then threw the original and revised versions into Word, ran a compare, and voila! I’ve posted the whole thing at the end.
Herein we investigate and ruminate.
Recap: Yesterday, a preposterously upbeat article on a study that showed that less than 40% of the results presented in a set of 100 research papers were reproducible. I say ‘preposterously upbeat’ because, in the real world of real science, it would be a tire-fire smoking, Death Star klaxon sounding DISASTER for any field to admit that 60%+ of its results were not reproducible. But here, it’s just business as usual:
The results are now in: More than 60 of the studies did not hold up. They include findings that were circulated at the time — that a strong skepticism of free will increases the likelihood of cheating; that physical distances could subconsciously influence people’s sense of personal closeness; that attached women are more attracted to single men when highly fertile than when less so.
This was the original 4th paragraph in the article. Note how trivial the claims being disputed are – nothing to see here, move along. Everything is Just Fine.
I objected to this. Imagine.
The revision eliminates this paragraph, and kicks the criticism up a dozen notches (Bold is my emphasis):
Now, a painstaking years long effort to reproduce 100 studies published in three leading psychology journals has found that more than half of the findings did not hold up when retested. The analysis was done by research psychologists, many of whom volunteered their time to double-check what they considered important work. Their conclusions, reported Thursday in the journal Science, have confirmed the worst fears of scientists who have long worried that the field needed a strong correction.
The vetted studies were considered part of the core knowledge by which scientists understand the dynamics of personality, relationships, learning and memory. Therapists and educators rely on such findings to help guide decisions, and the fact that so many of the studies were called into question could sow doubt in the scientific underpinnings of their work.
“I think we knew or suspected that the literature had problems, but to see it so clearly, on such a large scale — it’s unprecedented,” said Jelte Wicherts, an associate professor in the department of methodology and statistics at Tilburg University in the Netherlands.
Puts it in a slightly different light, huh?
I admit to not knowing what’s going on here. In every other case I’ve run across, one can count on the press to report on challenges to the cargo cult sciences – psychology, sociology, political science and their spawn – in such a way as to reassure the sheep that Nothing Is Wrong. Oh sure, everybody makes mistakes, but Science is Self Correcting(tm), so don’t trouble your fuzzy, bleating little heads about it.(1)
There are some other rather glaring bits evident to the reader’s gimlet eye. In general, though, I’m pleased at some of the tone here, although, eventually, when you get to the end, it’s still largely apologetics for the idea that we must accept the conclusions of psychology or sociology because our cell phone actually works!
Here is the compared versions. Had to embed a Word doc (we’ll see if that even works), as pasting into WordPress loses all the markup. Red and Green are the changes from the original. Don’t know what’s up with all the weird typefaces:
- See here and here for examples – the author does not distinguish between psychology and, say, chemistry. They are both, we are to believe, science in the same sense. Science is hard; cargo cult science is even harder, it seems. And because *technology* is self-correcting (get it wrong, and the tablet won’t boot, the rocket crashes, and lights won’t go on), and, as a side effect, often spurs correction of the underlying science, we are take as dogma that sociology, say, will miraculously fix itself! Marxist sociologists will eventually see the evidence that their theories don’t correlate to anything in the real world, and reject them, one hopes before those theories put another 100 million people in their graves. I’m pretty sure there are few logical steps missing here.