Science Attempts to Replicate Science! Hilarity Ensues

Via Al Perrella, a reader of the Statistician to the Stars, today’s call to wailing and despair: 

Some psychologists tried to do what – pardon the caps, but just this one time they are warranted – EVERY LEGITIMATE SCIENCE DEMANDS BE DONE *EVERY* *SINGLE* *TIME*: REPLICATE RESULTS OF NOVEL STUDIES. Since they didn’t always get the exact same results of the original researchers, a food fight broke out. People doing – so sorry, gotta go caps again – WHAT EVERY LEGITIMATE BRANCH OF SCIENCE INSISTS BE DONE – ATTEMPTING TO REPLICATE ORIGINAL RESULTS are being accused of being

…the “replication police,” “shameless little bullies,” “self-righteous, self-appointed sheriffs” engaged in a process “clearly not designed to find truth,” “second stringers” who were incapable of making novel contributions of their own to the literature, and—most succinctly—“assholes.”

So, doing what must be done before any claims that ‘science has shown’ anything at all is now worthy of nasty condemnation. And defending novel finding against failed attempts to replicate them is the High Moral Ground. Whoa.

You’re not going to believe this, but in psychology and sociology and related fields, people publish novel studies and expect their results to be accepted before anyone has replicated them, and then call people names if they dare try to replicate those findings and fail. In science, as opposed to Science!, when attempts to replicate results fail, and those who attempted the replication followed all the right steps, the fingers are pointed at the original results. The people attempting to do the replication know, of course, that, should they fail to replicate, their protocols will be subject to brutal criticism by the people who did the original study, and so they work extra special hard to make sure they did it right, and include every detail of how they did it in their report, so that people can decide for themselves if they are being fair and rigorous. This is not some EXTRA  thing added to science – it is what science is all about.

Thus, Galileo lets people look through his telescope. He doesn’t publish a study about the moons of Jupiter and throw a fit if others don’t see them – he tells them to look for themselves (more or less – Galileo, God love him, was more than a bit of a jerk. You’d probably have needed to borrow somebody else’s telescope to look through.). Thus, Millikan, as part of his results, tells people exactly the nature of the device he built to measure the charge of an electron, so that others can build their own and check it out. But psychologists, it appears, don’t just release their results into the wild with enough information that anybody who wants to can try the experiment themselves – no, no, no, their results are too precious to be subjected to such rough treatment! You can only test their results if you ask nicely for the details of the protocols and agree to let them look over your results before you publish and, most of all, don’t say mean things about them.

In the scenario below, someone had the nerve to try to replicate the incredibly convincing, rigorously logical and not at all made up study of 40 – 40! – WEIRDs, and then 43 – 43!- more,  that showed that thinking right thoughts that affirm your OK-ness makes you less judgemental, something completely surprising and not at all confirming current cultural biases:

One of the articles in the special issue reported a failure to replicate a widely publicized 2008 study by Simone Schnall, now tenured at Cambridge University, and her colleagues. In the original study, two experiments measured the effects of people’s thoughts or feelings of cleanliness on the harshness of their moral judgments. In the first experiment, 40 undergraduates were asked to unscramble sentences, with one-half assigned words related to cleanliness (like pure or pristine) and one-half assigned neutral words. In the second experiment, 43 undergraduates watched the truly revolting bathroom scene from the movie Trainspotting, after which one-half were told to wash their hands while the other one-half were not. All subjects in both experiments were then asked to rate the moral wrongness of six hypothetical scenarios, such as falsifying one’s résumé and keeping money from a lost wallet. The researchers found that priming subjects to think about cleanliness had a “substantial” effect on moral judgment: The hand washers and those who unscrambled sentences related to cleanliness judged the scenarios to be less morally wrong than did the other subjects. The implication was that people who feel relatively pure themselves are—without realizing it—less troubled by others’ impurities. The paper was covered by ABC News, the Economist, and the Huffington Post, among other outlets, and has been cited nearly 200 times in the scientific literature.

Because mean people tried to do the same experiments on a large group of people – what could they be thinking? Wasn’t the original study good enough to get the lead researcher tenure at Cambridge? Anyway, because they found the results of this study to be wrong, they did the worst possible thing in their field: they threatened the ability of the lead researcher to get more grants!. Is outrage!

Schnall said that her work had been “defamed,” endangering both her reputation and her ability to win grants.

So: having your results subjected to the brutal review of your peers – standard procedure in all real science – is too tough? Maybe – well, no maybe about it – psychology isn’t a science, but is rather a weirdly geeky cool kids club, where everybody agrees to play nice so that tenure and grants keep flowing. Beats getting a real job, I suppose.

Closing aside: that the actual study itself as presented is a meaningless jumble of babble and magical thinking unable to yield anything but trivia is not being discussed here, except to mention there are steeplechases with fewer leaps than are required to get from pure word scrambles and handwashing to what people think and how they judge. Ritual incantations are made, and ‘results’ found. Sheesh.


Author: Joseph Moore

Enough with the smarty-pants Dante quote. Just some opinionated blogger dude.

2 thoughts on “Science Attempts to Replicate Science! Hilarity Ensues”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s