UPDATE: score one for Truth, Justice and the American Way. The Paper of Record totally redid the article since I cut and pasted those couple paragraphs below. Gone are the poo-pooing of the issue with claims that things are generally OK, we just need to tighten up a little – now we’ve got:
Their conclusions, reported Thursday in the journal Science, have confirmed the worst fears of scientists who have long worried that the field needed a strong correction.
The vetted studies were considered part of the core knowledge by which scientists understand the dynamics of personality, relationships, learning and memory. Therapists and educators rely on such findings to help guide decisions, and the fact that so many of the studies were called into question could sow doubt in the scientific underpinnings of their work.
“I think we knew or suspected that the literature had problems, but to see it so clearly, on such a large scale — it’s unprecedented,” said Jelte Wicherts, an associate professor in the department of methodology and statistics at Tilburg University in the Netherlands.
Whoa. That’s a little different, and a lot more to the point. Not perfect – perfect would be a call for disbanding all cargo cult science departments, tarring and feathering enough of the more egregious offenders to offer a cautionary tale to undergrads, and having some real scientists pronounce on the validity of all existing studies and pre-approve proposed protocols and methods before any future studies are done. But hey, a world better than the self-congratulating stuff I captured this morning.
There’s also this gem:
The act of double-checking another scientist’s work has been divisive. Many senior researchers resent the idea that an outsider, typically a younger scientist, with less expertise, would critique work that often has taken years of study to pull off.
“There’s no doubt replication is important, but it’s often just an attack, a vigilante exercise,” said Norbert Schwarz, a professor of psychology at the University of Southern California.
Riiight. You mean, younger scientists without careers, tenure and grants to defend?
Could it be that someone at the Times read my blog and said – stop the presses! We must tone this down a little in view of that Yard Sale of the Mind dude’s incisive and damning points! Hey, it’s possible, stop laughing! Or maybe enough serious researchers coughed up a hairball or two, and the Times reacted? Who knows?
I’m leaving the original essay intact, for posterity. My only regret is that I failed to capture the whole thing, which may now be down the memory hole.
With, one presumes, a straight face, the New York Times informs us:
Ya think? I also like the saw off the limb you’re sitting on nature of a study casting doubt on studies. Anyways:
The past several years have been bruising ones for the credibility of the social sciences. A star social psychologist was caught fabricating data, leading to more than 50 retracted papers. A top journal published a study supporting the existence of ESP. The journal Science pulled a political science paper on the effect of gay canvassers on voters’ behavior – also because of concerns about fake data.
A University of Virginia psychologist decided in 2011 to find out whether such suspect science was a widespread problem. He and his team recruited more than 250 researchers, identified 100 studies that had each been published in one of three leading journals in 2008, and rigorously redid the experiments in close collaboration with the original authors.
The results are now in: More than 60 of the studies did not hold up. They include findings that were circulated at the time — that a strong skepticism of free will increases the likelihood of cheating; that physical distances could subconsciously influence people’s sense of personal closeness; that attached women are more attracted to single men when highly fertile than when less so.
The new analysis, called the Reproducibility Project and posted Thursday by Science, found no evidence of fraud or that any original study was definitively false. Rather, it concluded that the evidence for most published findings was not nearly as strong as originally claimed.
My first thought: almost 40% of the studies were good to go? Way too high. Also, you’d have to look for evidence of fraud to find it. Suspect they went light on that. Problem is, the fraud chiefly lies in the premises and methods of the social sciences themselves, while the callow researchers are but victims – victims! – of a system that makes sure anyone skeptical of the whole program never gets grants or tenure. Maybe if we had someone other than true believers attempt replication? Like a bunch of chemists or physicists? That would be cool.(1)
My second thought: what’s with this “rigorously redid the experiments in close collaboration with the original authors”? Isn’t that kind of missing the point? If the original authors were doing real science in the first place, they’d have published everything a future researcher would need in order to replicate the results. At most, the new researchers might check back with the original authors to clarify a point or two. But close collaboration between the original researchers and those trying to replicate the results is exactly what is to be avoided in real science – you want *independent* validation, not incestuous confirmation. That’s how every real science works.
My third thought: “The past several years have been bruising ones for the credibility of the social sciences.” Social sciences have credibility? To the extent that they do, it’s a desired result of the production by the schools of the sort of hopelessly gullible and ignorant rubes that would fall for this nonsense. Critical thinking, indeed.
My last thought: Does’t this make the people behind this study “…the “replication police,” “shameless little bullies,” “self-righteous, self-appointed sheriffs” engaged in a process “clearly not designed to find truth,” “second stringers” who were incapable of making novel contributions of their own to the literature, and—most succinctly—“assholes.” Because, after all, what is truth, compared to the need for grants and tenure? I’m having the uncharitable thought that this effort is much like our future first woman president telling us that there was nothing interesting in all those missing emails – she checked! – and we’d be all kinds of misogynistic and mean to want to take a look at ’em with our own eyes. If you can’t believe a candidate for President, who can you believe?
- The notion that (Orwell Newspeak Alert!) social sciences are somehow such exquisite specialties that no mere mortal uninitiated in the Mysteries could hope to fathom them is ridiculous. Anybody trained in real science can spot the issues from a block away.