Real Science is Real Nit-Picky

You know, that Galileo dude – what a nitpicker! Anybody can see that a feather falls more slowly than a rock. Clearly, things fall in some proportion to their weight. All this elaborate climb the tower of Pisa, drop two balls of different weight, time it – he’s just trying to dispute what everybody knows and can see with their own two eyes with a bunch of rigmarole and irrelevant minutiae.

Right?

And then he goes and builds a bunch of inclined planes! Will the preposterous shenanigans never stop?

Or that germ theory of disease. Really, little invisible beasties are making us sick? The medical consensus on the effectiveness of bloodletting is overwhelming, and has been for centuries! If it was good enough for George Washington, what kind of nut would dispute it?

And so on. I was called a nitpicker for pointing out that, based on their abstracts, the first 5 of the now-legendary 70 Studies(tm) that prove masks stop the spread of COVID do no such thing, because – well, now we get into nitpicking. My critic would no doubt have no idea what I even mean by saying, for example, a non-pre-registered meta-study comparing results from different states at different times is farcically flawed (1). Or that extrapolating results from clinical settings to the real world, and assuming any net reduction in infections in a surgery will just magically flow through to us greasy-fingered fumbling non-professionals using the same mask for hours, days, weeks at a time, is not justified, to put it mildly.

Then there’s the whole question of what a study is: it’s a study, an approach, a stab. At best – and rarely are any studies you’ve heard of ‘best’ in any positive sense – a study says that, this one time, doing things exactly like this, upon this population or data set, at this time and place, here’s what we got. The ubiquitous ‘further study is warranted’ with which modern studies always conclude is not always just a ‘please keep the funding coming’ pity-pitch. In those cases where a study uses a solid methodology upon a well-understood and defined population, AND gets interesting results, further study may very well be warranted. Maybe.

Further, as anyone paying attention knows, you can find studies that conclude whatever you want to conclude. Everything from ‘family beds kill babies’ to ‘space aliens built the pyramids’. Of course, there are also studies showing family beds are net better for babies and moms, and that the pyramids were built using simple tools, lots of smarts, and an insane amount of human muscle power. Thus, there are no doubt studies showing COVID is worse than the Black Death, social distancing and masks have saved millions of lives, and that this time, a vaccine for a constantly mutating airborne respiratory virus will work all the time with no unintended consequences at all.

It’s Science! Pay no attention to all the other studies and common sense that say otherwise.

Peer review? Please. You want to work in this town again, you don’t bash any of the thundering herd of our sacred cows stretching to the horizon. It would be cool if disinterested experts looked at papers critically and pointed out all their strengths and weaknesses – but, in the real world, that’s not what peer review means. It means: I have my buddies, people who might hire me in the future or who I might hire in the future, or be asked to recommend, or have to work alongside, take a look at my work, knowing full well everything that’s at stake if they look too critically.

Replication? Heard of that? Now you’re talking science. But you can’t get that PhD by replicating somebody else’s study – you have to do *original* work! Thus, with rare exceptions, efforts to replicate studies are works of charity – nobody is paying for it. Thus, in our fallen world, replication usually doesn’t happen.

What replication does for you is prove out the methodology. If I do – pretty much – exactly as you did in your study, I should get – pretty much – the same results. I can – and should – change data sets or populations to ones to which the ‘findings’ of the original study are said to apply. In other words, if the original study used college students but claimed or implied that the findings were generally applicable to entire population, I should be able to replicate the study using, I don’t know, long haul truckers and office clerks. Anything but more college students – unless the study specifically applies only to college students. In that latter case, if the study was done on Cambridge undergrads, I should run the protocol on Iowa State undergrads. And so on.

But none of this happens, except on rare occasions. A non-replicated study should be regarded as at best nothing more than an interesting rumor, something not to be believed or acted upon until a lot of supporting evidence comes in. Having a long series of non-replicated studies that all make the same or similar claims doesn’t make it any better. Lots of the same problem over and over again doesn’t make the problem go away. I’d be a lot more impressed if there were one study showing, say, that masks meaningfully reduce the spread of airborne viruses that got replicated 69 times with similar results, than I am by 70 independent studies showing kinda sorta comparable results. In fact, I’m not impressed by those 70 studies at all.

Because that’s how science – real science – really works.

Models are an expression of prejudice, and nothing more. Computer models even more so. Reliance on models is, in itself, proof you’re not talking about science at all. This is harsh, and years of training has made it difficult for many people, especially people with ‘science’ somewhere in their title or job description, to see this, but it is obvious and essential. If you knew the answer, you wouldn’t be building a model; since you don’t know the answer, you must build your model on assumptions. These assumptions are, necessarily and inescapably, expressions of your beliefs in the face of missing evidence. Your judgements in advance of the evidence. Your prejudices. No amount of math or processing cycles will make this problem go away. Models produce no evidence. At best, they point to places where you might look for evidence. Pretending models produce evidence is a sure sign of incompetence or fraud.

Finally, there’s Eisenhower’s warning to beware the government/science complex. He who pays the piper calls the tune. Thus, because science even back in 1960 was rarely the work of a lone genius laboring away in his meager, self-funded lab, but even then and more so now is rather something done in huge labs funded by the government and run by government bureaucracies, you’re only going to get to study issues the government – as bodied forth in those bureaucrats – is interested in; results they don’t like are unlikely to get funded.

And there’s nobody else out there to fund them. Research, and the findings of research, that the government doesn’t like are unlikely to ever see the light of day. This is reality.

I, a scientifically literate person, who, as this claim implies, have spent a lifetime, 50+ years, reading science, keep all these things and more in mind whenever presented with a claim that ‘science has shown’. I want to see the evidence, and how it was collected, and what reasoning was used, and what criticisms were offered, and how those criticisms were addressed. I want to see replication, and don’t give a flying goatherder if it’s been peer-reviewed or not. Failure at any of those points of examination means: science has NOT shown. I don’t care how pretty the claims are, or how passionate the clamant is. I don’t care how urgent the ‘problem’ is, nor how many people are sure it’s true. Science is built on evidence that can withstand withering criticism from ardent enemies. If your evidence is too fragile to stand up to opposition – if you feel the need to silence your opposition – you don’t have evidence. If you seek to silence your opponents while claiming science, you’re a despicable fraud.

Yep, I’m a nitpicker. That’s science for you.

1. What I mean: Pre-registering your research means stating before you do it what exactly you’re going to do, to prevent you from cherry-picking your data for the results you want, even if you only ‘want’ them unconsciously. Meta-studies compare results from different studies, which raises a host of questions about if the methodologies and populations of the studies are even comparable: think: comparing, say, blood pressure between Bantu tribesmen in 1920 and Chicago suburbanites in 2020. Raises more questions than it answers. Similarly, if less obvious, it is ridiculous to compare infection rates pre-mask mandates and post mask mandates between, say, New York in March and April and Florida in May and June, without an impossible boatload of controls and caveats – which were not done. Dressing it up in a study just creates a well-dressed piece of crap.