Been thinking further on the quotation used in this post, where a young astrophysicist captures the essence of what makes a study scientific:
“Now apparently the authors spent months checking their work, just to make sure it was robust and would stand up to very intense scrutiny because they knew it was going to get that. They checked it for measurement error and systematic error and statistical errors, but by the end of it they just couldn’t deny what they found. You know, even the most die-hard of dark matter fans could not deny what they found.”
What a study is is a proposal. A study says: here is what we found, here is how we found it, here is what we conclude from it – what do you think? Any individual study is an invitation to criticism. See that quotation above – that’s what that study is doing. The authors knew that throwing some evidence out there that calls into question an accepted and beloved theory was going to get criticized, so, preemptively, they worked hard – over the course of months – to hammer on the data themselves to toughen it up for the waves of scrutiny and criticism they knew were coming. They knew it was coming, because they, themselves, had a working knowledge of how science works. The very idea that you would criticize the critics for being critics never occurred to them, and, if it had, would have been laughed out of the room. Critics are not some necessary evil. Critics are how science works.
What makes a study ‘scientific’ in the precise sense of making its claims something an honest, reasonable person would be obliged to (conditionally – always conditionally) accept, is exactly this process of getting hammered on by critics. A study too rushed or too delicate to be able to run this gauntlet of criticism is, by that fact alone, abandoning any right to claim an honest man’s consent.
So, when the infamous 70 studies showing masks slowed the spread of COVID came out, you knew right off the bat that we were being snowed. The ink was still wet on those studies – had to be, if they were studying COVID, which, at that time, was purported to have been around only a few months. Generally, science takes time, yet no time was allowed. Where were the criticisms and replication? Attempts at criticism were roundly denounced as science denial, the epitome of Orwellian propaganda: labeling wrongthink with a name that is the opposite of reality. War is peace. Insisting on science is science denial.
Testing of your claims and replication of your results by people who would be happy to disprove them are the highest level of such scientific scrutiny, and have a central and honored place in science history. When Einstein claimed that starlight would detectably bend when it passed close to a very massive object like the sun, I’m guessing any number of scientists who set out to see if this were true would have been just as happy if it were not. When the results came in, and Einstein was proven correct, there was not universal rejoicing. Physicist ever since have been trying to poke holes in relativity, and there are some cracks (so I am told – a lot of this stuff is over my head). But relativity is accepted today BECAUSE it has been beaten up, raked over the coals, poked and prodded for a century now. Then, best of all, it has been successfully USED – in astronomy, astrophysics, rocket science, and in the GPS system your phone uses. That’s science in all its glory.
Not so the COVID, not so! While science works, insofar as it does, precisely as a result of skeptical but rational critics beating up your theory, your paper, your study, in the case of COVID, compliance is so important that the flimsiest, most rushed, most laughably flawed study is somehow beyond criticism.
Yard Sale of the Mind 
Very good thinking!
something an honest, reasonable person would be obliged to (conditionally – always conditionally) accept
I would quibble.
There is no such obligation.
By that I mean that no one has a general obligation to believe the established results in every scientific field, no matter how solid the scholarship.
There are aspects where one may have an obligation that is specific, but not one that is general. Look at the question of who killed Jimmy Hoffa as a test example. I can willingly accept considering, or be forced to consider a specific question, but I cannot be forced to look at every single question. Of the questions I do consider, only a subset are worth my time and effort going towards a good answer. With Hoffa, the only obligation that would stop me from accusing Walter Mondale is my responsibility for my own mental health, and hence not getting invested in bizarre conspiracy theories.
Like with the professional obligations of lawyers and accountants, there are occupations that might have an obligation to consider certain scientific questions, and come to a defensible answer. Again, this would be within a field or discipline of science, not a matter of being forced to weigh every field as significantly as the field one was trained in.
One occupation, you apply for a credential for solving a type of problem, a bureaucracy considers the application, then they hand it out if they think you know how to get good answers. They can take the credential away if they do not like the way you use it, or you cannot defend your answers, but they cannot go around using the credential to force you to say something just because a paper in the field has been well received and makes that claim. The bureaucracy doesn’t know much about the problems you are solving, so if they need help figuring out if you’ve done it badly, they ask your peers that solve the same sort of problem. You have a lot of leeway to pick the problems you solve, and you can learn what the peers who solve a sort of problem would tend to say, so even this obligation is pretty weak in terms of forced speech.
I don’t think there’s a general obligation to believe anything said in the name of science. In fact, I often said here (and will in the book as well) the right answer to almost every question is “I don’t know”.
My example would be something like: the melting point of iron, or that bodies (within certain conditions) fall at the same speed regardless of their relative weights. A explanation – a study, say, – could walk me through the steps and the reasoning, and I, an honest, reasonable man, would be compelled by my honesty and reason to CONDITIONALLY accept 1535C and 32’sec^2. The conditions being, in this case, primarily range – for bodies within a certain size and distance, for iron of a certain purity, that sort of thing – and reason – assuming you didn’t miss something material.
No appeals to authority are allowed. Owning a lab coat or a PhD or being president of the RightThink Society gets you nothing.
A large part of what I’m attempting to do here is reign in the range of claims people are willing to accept on the basis of science.
The really convincing thing for scientific claims is when they are used, for two reasons: Claims for example for dark matter are interesting, but they are unlikely to ever be used, so who cares? I personally don’t think I have any reason to accept them despite the vast bulk of professional cosmologists and astrophysicists thinking its true. But even better: acceleration under gravity and the melting point of iron are used in engineering and ironworking every minute. If they weren’t pretty accurate, stuff would fall down and way fewer fancy metal object would exist.