It's also the norm in some fields to provide only high level info in the methods section, often without supplemental method details accompanying. This makes it even harder to tell if they did the work correctly, because usually half the methods are intermediate steps which don't have any results directly reported in the paper. In a perfect world those methods would be uniform across labs, but in practice they definitely are not, and it makes tracing down the source of honest replication differences a nightmare.
There's also no way to really know if the researcher entirely left out 10 other tests they tried that failed. Sometimes you can guess that it's a stretch because of the stupid categories they use (I'm reminded of those ESPN graphics that say things like "most home runs on a rainy Tuesday in June"). But it's harder to detect if someone straight up removes data points, repeats tests and reports only the nicest, etc.
So at some point you basically need to be an insider in the field so you hear the gossip about what doesn't replicate. Or if you have access to thousands of dollars to blow you could try a dozen different variations to try getting it to replicate yourself.
I think for something like COVID that is actively affecting many people, there should be funding explicitly for replicating studies, and some slots reserved in a prestigious journal for the findings of the replications. I get that it is not feasible to be replicating everything in science, but I don't see why we can't have ~one lab per relevant university department that specializes in replicating important studies. If you make that a path towards becoming a tenured prof I think that could change the culture surrounding replication studies in general.
There's also no way to really know if the researcher entirely left out 10 other tests they tried that failed. Sometimes you can guess that it's a stretch because of the stupid categories they use (I'm reminded of those ESPN graphics that say things like "most home runs on a rainy Tuesday in June"). But it's harder to detect if someone straight up removes data points, repeats tests and reports only the nicest, etc.
So at some point you basically need to be an insider in the field so you hear the gossip about what doesn't replicate. Or if you have access to thousands of dollars to blow you could try a dozen different variations to try getting it to replicate yourself.
I think for something like COVID that is actively affecting many people, there should be funding explicitly for replicating studies, and some slots reserved in a prestigious journal for the findings of the replications. I get that it is not feasible to be replicating everything in science, but I don't see why we can't have ~one lab per relevant university department that specializes in replicating important studies. If you make that a path towards becoming a tenured prof I think that could change the culture surrounding replication studies in general.