I’m not convinced this fixes anything. Even when a result is genuine, it’s very easy to fail to replicate it. We all know this from software development: it’s a lot easier to say “couldn’t reproduce” about a genuine bug than it is to track down the precise context in which the bug actually manifests. So if you get rewarded for failing to replicate a result, all the fraudsters will do that. If you get funded only when you actually replicate the result, the fraudsters will pretend to replicate the result.
> We all know this from software development: it’s a lot easier to say “couldn’t reproduce” about a genuine bug than it is to track down the precise context in which the bug actually manifests.
If a study claims to prove something, it should be repeatedly provable or it's a) fraud or b) not proven solidly enough.
I think replication is a key component of a functional research.
> So if you get rewarded for failing to replicate a result, all the fraudsters will do that. If you get funded only when you actually replicate the result, the fraudsters will pretend to replicate the result.
Yes, but the original author is incentivized to attempt to show where the replicators got it wrong, so there will still be a push to correct the bad data from the false replication failure. With that said, I am convinced it will be a panacea.
A bigger issue is that... 3-sigmas is really a weak signal in a high cardinality state-space, which is basically everything above physics of small numbers of elementary particles, and it is the elementary physicists that go for higher. This is what is feeding the replication crisis: Weak signals from very poorly sampled studies.
The meta issue is that we as a society need to start accepting that some things will take longer and require more investment to achieve results. Do fewer studies per unit grant, but do the three-sigma ones only to justify a real experiment/study, not as an acceptance criteria for "discovery!".
I'm not even going to touch politicalization, since I really have no idea what to do about it without a worse cure than the disease.
I’m not convinced this fixes anything. Even when a result is genuine, it’s very easy to fail to replicate it. We all know this from software development: it’s a lot easier to say “couldn’t reproduce” about a genuine bug than it is to track down the precise context in which the bug actually manifests. So if you get rewarded for failing to replicate a result, all the fraudsters will do that. If you get funded only when you actually replicate the result, the fraudsters will pretend to replicate the result.