I think you’re extrapolating too much. Again, you’re making a strong claim based on limited evidence which is the hallmark of bad reasoning. I work in research and publish/referee articles regularly. I have no qualms with scientists, but I’m also acutely aware of how the system is flawed.

>I think the first claim is made without evidence

I get the impression you’re too riled to actually see my point so I’ll be explicit. We have lots of evidence that bad science is pretty widespread, especially in certain disciplines. We also have anecdotal examples of fraud. The latter is a subset of the former, with the remainder being good work. How big of a subset is it? Your claim is that it's small; my claim is we don't know.

There’s no goalpost shifting. All of the points made are related to the issue of fraud. Asking for evidence of your claim is not goalpost shifting, nor is understanding that fraud is a consequence of human incentives for status that exist in practically every domain. I simply made the point that we know there’s a problem, but likely don’t know the extent of it. Meanwhile, you made the claim that academic fraud is rare. And yet you also claim it’s impossible to prove that it’s not rare. It’s pretty clear that you’re not making a reasoned argument, which is the irony of someone defending the current state of science.

I read back through my posts to find the claim I made that you’re talking about and I can’t find it. Please let me know what you’re talking about, maybe quote it. I’m gathering this has mostly become a troll at this point, and I’m sorry I kept responding.

>examples tend to illustrate how relatively rare it is

Given the fraud and replicability issues that have been shown to exist even within the most reputable institutions and journals, I think this claim needs some substance to back it up.

>this has mostly become a troll at this point

Please review the HN guidelines. They specifically reference assuming good faith and avoiding shallow dismissals. I’ve tried giving that to you by asking questions and giving you multiple opportunities to give evidence to your position.

1. The initial post made a strong claim about high levels of fraud. 2. I asked for evidence to support this. The evidence given was arguably very sparse.

The quote you shared is from my response to point (2). Arguably in a world of infinite pedantry it could have been phrased differently. For example: “given how relatively rate cases of fraud supported by actual evidence are” might have been clearer. However given the context I assumed that claims of scientific fraud would involve evidence.

This seemed a good place to end the conversation. Yet:

3. You came in and “judo flipped” the conversation to “aha; but what about the fraud that could theoretically exist but that somehow had produced no evidence of its existence. Perhaps these levels are very high!” 4. I said that evidence of absence would be enormously difficult to prove, given that it would require resources beyond those we currently spend on science to identify this. 5. And moreover I pointed out that we have never had the means to detect this evidence-non-producing fraud and yet science has managed to produce massive and visible benefits that we experience on a daily basis.

And thus this leaves us at an impasse. All science may be full of fraud and bad results that we cannot detect. Perhaps it always had been! And yet this has not prevented science from producing results that are enormously valuable. I am left with some questions:

1. What strategy would you pursue to answer your extremely difficult question, given that real-world resources are extremely finite. Please do not say “well, we just need to replicate every result” becayse this simply misunderstands the point about resources. 2. If we cannot devise a feasible policy to test your “perhaps science is full of fraud” question, what should we do instead? Is there an actionable policy you’d pursue?

My view is that strong answers to both questions are not so much interesting to me, but interesting in terms of whether your world-view has value. Surprise me!

The other commenter did provide some evidence (albeit anectodal) of obvious fraud. In the context of non-replicable studies, it can be as high as 2/3rds are shown to be non-reproducable in some domains. This includes some of the highest quality journals and institutions. There's quite a bit of evidence of this. Now I'm not claiming that is all fraud, but I am claiming we don't know how much of the problem is fraud.

>evidence of absence would be enormously difficult to prove

I disagree. I'm taking the point that it's not hard to prove, but rather the incentives are aligned against it. If data and methods are truly open-sourced, much of this can be proven. But there is little status or money to be gained from reproducing work, leading to a situation where we have a system where it is rarely done.

>yet science has managed to produce massive and visible benefits

I don't disagree, but I think you're are misinterpreting my position. I said the system is flawed, not that it's worthless. Put differently, it still produces good science but it's hard to separate the good science from the bad. The signal-to-noise ratio is potentially low, and the rate of publication exacerbates this. I would argue that boosting the signal-to-noise ratio by weeding out bad science could produce much more benefit because less time is wasted chasing trails of bad science.

>What strategy would you pursue

You keep framing this as some impossible scenario. I'm saying it only appears that way within the context of the current incentive struture. So my goal would be to create a change to that incentive structure. This is off the top of my head, so don't take it too seriously but here's what I would try:

    1) We need to change the way academics are measured. Currently, it's a "publish or perish" paradigm that puts a premium on novelty and essentially disregards most everything else. There is also a glut of PhDs; far too many given the number of tenured academic position. This puts an enormous amount of pressure on a PhD to produce publications and this can manifest itself in pushing bad science. There's two things we could do to help this:

    1.(a) Provide an incentive where reproducing research is valuable. Currently, researchers don't want to spend time reproducing studies because there's very little upside. They won't generally get published so it doesn't help their career. I would say we need to start giving more status to reproduction studies. Probably not as much as more "novel" studies, but there should be an avenue to publish them. Maybe a two-tiered journal structure where we have *Nature* and *Nature:reproductions*, where the latter is used to publish works that re-create the studies in the first. Metrics like citeScore could be weighted so that the novel studies give you a higher metric than the reproductions, but you still get some credit for either.

    1.(b) Provide a pathway to also publish "non surprising" studies. In the current structure, research is more prioritized if it produces surprising results. Thus, if a researcher produces work that is unsurprising, they tend to just throw it out and try something else. This can lead to wasted effort when another researcher ends up doing the same thing. We should create an avenue to also publish "unsurprising" results. This has already started to get applied in some domains, like economics.

    2. Right now, publications can hide their data. I think there should be a cultural shift where the highest quality journals make sharing the data and code a mandatory requirement. This ties directly in to facilitating the replicability as well as allowing others to look at your methods and see if there's any wonkiness going on (whether fraud or otherwise). Some of the issues of fraud that we've seen are because researchers couldn't hide their data because it came from a third party (eg, Dan Ariely and Francesca Gino were outted because their data was owned by an insurance company that was willing provide insight into the raw data).

    3. Journals should set aside a portion of fees to fund replication. As it stands, journals get free labor from referees. I would argue this should be balanced by having some money set aside to fund a process aimed at replicating some of the studies in journal. It's wild to me that we have a system where a referee is expected to work for free and also to pay for a subscription to the journal they are helping.