I think that we should not accept peer review as some kind of gold standard anymore for several reasons. These are my opinions based on my experience as a scientist for the last 11 years.
- its unpaid work and often you are asked to do it too much and therefore may not give your best effort
- editors want to have high profile papers and minimise review times so glossy journals like nature or science often reject things that require effort on the review
- the peers doing a review are often anything but. I have seen self professed machine learning “experts” not know the difference between regression and classification yet proudly sign their names to their review. I’ve seen reviewers ask you to write prompts that are mean and cruel to an LLM to see if it would classify test data the same (text data from geologists writing about rocks). As an editor I have had to explain to adult tenured professor that she cannot write in her review that the authors were “stupid” and “should never be allowed to publish again”.
A further issue is peer review quid pro quo corruption. The reviewer loves your paper but requests one small change: cite some of his papers and he’ll approve your paper.
I don’t know how prevalent this sort of corruption is (I haven’t read any statistical investigations) but I have heard of researchers complaining about it. In all likelihood it’s extremely prevalent in less reputable journals but for all we know it could be happening at the big ones.
The whole issue of citations functioning like a currency recalls Goodhart’s Law [1]:
”When a measure becomes a target, it ceases to be a good measure.”
Tbh I used to have an issue with that but these days it really is a small issue in the grand scheme of things. You can say No but also, there are larger systemic problems in science.
Scientific peer review is another facit of civilization that its current design does not allow it to scale well. More and more people are being involved in the process, but the qualityis forever going down.
Yes that’s right. It’s a scaling problem and there isn’t a clear answer. It’s easy to complain about it though haha. I think what is happening is science is atomitizing. People are publishing smaller amounts or simply creating ideas from nothing (like that science advances paper on hacker news a couple days ago that created a hydrogen rich crust from thin air).
- its unpaid work and often you are asked to do it too much and therefore may not give your best effort
- editors want to have high profile papers and minimise review times so glossy journals like nature or science often reject things that require effort on the review
- the peers doing a review are often anything but. I have seen self professed machine learning “experts” not know the difference between regression and classification yet proudly sign their names to their review. I’ve seen reviewers ask you to write prompts that are mean and cruel to an LLM to see if it would classify test data the same (text data from geologists writing about rocks). As an editor I have had to explain to adult tenured professor that she cannot write in her review that the authors were “stupid” and “should never be allowed to publish again”.