Limits...
Mechanism change in a simulation of peer review: from junk support to elitism.

Paolucci M, Grimaldo F - Scientometrics (2014)

Bottom Line: Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

View Article: PubMed Central - PubMed

Affiliation: Institute of Cognitive Sciences and Technologies, Italian National Research Council, Via Palestro 32, 00185 Rome, Italy.

ABSTRACT
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

No MeSH data available.


Number of published papers for different mechanisms. Scenarios that perform better (restraint and conf decide) allow only few papers to be published. The no bad papers scenario is in a middle position; allowing bad papers in in the replication scenario increases massively the number of published papers
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC4016809&req=5

Fig7: Number of published papers for different mechanisms. Scenarios that perform better (restraint and conf decide) allow only few papers to be published. The no bad papers scenario is in a middle position; allowing bad papers in in the replication scenario increases massively the number of published papers

Mentions: In the discussion so far, we focused on quality only. What about quantity, that is, how many papers are accepted in each of the different scenarios? Obviously enough: scenarios that accept more papers are bound to exhibit lower quality. In Fig. 7 we present a summary, showing the number of accepted papers by condition. In the replication scenario, an initial explosion in the number of accepted papers corresponds to the sudden drop in quality; the scenario where rational cheaters do not push bad papers also increases the number of accepted papers, but only moderately when compared with the previous one. Both the conferences decide scenario and the restraint one, on the contrary, decrease the number of accepted papers with the increase of the rational cheaters ration; this maps, respectively, on the slight decrease and slight increase that we see in Figs. 5 and 6. Regrettably, we do not have indications on the quantity of papers accepted in the TH-HA original formulation, and thus we cannot confirm or disconfirm the validity of our replication from this point of view.Fig. 7


Mechanism change in a simulation of peer review: from junk support to elitism.

Paolucci M, Grimaldo F - Scientometrics (2014)

Number of published papers for different mechanisms. Scenarios that perform better (restraint and conf decide) allow only few papers to be published. The no bad papers scenario is in a middle position; allowing bad papers in in the replication scenario increases massively the number of published papers
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC4016809&req=5

Fig7: Number of published papers for different mechanisms. Scenarios that perform better (restraint and conf decide) allow only few papers to be published. The no bad papers scenario is in a middle position; allowing bad papers in in the replication scenario increases massively the number of published papers
Mentions: In the discussion so far, we focused on quality only. What about quantity, that is, how many papers are accepted in each of the different scenarios? Obviously enough: scenarios that accept more papers are bound to exhibit lower quality. In Fig. 7 we present a summary, showing the number of accepted papers by condition. In the replication scenario, an initial explosion in the number of accepted papers corresponds to the sudden drop in quality; the scenario where rational cheaters do not push bad papers also increases the number of accepted papers, but only moderately when compared with the previous one. Both the conferences decide scenario and the restraint one, on the contrary, decrease the number of accepted papers with the increase of the rational cheaters ration; this maps, respectively, on the slight decrease and slight increase that we see in Figs. 5 and 6. Regrettably, we do not have indications on the quantity of papers accepted in the TH-HA original formulation, and thus we cannot confirm or disconfirm the validity of our replication from this point of view.Fig. 7

Bottom Line: Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

View Article: PubMed Central - PubMed

Affiliation: Institute of Cognitive Sciences and Technologies, Italian National Research Council, Via Palestro 32, 00185 Rome, Italy.

ABSTRACT
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

No MeSH data available.