Limits...
Mechanism change in a simulation of peer review: from junk support to elitism.

Paolucci M, Grimaldo F - Scientometrics (2014)

Bottom Line: Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

View Article: PubMed Central - PubMed

Affiliation: Institute of Cognitive Sciences and Technologies, Italian National Research Council, Via Palestro 32, 00185 Rome, Italy.

ABSTRACT
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

No MeSH data available.


Influences on our peer review model. The different configurations inside the redesign box report the section number where the relative experiment will be discussed
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC4016809&req=5

Fig1: Influences on our peer review model. The different configurations inside the redesign box report the section number where the relative experiment will be discussed

Mentions: The simple-agent simulation we take as our starting point is the one proposed in Thurner and Hanel (2011). After replicating it with our BDI model, we explore some algorithmic variations that are inspired by some of the models examined above in the state of the art. In Fig. 1 we present the sources that will influence our model in the rest of the work in a schematic form.Fig. 1


Mechanism change in a simulation of peer review: from junk support to elitism.

Paolucci M, Grimaldo F - Scientometrics (2014)

Influences on our peer review model. The different configurations inside the redesign box report the section number where the relative experiment will be discussed
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC4016809&req=5

Fig1: Influences on our peer review model. The different configurations inside the redesign box report the section number where the relative experiment will be discussed
Mentions: The simple-agent simulation we take as our starting point is the one proposed in Thurner and Hanel (2011). After replicating it with our BDI model, we explore some algorithmic variations that are inspired by some of the models examined above in the state of the art. In Fig. 1 we present the sources that will influence our model in the rest of the work in a schematic form.Fig. 1

Bottom Line: Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

View Article: PubMed Central - PubMed

Affiliation: Institute of Cognitive Sciences and Technologies, Italian National Research Council, Via Palestro 32, 00185 Rome, Italy.

ABSTRACT
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

No MeSH data available.