Limits...
Mechanism change in a simulation of peer review: from junk support to elitism.

Paolucci M, Grimaldo F - Scientometrics (2014)

Bottom Line: Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

View Article: PubMed Central - PubMed

Affiliation: Institute of Cognitive Sciences and Technologies, Italian National Research Council, Via Palestro 32, 00185 Rome, Italy.

ABSTRACT
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

No MeSH data available.


No bad papers scenario; average quality (error bars are not visible at this scale) of accepted papers by percentage of rational cheaters, averaged on the last 10 years of simulation. The decrease in quality is approximately linear. Quality remains higher than in Fig. 3
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC4016809&req=5

Fig4: No bad papers scenario; average quality (error bars are not visible at this scale) of accepted papers by percentage of rational cheaters, averaged on the last 10 years of simulation. The decrease in quality is approximately linear. Quality remains higher than in Fig. 3

Mentions: Results (again for the last 10 years in a 40 years simulation) are shown in Fig. 4 by percentage of rational cheaters. With the removal of low quality papers, also the initial marked sensitivity disappears, making the response of the system to the injection of rational cheaters nearly linear. Also the quality of papers remains higher than the average quality even in the worst case, arriving just below seven. The initial sensitivity of the model to rational cheaters disappears; thus, by comparing this configuration with the previous one (Fig. 3), the indication is that pushing bad papers has a critical role in bringing the system to failure with few rational cheaters. In other words, the model indicates that rational cheaters are fatal for the functioning of peer review only if, in addition to being hostile to papers better than their own, they also promote low-quality papers. If they do not, they remain detrimental, but much less dramatically.Fig. 4


Mechanism change in a simulation of peer review: from junk support to elitism.

Paolucci M, Grimaldo F - Scientometrics (2014)

No bad papers scenario; average quality (error bars are not visible at this scale) of accepted papers by percentage of rational cheaters, averaged on the last 10 years of simulation. The decrease in quality is approximately linear. Quality remains higher than in Fig. 3
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC4016809&req=5

Fig4: No bad papers scenario; average quality (error bars are not visible at this scale) of accepted papers by percentage of rational cheaters, averaged on the last 10 years of simulation. The decrease in quality is approximately linear. Quality remains higher than in Fig. 3
Mentions: Results (again for the last 10 years in a 40 years simulation) are shown in Fig. 4 by percentage of rational cheaters. With the removal of low quality papers, also the initial marked sensitivity disappears, making the response of the system to the injection of rational cheaters nearly linear. Also the quality of papers remains higher than the average quality even in the worst case, arriving just below seven. The initial sensitivity of the model to rational cheaters disappears; thus, by comparing this configuration with the previous one (Fig. 3), the indication is that pushing bad papers has a critical role in bringing the system to failure with few rational cheaters. In other words, the model indicates that rational cheaters are fatal for the functioning of peer review only if, in addition to being hostile to papers better than their own, they also promote low-quality papers. If they do not, they remain detrimental, but much less dramatically.Fig. 4

Bottom Line: Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

View Article: PubMed Central - PubMed

Affiliation: Institute of Cognitive Sciences and Technologies, Italian National Research Council, Via Palestro 32, 00185 Rome, Italy.

ABSTRACT
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

No MeSH data available.