Limits...
Mechanism change in a simulation of peer review: from junk support to elitism.

Paolucci M, Grimaldo F - Scientometrics (2014)

Bottom Line: Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

View Article: PubMed Central - PubMed

Affiliation: Institute of Cognitive Sciences and Technologies, Italian National Research Council, Via Palestro 32, 00185 Rome, Italy.

ABSTRACT
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

No MeSH data available.


Results from restrained rational cheaters. Average quality with error bars of accepted papers by percentage of rational cheaters, calculated on the last ten years. Surprisingly, quality increases initially with the number of cheaters—the restraint allowing the very best papers to pass, and only those, causing an elitist effect
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC4016809&req=5

Fig6: Results from restrained rational cheaters. Average quality with error bars of accepted papers by percentage of rational cheaters, calculated on the last ten years. Surprisingly, quality increases initially with the number of cheaters—the restraint allowing the very best papers to pass, and only those, causing an elitist effect

Mentions: In this section, we present results from another variation. Until now, we had not taken advantage of the rd parameter that controls, so to say, the self restraint of rational cheaters, preventing them from attributing to papers a score that is too distant from the actual one (previous settings deactivated this restraint by setting rd = 10). However, issuing reviews that are bound to be in disagreement with other supposedly non-cheating reviewers is risky; while a certain amount of disagreement is unavoidable and perhaps even healthy, giving widely diverging values puts the reviewer to the risk of being detected (Grimaldo and Paolucci 2013). Now we run a set of simulations with the same scenario as the last one (“Conferences decide” section) but we activate this “restraint” mechanism; here we show results obtained for rd = 5. In Fig. 6, a surprising result awaits: the trend inverts at starts, rational cheaters causing an slight increase instead of a decrease for the overall quality of the system. How is this possible? Simple enough: in a setting where rational cheaters show restraint, the papers that get accepted are only the excellent ones. The strategy of rational cheaters retorts against them, ending in an elitist situation - the mechanism of acceptation locks up so much that normal papers cannot get through, while the very best ones can. Rational agents, designed for blocking papers that are “too good”, end up instead in promoting excellence.Fig. 6


Mechanism change in a simulation of peer review: from junk support to elitism.

Paolucci M, Grimaldo F - Scientometrics (2014)

Results from restrained rational cheaters. Average quality with error bars of accepted papers by percentage of rational cheaters, calculated on the last ten years. Surprisingly, quality increases initially with the number of cheaters—the restraint allowing the very best papers to pass, and only those, causing an elitist effect
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC4016809&req=5

Fig6: Results from restrained rational cheaters. Average quality with error bars of accepted papers by percentage of rational cheaters, calculated on the last ten years. Surprisingly, quality increases initially with the number of cheaters—the restraint allowing the very best papers to pass, and only those, causing an elitist effect
Mentions: In this section, we present results from another variation. Until now, we had not taken advantage of the rd parameter that controls, so to say, the self restraint of rational cheaters, preventing them from attributing to papers a score that is too distant from the actual one (previous settings deactivated this restraint by setting rd = 10). However, issuing reviews that are bound to be in disagreement with other supposedly non-cheating reviewers is risky; while a certain amount of disagreement is unavoidable and perhaps even healthy, giving widely diverging values puts the reviewer to the risk of being detected (Grimaldo and Paolucci 2013). Now we run a set of simulations with the same scenario as the last one (“Conferences decide” section) but we activate this “restraint” mechanism; here we show results obtained for rd = 5. In Fig. 6, a surprising result awaits: the trend inverts at starts, rational cheaters causing an slight increase instead of a decrease for the overall quality of the system. How is this possible? Simple enough: in a setting where rational cheaters show restraint, the papers that get accepted are only the excellent ones. The strategy of rational cheaters retorts against them, ending in an elitist situation - the mechanism of acceptation locks up so much that normal papers cannot get through, while the very best ones can. Rational agents, designed for blocking papers that are “too good”, end up instead in promoting excellence.Fig. 6

Bottom Line: Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

View Article: PubMed Central - PubMed

Affiliation: Institute of Cognitive Sciences and Technologies, Italian National Research Council, Via Palestro 32, 00185 Rome, Italy.

ABSTRACT
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

No MeSH data available.