Limits...
Effectiveness of journal ranking schemes as a tool for locating information.

Stringer MJ, Sales-Pardo M, Nunes Amaral LA - PLoS ONE (2008)

Bottom Line: Here, we systematically evaluate the effectiveness of journals, through the work of editors and reviewers, at evaluating unpublished research.Our model enables us to quantify both the typical impact and the range of impacts of papers published in a journal.Finally, we propose a journal-ranking scheme that maximizes the efficiency of locating high impact research.

View Article: PubMed Central - PubMed

Affiliation: Department of Physics and Astronomy, Northwestern University, Evanston, Illinois, USA.

ABSTRACT

Background: The rise of electronic publishing, preprint archives, blogs, and wikis is raising concerns among publishers, editors, and scientists about the present day relevance of academic journals and traditional peer review. These concerns are especially fuelled by the ability of search engines to automatically identify and sort information. It appears that academic journals can only remain relevant if acceptance of research for publication within a journal allows readers to infer immediate, reliable information on the value of that research.

Methodology/principal findings: Here, we systematically evaluate the effectiveness of journals, through the work of editors and reviewers, at evaluating unpublished research. We find that the distribution of the number of citations to a paper published in a given journal in a specific year converges to a steady state after a journal-specific transient time, and demonstrate that in the steady state the logarithm of the number of citations has a journal-specific typical value. We then develop a model for the asymptotic number of citations accrued by papers published in a journal that closely matches the data.

Conclusions/significance: Our model enables us to quantify both the typical impact and the range of impacts of papers published in a journal. Finally, we propose a journal-ranking scheme that maximizes the efficiency of locating high impact research.

Show MeSH

Related in: MedlinePlus

Modeling the steady-state distributions of the number of citations for papers published in a given journal.(A) Our model assumes that the “quality” of the papers published by a journal obeys a normal distribution with mean μ and standard deviation σ. The number of citations of a paper with quality q∈N(μ,σ) is given by Eq. (3). Because the quality is a continuous variable whereas the number of citations is an integer quantity, the same number of citations will occur for papers with qualities spanning a certain range of q. In particular, all papers for which q<log10(1+γ) will receive no citations. In the panel, the areas of differently shaded regions yield the probability of a paper accruing a given number of citations. (B) Scatter plot of the estimated value of σ versus  for all 2,267 journals considered in our analysis (see Methods and Appendices S1 and S4 for details on the fits). Notice that σ is almost independent of . The solid line corresponds to σ = 0.419, the mean of the estimated values of σ for all journals (see Methods). (C) Scatter plot of the estimated value of γ+1 for versus . Notice the strong correlation between the two variables. The solid line corresponds to  (see Methods for details on the fit). (D) Fraction of uncited papers as a function of . For this and all subsequent panels, solid lines show the predictions of the model using , σ = 0.419, and a value of μ for each  (see Methods). (E) Variance of ℓ as a function of . (F) Skewness of ℓ as a function of . The skewness of the normal distribution is zero. (G) Kurtosis excess of ℓ as a function of . The kurtosis excess of the normal distribution is zero. Note how, for the case of , the moments of the distribution of citations for cited papers deviate significantly from those expected for a normal distribution. In contrast, for , only a small fraction of papers remains uncited, so deviations from the expectations for a normal distribution are small.
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC2244807&req=5

pone-0001683-g002: Modeling the steady-state distributions of the number of citations for papers published in a given journal.(A) Our model assumes that the “quality” of the papers published by a journal obeys a normal distribution with mean μ and standard deviation σ. The number of citations of a paper with quality q∈N(μ,σ) is given by Eq. (3). Because the quality is a continuous variable whereas the number of citations is an integer quantity, the same number of citations will occur for papers with qualities spanning a certain range of q. In particular, all papers for which q<log10(1+γ) will receive no citations. In the panel, the areas of differently shaded regions yield the probability of a paper accruing a given number of citations. (B) Scatter plot of the estimated value of σ versus for all 2,267 journals considered in our analysis (see Methods and Appendices S1 and S4 for details on the fits). Notice that σ is almost independent of . The solid line corresponds to σ = 0.419, the mean of the estimated values of σ for all journals (see Methods). (C) Scatter plot of the estimated value of γ+1 for versus . Notice the strong correlation between the two variables. The solid line corresponds to (see Methods for details on the fit). (D) Fraction of uncited papers as a function of . For this and all subsequent panels, solid lines show the predictions of the model using , σ = 0.419, and a value of μ for each (see Methods). (E) Variance of ℓ as a function of . (F) Skewness of ℓ as a function of . The skewness of the normal distribution is zero. (G) Kurtosis excess of ℓ as a function of . The kurtosis excess of the normal distribution is zero. Note how, for the case of , the moments of the distribution of citations for cited papers deviate significantly from those expected for a normal distribution. In contrast, for , only a small fraction of papers remains uncited, so deviations from the expectations for a normal distribution are small.

Mentions: One can interpret γ as the value of q at which one can expect a paper to get cited once (Figure 2A). More generally, one could write n = floor(10q+ε−γ), where ε∈N(0,σε), to account for external influences to the number of citations. For example, assuming γ = 0 and q = 3, one would get n = 794 for ε = −0.1 and n = 1258 for ε = 0.1. However, if ε is independent of J, will not be significantly affected by ε. Thus, even though the number of citations to individual papers may change, the mean for a journal will not. To demonstrate the agreement between our model and the data, in Figure 2 we plot the moments of the empirical distributions for each journal together with the predictions of our model for those quantities. It is visually apparent that the model provides a close description of the data.


Effectiveness of journal ranking schemes as a tool for locating information.

Stringer MJ, Sales-Pardo M, Nunes Amaral LA - PLoS ONE (2008)

Modeling the steady-state distributions of the number of citations for papers published in a given journal.(A) Our model assumes that the “quality” of the papers published by a journal obeys a normal distribution with mean μ and standard deviation σ. The number of citations of a paper with quality q∈N(μ,σ) is given by Eq. (3). Because the quality is a continuous variable whereas the number of citations is an integer quantity, the same number of citations will occur for papers with qualities spanning a certain range of q. In particular, all papers for which q<log10(1+γ) will receive no citations. In the panel, the areas of differently shaded regions yield the probability of a paper accruing a given number of citations. (B) Scatter plot of the estimated value of σ versus  for all 2,267 journals considered in our analysis (see Methods and Appendices S1 and S4 for details on the fits). Notice that σ is almost independent of . The solid line corresponds to σ = 0.419, the mean of the estimated values of σ for all journals (see Methods). (C) Scatter plot of the estimated value of γ+1 for versus . Notice the strong correlation between the two variables. The solid line corresponds to  (see Methods for details on the fit). (D) Fraction of uncited papers as a function of . For this and all subsequent panels, solid lines show the predictions of the model using , σ = 0.419, and a value of μ for each  (see Methods). (E) Variance of ℓ as a function of . (F) Skewness of ℓ as a function of . The skewness of the normal distribution is zero. (G) Kurtosis excess of ℓ as a function of . The kurtosis excess of the normal distribution is zero. Note how, for the case of , the moments of the distribution of citations for cited papers deviate significantly from those expected for a normal distribution. In contrast, for , only a small fraction of papers remains uncited, so deviations from the expectations for a normal distribution are small.
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC2244807&req=5

pone-0001683-g002: Modeling the steady-state distributions of the number of citations for papers published in a given journal.(A) Our model assumes that the “quality” of the papers published by a journal obeys a normal distribution with mean μ and standard deviation σ. The number of citations of a paper with quality q∈N(μ,σ) is given by Eq. (3). Because the quality is a continuous variable whereas the number of citations is an integer quantity, the same number of citations will occur for papers with qualities spanning a certain range of q. In particular, all papers for which q<log10(1+γ) will receive no citations. In the panel, the areas of differently shaded regions yield the probability of a paper accruing a given number of citations. (B) Scatter plot of the estimated value of σ versus for all 2,267 journals considered in our analysis (see Methods and Appendices S1 and S4 for details on the fits). Notice that σ is almost independent of . The solid line corresponds to σ = 0.419, the mean of the estimated values of σ for all journals (see Methods). (C) Scatter plot of the estimated value of γ+1 for versus . Notice the strong correlation between the two variables. The solid line corresponds to (see Methods for details on the fit). (D) Fraction of uncited papers as a function of . For this and all subsequent panels, solid lines show the predictions of the model using , σ = 0.419, and a value of μ for each (see Methods). (E) Variance of ℓ as a function of . (F) Skewness of ℓ as a function of . The skewness of the normal distribution is zero. (G) Kurtosis excess of ℓ as a function of . The kurtosis excess of the normal distribution is zero. Note how, for the case of , the moments of the distribution of citations for cited papers deviate significantly from those expected for a normal distribution. In contrast, for , only a small fraction of papers remains uncited, so deviations from the expectations for a normal distribution are small.
Mentions: One can interpret γ as the value of q at which one can expect a paper to get cited once (Figure 2A). More generally, one could write n = floor(10q+ε−γ), where ε∈N(0,σε), to account for external influences to the number of citations. For example, assuming γ = 0 and q = 3, one would get n = 794 for ε = −0.1 and n = 1258 for ε = 0.1. However, if ε is independent of J, will not be significantly affected by ε. Thus, even though the number of citations to individual papers may change, the mean for a journal will not. To demonstrate the agreement between our model and the data, in Figure 2 we plot the moments of the empirical distributions for each journal together with the predictions of our model for those quantities. It is visually apparent that the model provides a close description of the data.

Bottom Line: Here, we systematically evaluate the effectiveness of journals, through the work of editors and reviewers, at evaluating unpublished research.Our model enables us to quantify both the typical impact and the range of impacts of papers published in a journal.Finally, we propose a journal-ranking scheme that maximizes the efficiency of locating high impact research.

View Article: PubMed Central - PubMed

Affiliation: Department of Physics and Astronomy, Northwestern University, Evanston, Illinois, USA.

ABSTRACT

Background: The rise of electronic publishing, preprint archives, blogs, and wikis is raising concerns among publishers, editors, and scientists about the present day relevance of academic journals and traditional peer review. These concerns are especially fuelled by the ability of search engines to automatically identify and sort information. It appears that academic journals can only remain relevant if acceptance of research for publication within a journal allows readers to infer immediate, reliable information on the value of that research.

Methodology/principal findings: Here, we systematically evaluate the effectiveness of journals, through the work of editors and reviewers, at evaluating unpublished research. We find that the distribution of the number of citations to a paper published in a given journal in a specific year converges to a steady state after a journal-specific transient time, and demonstrate that in the steady state the logarithm of the number of citations has a journal-specific typical value. We then develop a model for the asymptotic number of citations accrued by papers published in a journal that closely matches the data.

Conclusions/significance: Our model enables us to quantify both the typical impact and the range of impacts of papers published in a journal. Finally, we propose a journal-ranking scheme that maximizes the efficiency of locating high impact research.

Show MeSH
Related in: MedlinePlus