Limits...
Peer Review Evaluation Process of Marie Curie Actions under EU's Seventh Framework Programme for Research.

Pina DG, Hren D, Marušić A - PLoS ONE (2015)

Bottom Line: For proposals where one rater disagreed with the other two raters (n=1424; 5.7%), or where all 3 raters disagreed (n=2075; 8.3%), the average IER and CR scores were still highly associated.CR scores for proposals with initial disagreement were also significantly lower.Consensus meetings appear to be relevant for particular panels and subsets of proposals with large differences among raters' scores.

View Article: PubMed Central - PubMed

Affiliation: Research Executive Agency, European Commission, Brussels, Belgium.

ABSTRACT
We analysed the peer review of grant proposals under Marie Curie Actions, a major EU research funding instrument, which involves two steps: an independent assessment (Individual Evaluation Report, IER) performed remotely by 3 raters, and a consensus opinion reached during a meeting by the same raters (Consensus Report, CR). For 24,897 proposals evaluated from 2007 to 2013, the association between average IER and CR scores was very high across different panels, grant calls and years. Median average deviation (AD) index, used as a measure of inter-rater agreement, was 5.4 points on a 0-100 scale (interquartile range 3.4-8.3), overall, demonstrating a good general agreement among raters. For proposals where one rater disagreed with the other two raters (n=1424; 5.7%), or where all 3 raters disagreed (n=2075; 8.3%), the average IER and CR scores were still highly associated. Disagreement was more frequent for proposals from Economics/Social Sciences and Humanities panels. Greater disagreement was observed for proposals with lower average IER scores. CR scores for proposals with initial disagreement were also significantly lower. Proposals with a large absolute difference between the average IER and CR scores (≥10 points; n=368, 1.5%) generally had lower CR scores. An inter-correlation matrix of individual raters' scores of evaluation criteria of proposals indicated that these scores were, in general, a reflection of raters' overall scores. Our analysis demonstrated a good internal consistency and general high agreement among raters. Consensus meetings appear to be relevant for particular panels and subsets of proposals with large differences among raters' scores.

No MeSH data available.


Related in: MedlinePlus

Association between raters' agreement (AD index—lower score means greater agreement) and the average Individual Evaluation Report (AVIER) (A) or consensus report (CR) (B).Line indicates the regression line. Circles—individual proposals.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4488366&req=5

pone.0130753.g002: Association between raters' agreement (AD index—lower score means greater agreement) and the average Individual Evaluation Report (AVIER) (A) or consensus report (CR) (B).Line indicates the regression line. Circles—individual proposals.

Mentions: To assess the relationship between AD indices as a measure of inter-rater agreement and the scores of the proposals, we first performed a simple linear regression analysis with CR or AVIER scores as a dependent and AD index as a predictor variable. Both measures were significantly and negatively associated with AD index (AVIER = 86.4–1.1×AD, R2 = 0.19, p<0.001 and CR = 86.3–1.0×AD, R2 = 0.15, p<0.001; Fig 2), indicating more disagreement for proposals with lower scores. We found statistically significant differences in CR scores depending on the initial agreement (Table 1) and grouped together the proposals where either one rater differed from the other two who agreed or where all of them differed among themselves and compared this subgroup to all other proposals. CR scores were on average 9 points lower in cases where raters initially disagreed (no initial disagreement: M±SD = 81.0±10.1 vs. initial disagreement: M±SD = 72.3±13.0; p<0.001, independent samples t-test).


Peer Review Evaluation Process of Marie Curie Actions under EU's Seventh Framework Programme for Research.

Pina DG, Hren D, Marušić A - PLoS ONE (2015)

Association between raters' agreement (AD index—lower score means greater agreement) and the average Individual Evaluation Report (AVIER) (A) or consensus report (CR) (B).Line indicates the regression line. Circles—individual proposals.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4488366&req=5

pone.0130753.g002: Association between raters' agreement (AD index—lower score means greater agreement) and the average Individual Evaluation Report (AVIER) (A) or consensus report (CR) (B).Line indicates the regression line. Circles—individual proposals.
Mentions: To assess the relationship between AD indices as a measure of inter-rater agreement and the scores of the proposals, we first performed a simple linear regression analysis with CR or AVIER scores as a dependent and AD index as a predictor variable. Both measures were significantly and negatively associated with AD index (AVIER = 86.4–1.1×AD, R2 = 0.19, p<0.001 and CR = 86.3–1.0×AD, R2 = 0.15, p<0.001; Fig 2), indicating more disagreement for proposals with lower scores. We found statistically significant differences in CR scores depending on the initial agreement (Table 1) and grouped together the proposals where either one rater differed from the other two who agreed or where all of them differed among themselves and compared this subgroup to all other proposals. CR scores were on average 9 points lower in cases where raters initially disagreed (no initial disagreement: M±SD = 81.0±10.1 vs. initial disagreement: M±SD = 72.3±13.0; p<0.001, independent samples t-test).

Bottom Line: For proposals where one rater disagreed with the other two raters (n=1424; 5.7%), or where all 3 raters disagreed (n=2075; 8.3%), the average IER and CR scores were still highly associated.CR scores for proposals with initial disagreement were also significantly lower.Consensus meetings appear to be relevant for particular panels and subsets of proposals with large differences among raters' scores.

View Article: PubMed Central - PubMed

Affiliation: Research Executive Agency, European Commission, Brussels, Belgium.

ABSTRACT
We analysed the peer review of grant proposals under Marie Curie Actions, a major EU research funding instrument, which involves two steps: an independent assessment (Individual Evaluation Report, IER) performed remotely by 3 raters, and a consensus opinion reached during a meeting by the same raters (Consensus Report, CR). For 24,897 proposals evaluated from 2007 to 2013, the association between average IER and CR scores was very high across different panels, grant calls and years. Median average deviation (AD) index, used as a measure of inter-rater agreement, was 5.4 points on a 0-100 scale (interquartile range 3.4-8.3), overall, demonstrating a good general agreement among raters. For proposals where one rater disagreed with the other two raters (n=1424; 5.7%), or where all 3 raters disagreed (n=2075; 8.3%), the average IER and CR scores were still highly associated. Disagreement was more frequent for proposals from Economics/Social Sciences and Humanities panels. Greater disagreement was observed for proposals with lower average IER scores. CR scores for proposals with initial disagreement were also significantly lower. Proposals with a large absolute difference between the average IER and CR scores (≥10 points; n=368, 1.5%) generally had lower CR scores. An inter-correlation matrix of individual raters' scores of evaluation criteria of proposals indicated that these scores were, in general, a reflection of raters' overall scores. Our analysis demonstrated a good internal consistency and general high agreement among raters. Consensus meetings appear to be relevant for particular panels and subsets of proposals with large differences among raters' scores.

No MeSH data available.


Related in: MedlinePlus