Limits...
Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle-Ottawa Scale and the RTI item bank.

Margulis AV, Pladevall M, Riera-Guardia N, Varas-Lorenzo C, Hazell L, Berkman ND, Viswanathan M, Perez-Gutthann S - Clin Epidemiol (2014)

Bottom Line: The study objective was to compare the Newcastle-Ottawa Scale (NOS) and the RTI item bank (RTI-IB) and estimate interrater agreement using the RTI-IB within a systematic review on the cardiovascular safety of glucose-lowering drugs.Median observed interrater agreement for the RTI-IB was 75% (25th percentile [p25] =61%; p75 =89%); median AC1 statistic was 0.64 (p25 =0.51; p75 =0.86).The observed agreement and AC1 statistic in this study were higher than those reported by the RTI-IB's developers.

View Article: PubMed Central - PubMed

Affiliation: RTI Health Solutions, Barcelona, Spain.

ABSTRACT

Background: The study objective was to compare the Newcastle-Ottawa Scale (NOS) and the RTI item bank (RTI-IB) and estimate interrater agreement using the RTI-IB within a systematic review on the cardiovascular safety of glucose-lowering drugs.

Methods: We tailored both tools and added four questions to the RTI-IB. Two reviewers assessed the quality of the 44 included studies with both tools, (independently for the RTI-IB) and agreed on which responses conveyed low, unclear, or high risk of bias. For each question in the RTI-IB (n=31), the observed interrater agreement was calculated as the percentage of studies given the same bias assessment by both reviewers; chance-adjusted interrater agreement was estimated with the first-order agreement coefficient (AC1) statistic.

Results: The NOS required less tailoring and was easier to use than the RTI-IB, but the RTI-IB produced a more thorough assessment. The RTI-IB includes most of the domains measured in the NOS. Median observed interrater agreement for the RTI-IB was 75% (25th percentile [p25] =61%; p75 =89%); median AC1 statistic was 0.64 (p25 =0.51; p75 =0.86).

Conclusion: The RTI-IB facilitates a more complete quality assessment than the NOS but is more burdensome. The observed agreement and AC1 statistic in this study were higher than those reported by the RTI-IB's developers.

No MeSH data available.


Risk of bias by domain (in bold) and item in all studies (n=44) using the RTI item bank.Notes: Numbers on the bars represent the number of studies with low risk of bias (green bars), unclear risk of bias (yellow bars), or high risk of bias (red bars) over the number of studies for which the item was applicable, and percentages. Due to limited room, some of these numbers were not included. Percentages were calculated over the studies for which the items were applicable. In the first domain, we replaced “Sample” with “Study population,” and removed “Interventions” from the domain “Interventions/exposure.”
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4199858&req=5

f3-clep-6-359: Risk of bias by domain (in bold) and item in all studies (n=44) using the RTI item bank.Notes: Numbers on the bars represent the number of studies with low risk of bias (green bars), unclear risk of bias (yellow bars), or high risk of bias (red bars) over the number of studies for which the item was applicable, and percentages. Due to limited room, some of these numbers were not included. Percentages were calculated over the studies for which the items were applicable. In the first domain, we replaced “Sample” with “Study population,” and removed “Interventions” from the domain “Interventions/exposure.”

Mentions: Results of the application of the RTI item bank cannot be summarized as easily as results from the NOS because there is no recommendation to aggregate RTI item bank results into a summary score. In the extremes, questions on study design and on whether the study outcome had been prespecified showed low risk of bias for 100% of the 44 studies, whereas the question on whether outcome validation had been independent of exposure status showed high risk of bias in 100% of the two studies to which the item was applicable (Figure 3).


Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle-Ottawa Scale and the RTI item bank.

Margulis AV, Pladevall M, Riera-Guardia N, Varas-Lorenzo C, Hazell L, Berkman ND, Viswanathan M, Perez-Gutthann S - Clin Epidemiol (2014)

Risk of bias by domain (in bold) and item in all studies (n=44) using the RTI item bank.Notes: Numbers on the bars represent the number of studies with low risk of bias (green bars), unclear risk of bias (yellow bars), or high risk of bias (red bars) over the number of studies for which the item was applicable, and percentages. Due to limited room, some of these numbers were not included. Percentages were calculated over the studies for which the items were applicable. In the first domain, we replaced “Sample” with “Study population,” and removed “Interventions” from the domain “Interventions/exposure.”
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4199858&req=5

f3-clep-6-359: Risk of bias by domain (in bold) and item in all studies (n=44) using the RTI item bank.Notes: Numbers on the bars represent the number of studies with low risk of bias (green bars), unclear risk of bias (yellow bars), or high risk of bias (red bars) over the number of studies for which the item was applicable, and percentages. Due to limited room, some of these numbers were not included. Percentages were calculated over the studies for which the items were applicable. In the first domain, we replaced “Sample” with “Study population,” and removed “Interventions” from the domain “Interventions/exposure.”
Mentions: Results of the application of the RTI item bank cannot be summarized as easily as results from the NOS because there is no recommendation to aggregate RTI item bank results into a summary score. In the extremes, questions on study design and on whether the study outcome had been prespecified showed low risk of bias for 100% of the 44 studies, whereas the question on whether outcome validation had been independent of exposure status showed high risk of bias in 100% of the two studies to which the item was applicable (Figure 3).

Bottom Line: The study objective was to compare the Newcastle-Ottawa Scale (NOS) and the RTI item bank (RTI-IB) and estimate interrater agreement using the RTI-IB within a systematic review on the cardiovascular safety of glucose-lowering drugs.Median observed interrater agreement for the RTI-IB was 75% (25th percentile [p25] =61%; p75 =89%); median AC1 statistic was 0.64 (p25 =0.51; p75 =0.86).The observed agreement and AC1 statistic in this study were higher than those reported by the RTI-IB's developers.

View Article: PubMed Central - PubMed

Affiliation: RTI Health Solutions, Barcelona, Spain.

ABSTRACT

Background: The study objective was to compare the Newcastle-Ottawa Scale (NOS) and the RTI item bank (RTI-IB) and estimate interrater agreement using the RTI-IB within a systematic review on the cardiovascular safety of glucose-lowering drugs.

Methods: We tailored both tools and added four questions to the RTI-IB. Two reviewers assessed the quality of the 44 included studies with both tools, (independently for the RTI-IB) and agreed on which responses conveyed low, unclear, or high risk of bias. For each question in the RTI-IB (n=31), the observed interrater agreement was calculated as the percentage of studies given the same bias assessment by both reviewers; chance-adjusted interrater agreement was estimated with the first-order agreement coefficient (AC1) statistic.

Results: The NOS required less tailoring and was easier to use than the RTI-IB, but the RTI-IB produced a more thorough assessment. The RTI-IB includes most of the domains measured in the NOS. Median observed interrater agreement for the RTI-IB was 75% (25th percentile [p25] =61%; p75 =89%); median AC1 statistic was 0.64 (p25 =0.51; p75 =0.86).

Conclusion: The RTI-IB facilitates a more complete quality assessment than the NOS but is more burdensome. The observed agreement and AC1 statistic in this study were higher than those reported by the RTI-IB's developers.

No MeSH data available.