Limits...
Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol.

da Costa BR, Resta NM, Beckett B, Israel-Stahre N, Diaz A, Johnston BC, Egger M, Jüni P, Armijo-Olivo S - Syst Rev (2014)

Bottom Line: This is a pilot study.To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals.If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

View Article: PubMed Central - PubMed

Affiliation: Department of Physical Therapy, Florida International University, AHC3-430 11200 8 St, Miami, FL, USA. bdacosta@fiu.edu.

ABSTRACT

Background: The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers.

Methods/design: This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals.

Discussion: This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

Show MeSH
Search strategy. *Search strategy conducted on PubMed (http://www.ncbi.nlm.nih.gov/pubmed). The search was last updated on 20 March 2014.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC4273317&req=5

Fig1: Search strategy. *Search strategy conducted on PubMed (http://www.ncbi.nlm.nih.gov/pubmed). The search was last updated on 20 March 2014.

Mentions: We will search PubMed from inception using database-specific search strategy (Figure 1). We will include every randomized or quasi-randomized clinical trial in patients with knee osteoarthritis that compared a physical therapy intervention to another physical therapy intervention, sham intervention, or no treatment, which assessed patient-reported pain. The following physical therapy interventions will be considered: land-based exercise, aquatic exercise, manual therapy, electric stimulation therapy, and diathermy. We will only consider studies published in English. No further restrictions will be applied. Two raters will screen reports for eligibility independently in duplicate. Disagreements will be resolved by a senior author (BdC).Figure 1


Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol.

da Costa BR, Resta NM, Beckett B, Israel-Stahre N, Diaz A, Johnston BC, Egger M, Jüni P, Armijo-Olivo S - Syst Rev (2014)

Search strategy. *Search strategy conducted on PubMed (http://www.ncbi.nlm.nih.gov/pubmed). The search was last updated on 20 March 2014.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC4273317&req=5

Fig1: Search strategy. *Search strategy conducted on PubMed (http://www.ncbi.nlm.nih.gov/pubmed). The search was last updated on 20 March 2014.
Mentions: We will search PubMed from inception using database-specific search strategy (Figure 1). We will include every randomized or quasi-randomized clinical trial in patients with knee osteoarthritis that compared a physical therapy intervention to another physical therapy intervention, sham intervention, or no treatment, which assessed patient-reported pain. The following physical therapy interventions will be considered: land-based exercise, aquatic exercise, manual therapy, electric stimulation therapy, and diathermy. We will only consider studies published in English. No further restrictions will be applied. Two raters will screen reports for eligibility independently in duplicate. Disagreements will be resolved by a senior author (BdC).Figure 1

Bottom Line: This is a pilot study.To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals.If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

View Article: PubMed Central - PubMed

Affiliation: Department of Physical Therapy, Florida International University, AHC3-430 11200 8 St, Miami, FL, USA. bdacosta@fiu.edu.

ABSTRACT

Background: The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers.

Methods/design: This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals.

Discussion: This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

Show MeSH