Limits...
Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma.

Hartling L, Bond K, Vandermeer B, Seida J, Dryden DM, Rowe BH - PLoS ONE (2011)

Bottom Line: Mean differences in effect sizes for RCTs with different RoB were calculated using inverse variance method and random effects model.Analyses comparing effect estimates and risk showed no important patterns.The extensive involvement of the pharmaceutical industry in this LABA/ICS research should raise concerns about potential overestimates of treatment effects.

View Article: PubMed Central - PubMed

Affiliation: Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, Alberta, Canada. hartling@ualberta.ca

ABSTRACT

Background: The Risk of Bias (RoB) tool is used to assess internal validity of randomized controlled trials (RCTs). Our objectives were to: 1) evaluate inter-rater agreement of the RoB tool; 2) determine the time to access supplemental study information; 3) compare the RoB tool with the Jadad scale and Schulz allocation concealment (AC); and 4) examine the relationship between RoB and effect estimates.

Methods: We conducted a systematic review of long-acting beta agonists (LABA) combined with inhaled corticosteroids (ICS) for adults with persistent asthma. Two reviewers independently assessed 107 trials using RoB, Jadad, and AC. One reviewer searched for study protocols. We assessed inter-rater agreement using weighted Kappa (κ) and the correlation between tools using Kendall's Tau (τ). Mean differences in effect sizes for RCTs with different RoB were calculated using inverse variance method and random effects model.

Results: Trials had good Jadad scores (median 4, IQR 3-4); however, 85% had unclear AC and 87% high RoB. The factor that most influenced RoB was the potential inappropriate influence of study sponsors (95% industry funded). Agreement on RoB domains was fair (κ = 0.40) to almost perfect (κ = 0.86), and moderate for overall RoB (κ = 0.41). Median time to complete RoB assessments was 21 minutes (IQR 14-27) and 12 minutes (IQR 9-16) to search for protocols. Protocols were identified for 5/42 studies (12%); in 3 cases the assessment of selective outcome reporting changed. There was low correlation between overall RoB vs. Jadad (τ = 0.04, p = 0.3) and AC (τ = -0.02, p = 0.7). Analyses comparing effect estimates and risk showed no important patterns.

Conclusions: Inter-rater agreement on RoB assessments was better than previously reported suggesting that review-specific guidelines are important. The correlation between RoB and Jadad was low suggesting measurement of different constructs (risk of bias vs. quality of reporting). The extensive involvement of the pharmaceutical industry in this LABA/ICS research should raise concerns about potential overestimates of treatment effects.

Show MeSH

Related in: MedlinePlus

Mean differences in effect estimates for symptom-free days across domains and overall risk of bias (with and without “other” sources of bias), high vs. low/unclear risk of bias.(Note: TE  =  treatment effect. Components were not estimable when all studies were rated the same [i.e., all high or all low/unclear risk of bias]).
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC3044729&req=5

pone-0017242-g004: Mean differences in effect estimates for symptom-free days across domains and overall risk of bias (with and without “other” sources of bias), high vs. low/unclear risk of bias.(Note: TE  =  treatment effect. Components were not estimable when all studies were rated the same [i.e., all high or all low/unclear risk of bias]).

Mentions: Figures 1 to 4 show the differences in effect estimates for studies at different risk of bias for each of the domains in the RoB tool as well as overall risk of bias, both with and without “other” sources of bias. Figures 1 and 3 compare high or unclear vs. low risk of bias, while Figures 2 and 4 compare high vs. unclear or low risk of bias. There were few notable differences observed which may be due to the homogeneity of study results in this review and small differences in effects within the original meta-analyses [4]. The one difference observed that may be of clinical importance is larger treatment effects for trials at high or unclear risk compared to those at low risk with a difference of 12 symptom-free days (Figure 3, other sources of bias).


Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma.

Hartling L, Bond K, Vandermeer B, Seida J, Dryden DM, Rowe BH - PLoS ONE (2011)

Mean differences in effect estimates for symptom-free days across domains and overall risk of bias (with and without “other” sources of bias), high vs. low/unclear risk of bias.(Note: TE  =  treatment effect. Components were not estimable when all studies were rated the same [i.e., all high or all low/unclear risk of bias]).
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC3044729&req=5

pone-0017242-g004: Mean differences in effect estimates for symptom-free days across domains and overall risk of bias (with and without “other” sources of bias), high vs. low/unclear risk of bias.(Note: TE  =  treatment effect. Components were not estimable when all studies were rated the same [i.e., all high or all low/unclear risk of bias]).
Mentions: Figures 1 to 4 show the differences in effect estimates for studies at different risk of bias for each of the domains in the RoB tool as well as overall risk of bias, both with and without “other” sources of bias. Figures 1 and 3 compare high or unclear vs. low risk of bias, while Figures 2 and 4 compare high vs. unclear or low risk of bias. There were few notable differences observed which may be due to the homogeneity of study results in this review and small differences in effects within the original meta-analyses [4]. The one difference observed that may be of clinical importance is larger treatment effects for trials at high or unclear risk compared to those at low risk with a difference of 12 symptom-free days (Figure 3, other sources of bias).

Bottom Line: Mean differences in effect sizes for RCTs with different RoB were calculated using inverse variance method and random effects model.Analyses comparing effect estimates and risk showed no important patterns.The extensive involvement of the pharmaceutical industry in this LABA/ICS research should raise concerns about potential overestimates of treatment effects.

View Article: PubMed Central - PubMed

Affiliation: Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, Alberta, Canada. hartling@ualberta.ca

ABSTRACT

Background: The Risk of Bias (RoB) tool is used to assess internal validity of randomized controlled trials (RCTs). Our objectives were to: 1) evaluate inter-rater agreement of the RoB tool; 2) determine the time to access supplemental study information; 3) compare the RoB tool with the Jadad scale and Schulz allocation concealment (AC); and 4) examine the relationship between RoB and effect estimates.

Methods: We conducted a systematic review of long-acting beta agonists (LABA) combined with inhaled corticosteroids (ICS) for adults with persistent asthma. Two reviewers independently assessed 107 trials using RoB, Jadad, and AC. One reviewer searched for study protocols. We assessed inter-rater agreement using weighted Kappa (κ) and the correlation between tools using Kendall's Tau (τ). Mean differences in effect sizes for RCTs with different RoB were calculated using inverse variance method and random effects model.

Results: Trials had good Jadad scores (median 4, IQR 3-4); however, 85% had unclear AC and 87% high RoB. The factor that most influenced RoB was the potential inappropriate influence of study sponsors (95% industry funded). Agreement on RoB domains was fair (κ = 0.40) to almost perfect (κ = 0.86), and moderate for overall RoB (κ = 0.41). Median time to complete RoB assessments was 21 minutes (IQR 14-27) and 12 minutes (IQR 9-16) to search for protocols. Protocols were identified for 5/42 studies (12%); in 3 cases the assessment of selective outcome reporting changed. There was low correlation between overall RoB vs. Jadad (τ = 0.04, p = 0.3) and AC (τ = -0.02, p = 0.7). Analyses comparing effect estimates and risk showed no important patterns.

Conclusions: Inter-rater agreement on RoB assessments was better than previously reported suggesting that review-specific guidelines are important. The correlation between RoB and Jadad was low suggesting measurement of different constructs (risk of bias vs. quality of reporting). The extensive involvement of the pharmaceutical industry in this LABA/ICS research should raise concerns about potential overestimates of treatment effects.

Show MeSH
Related in: MedlinePlus