Limits...
Accuracy of the Berger-Exner test for detecting third-order selection bias in randomised controlled trials: a simulation-based investigation.

Mickenautsch S, Fu B, Gudehithlu S, Berger VW - BMC Med Res Methodol (2014)

Bottom Line: The test was applied in both scenarios and the pooled sensitivity and specificity, with 95% confidence intervals for alpha levels of 1%, 5%, and 20%, were computed.An effect size inflation of 71% - 99% was established.Test sensitivity was 1.00 (95% CI: 0.99 - 1.00) for alpha level 1%, 5%, and 20%; test specificity was 0.94 (95% CI: 0.93 - 0.96); 0.82 (95% CI: 0.80 - 0.84), and 0.56 (95% CI: 0.54 - 0.58) for alpha 1%, 5%, and 20%, respectively.

View Article: PubMed Central - PubMed

Affiliation: SYSTEM Initiative/Department of Community Dentistry, Faculty of Health Sciences, University of the Witwatersrand, 7 York Rd,, Parktown, Johannesburg 2193, South Africa. neem@global.co.za.

ABSTRACT

Background: Randomised controlled trials (RCT) are highly influential upon medical decisions. Thus RCTs must not distort the truth. One threat to internal trial validity is the correct prediction of future allocations (selection bias). The Berger-Exner test detects such bias but has not been widely utilized in practice. One reason for this non-utilisation may be a lack of information regarding its test accuracy. The objective of this study is to assess the accuracy of the Berger-Exner test on the basis of relevant simulations for RCTs with dichotomous outcomes.

Methods: Simulated RCTs with various parameter settings were generated, using R software, and subjected to bias-free and selection bias scenarios. The effect size inflation due to bias was quantified. The test was applied in both scenarios and the pooled sensitivity and specificity, with 95% confidence intervals for alpha levels of 1%, 5%, and 20%, were computed. Summary ROC curves were generated and the relationships of parameters with test accuracy were explored.

Results: An effect size inflation of 71% - 99% was established. Test sensitivity was 1.00 (95% CI: 0.99 - 1.00) for alpha level 1%, 5%, and 20%; test specificity was 0.94 (95% CI: 0.93 - 0.96); 0.82 (95% CI: 0.80 - 0.84), and 0.56 (95% CI: 0.54 - 0.58) for alpha 1%, 5%, and 20%, respectively. Test accuracy was best with the maximal procedure used with a maximum tolerated imbalance (MTI) = 2 as the randomisation method at alpha 1%.

Conclusions: The results of this simulation study suggest that the Berger-Exner test is generally accurate for identifying third-order selection bias.

No MeSH data available.


Related in: MedlinePlus

Summary Receiver Operating Characteristic (SROC) curve concerning test accuracy at alpha level 20%. AUC = Area under curve; SE = Standard error.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC4209086&req=5

Fig3: Summary Receiver Operating Characteristic (SROC) curve concerning test accuracy at alpha level 20%. AUC = Area under curve; SE = Standard error.

Mentions: Computation of the pooled test sensitivity and specificity was based on the TP/FN and TN/FP data from the 63 separate parameter sets (Table 1, Additional file 2: Appendix file 2), per alpha level. The pooled sensitivity was 1.00 (95% CI: 0.99 – 1.00) for alpha level 1%, 5% and 20%. The pooled test specificity was 0.94 (95% CI: 0.93 – 0.96) for alpha level 1% and 0.82 (95% CI: 0.80 – 0.84) and 0.56 (95% CI: 0.54 – 0.58) for alpha 5% and 20%, respectively. The generated SROC curves (Figures 1, 2, 3) indicated highest overall test accuracy when alpha was set at 1%.Figure 1


Accuracy of the Berger-Exner test for detecting third-order selection bias in randomised controlled trials: a simulation-based investigation.

Mickenautsch S, Fu B, Gudehithlu S, Berger VW - BMC Med Res Methodol (2014)

Summary Receiver Operating Characteristic (SROC) curve concerning test accuracy at alpha level 20%. AUC = Area under curve; SE = Standard error.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC4209086&req=5

Fig3: Summary Receiver Operating Characteristic (SROC) curve concerning test accuracy at alpha level 20%. AUC = Area under curve; SE = Standard error.
Mentions: Computation of the pooled test sensitivity and specificity was based on the TP/FN and TN/FP data from the 63 separate parameter sets (Table 1, Additional file 2: Appendix file 2), per alpha level. The pooled sensitivity was 1.00 (95% CI: 0.99 – 1.00) for alpha level 1%, 5% and 20%. The pooled test specificity was 0.94 (95% CI: 0.93 – 0.96) for alpha level 1% and 0.82 (95% CI: 0.80 – 0.84) and 0.56 (95% CI: 0.54 – 0.58) for alpha 5% and 20%, respectively. The generated SROC curves (Figures 1, 2, 3) indicated highest overall test accuracy when alpha was set at 1%.Figure 1

Bottom Line: The test was applied in both scenarios and the pooled sensitivity and specificity, with 95% confidence intervals for alpha levels of 1%, 5%, and 20%, were computed.An effect size inflation of 71% - 99% was established.Test sensitivity was 1.00 (95% CI: 0.99 - 1.00) for alpha level 1%, 5%, and 20%; test specificity was 0.94 (95% CI: 0.93 - 0.96); 0.82 (95% CI: 0.80 - 0.84), and 0.56 (95% CI: 0.54 - 0.58) for alpha 1%, 5%, and 20%, respectively.

View Article: PubMed Central - PubMed

Affiliation: SYSTEM Initiative/Department of Community Dentistry, Faculty of Health Sciences, University of the Witwatersrand, 7 York Rd,, Parktown, Johannesburg 2193, South Africa. neem@global.co.za.

ABSTRACT

Background: Randomised controlled trials (RCT) are highly influential upon medical decisions. Thus RCTs must not distort the truth. One threat to internal trial validity is the correct prediction of future allocations (selection bias). The Berger-Exner test detects such bias but has not been widely utilized in practice. One reason for this non-utilisation may be a lack of information regarding its test accuracy. The objective of this study is to assess the accuracy of the Berger-Exner test on the basis of relevant simulations for RCTs with dichotomous outcomes.

Methods: Simulated RCTs with various parameter settings were generated, using R software, and subjected to bias-free and selection bias scenarios. The effect size inflation due to bias was quantified. The test was applied in both scenarios and the pooled sensitivity and specificity, with 95% confidence intervals for alpha levels of 1%, 5%, and 20%, were computed. Summary ROC curves were generated and the relationships of parameters with test accuracy were explored.

Results: An effect size inflation of 71% - 99% was established. Test sensitivity was 1.00 (95% CI: 0.99 - 1.00) for alpha level 1%, 5%, and 20%; test specificity was 0.94 (95% CI: 0.93 - 0.96); 0.82 (95% CI: 0.80 - 0.84), and 0.56 (95% CI: 0.54 - 0.58) for alpha 1%, 5%, and 20%, respectively. Test accuracy was best with the maximal procedure used with a maximum tolerated imbalance (MTI) = 2 as the randomisation method at alpha 1%.

Conclusions: The results of this simulation study suggest that the Berger-Exner test is generally accurate for identifying third-order selection bias.

No MeSH data available.


Related in: MedlinePlus