Limits...
The quality of research synthesis in surgery: the case of laparoscopic surgery for colorectal cancer.

Martel G, Duhaime S, Barkun JS, Boushey RP, Ramsay CR, Fergusson DA - Syst Rev (2012)

Bottom Line: Reviews were synthesized, and outcomes were compared qualitatively.Overall survival was evaluated by ten reviews, none of which found a significant difference.Further systematic reviews or meta-analyses are unlikely to be justified without specifying a significantly different research objective.

View Article: PubMed Central - HTML - PubMed

Affiliation: Department of Surgery, Department of Epidemiology & Community Medicine, and Ottawa Hospital Research Institute, The Ottawa Hospital, University of Ottawa, Ottawa, Ontario, Canada. guillaume.martel@gmail.com

ABSTRACT

Background: Several systematic reviews and meta-analyses populate the literature on the effectiveness of laparoscopic surgery for colorectal cancer. The utility of this body of work is unclear. The objective of this study was to synthesize all such systematic reviews in terms of clinical effectiveness, to appraise their quality, and to determine whether areas of duplication exist across reviews.

Methods: Systematic reviews comparing laparoscopic and open surgery for colorectal cancer were identified using a comprehensive search protocol (1991 to 2008). The primary outcome was overall survival. The methodological quality of reviews was appraised using the Assessment of Multiple Systematic Reviews (AMSTAR) instrument. Abstraction and quality appraisal was carried out by two independent reviewers. Reviews were synthesized, and outcomes were compared qualitatively. A citation analysis was carried out using simple matrices to assess the comprehensiveness of each review.

Results: In total, 27 reviews were included; 13 reviews included only randomized controlled trials. Rectal cancer was addressed exclusively by four reviews. There was significant overlap between review purposes, populations and, outcomes. The mean AMSTAR score (out of 11) was 5.8 (95% CI: 4.6 to 7.0). Overall survival was evaluated by ten reviews, none of which found a significant difference. Three reviews provided a selective meta-analysis of time-to-event data. Previously published systematic reviews were poorly and highly selectively referenced (mean citation ratio 0.16, 95% CI: 0.093 to 0.22). Previously published trials were not comprehensively identified and cited (mean citation ratio 0.56, 95% CI: 0.46 to 0.65).

Conclusions: Numerous overlapping systematic reviews of laparoscopic and open surgery for colorectal cancer exist in the literature. Despite variable methods and quality, survival outcomes are congruent across reviews. A duplication of research efforts appears to exist in the literature. Further systematic reviews or meta-analyses are unlikely to be justified without specifying a significantly different research objective. This works lends support to the registration and updating of systematic reviews.

Show MeSH

Related in: MedlinePlus

Citation of previously published randomized controlled trials.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC3351744&req=5

Figure 6: Citation of previously published randomized controlled trials.

Mentions: The current overview protocol identified 38 publications pertaining to 23 individual RCTs comparing laparoscopic and open surgery for colorectal cancer [46-83]. The patterns of citations of these RCTs are presented in Figure 6. Two systematic reviews did not cite any RCTs. In the case of Bernard et al. [16], there were no RCTs yet published in the literature. In the case of Yong et al. [20], a total of ten RCT publications could have been cited. The mean ratio of cited to total published RCT reports was 0.45 (95% CI: 0.35 to 0.54), ranging from 0 to 1. Excluding Abraham et al. [36], which sought to analyze only observational studies, yielded a comparable mean ratio of 0.46 (95% CI: 0.36 to 0.56). Given that certain RCTs were published over multiple papers, the ratio of cited to total published RCT reports was recalculated using each trial as the denominator rather than individual papers. This analysis yielded a mean ratio of 0.56 (95% CI: 0.46, 0.65). A total of only four reviews identified at least 75% of RCT publications, of which two were Cochrane Reviews and one was an HTA report [10,26,28,38]. Reviews that selected rectal cancer as their sole patient population had generally low citation ratios, ranging from 0.13 to 0.39 [27,30,31,40]. Among all RCT publications, two were cited disproportionately more frequently than others. Indeed, the Barcelona trial by Lacy et al. [57] and the Clinical Outcomes of Surgical Therapy (COST) trial [63] were both cited by 90% of systematic reviews. Finally, the correlation between the citation ratios and the AMSTAR scores (r = 0.43), and between the ratios and the journal's 5-year impact factors were moderate at best (r = 0.46).


The quality of research synthesis in surgery: the case of laparoscopic surgery for colorectal cancer.

Martel G, Duhaime S, Barkun JS, Boushey RP, Ramsay CR, Fergusson DA - Syst Rev (2012)

Citation of previously published randomized controlled trials.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC3351744&req=5

Figure 6: Citation of previously published randomized controlled trials.
Mentions: The current overview protocol identified 38 publications pertaining to 23 individual RCTs comparing laparoscopic and open surgery for colorectal cancer [46-83]. The patterns of citations of these RCTs are presented in Figure 6. Two systematic reviews did not cite any RCTs. In the case of Bernard et al. [16], there were no RCTs yet published in the literature. In the case of Yong et al. [20], a total of ten RCT publications could have been cited. The mean ratio of cited to total published RCT reports was 0.45 (95% CI: 0.35 to 0.54), ranging from 0 to 1. Excluding Abraham et al. [36], which sought to analyze only observational studies, yielded a comparable mean ratio of 0.46 (95% CI: 0.36 to 0.56). Given that certain RCTs were published over multiple papers, the ratio of cited to total published RCT reports was recalculated using each trial as the denominator rather than individual papers. This analysis yielded a mean ratio of 0.56 (95% CI: 0.46, 0.65). A total of only four reviews identified at least 75% of RCT publications, of which two were Cochrane Reviews and one was an HTA report [10,26,28,38]. Reviews that selected rectal cancer as their sole patient population had generally low citation ratios, ranging from 0.13 to 0.39 [27,30,31,40]. Among all RCT publications, two were cited disproportionately more frequently than others. Indeed, the Barcelona trial by Lacy et al. [57] and the Clinical Outcomes of Surgical Therapy (COST) trial [63] were both cited by 90% of systematic reviews. Finally, the correlation between the citation ratios and the AMSTAR scores (r = 0.43), and between the ratios and the journal's 5-year impact factors were moderate at best (r = 0.46).

Bottom Line: Reviews were synthesized, and outcomes were compared qualitatively.Overall survival was evaluated by ten reviews, none of which found a significant difference.Further systematic reviews or meta-analyses are unlikely to be justified without specifying a significantly different research objective.

View Article: PubMed Central - HTML - PubMed

Affiliation: Department of Surgery, Department of Epidemiology & Community Medicine, and Ottawa Hospital Research Institute, The Ottawa Hospital, University of Ottawa, Ottawa, Ontario, Canada. guillaume.martel@gmail.com

ABSTRACT

Background: Several systematic reviews and meta-analyses populate the literature on the effectiveness of laparoscopic surgery for colorectal cancer. The utility of this body of work is unclear. The objective of this study was to synthesize all such systematic reviews in terms of clinical effectiveness, to appraise their quality, and to determine whether areas of duplication exist across reviews.

Methods: Systematic reviews comparing laparoscopic and open surgery for colorectal cancer were identified using a comprehensive search protocol (1991 to 2008). The primary outcome was overall survival. The methodological quality of reviews was appraised using the Assessment of Multiple Systematic Reviews (AMSTAR) instrument. Abstraction and quality appraisal was carried out by two independent reviewers. Reviews were synthesized, and outcomes were compared qualitatively. A citation analysis was carried out using simple matrices to assess the comprehensiveness of each review.

Results: In total, 27 reviews were included; 13 reviews included only randomized controlled trials. Rectal cancer was addressed exclusively by four reviews. There was significant overlap between review purposes, populations and, outcomes. The mean AMSTAR score (out of 11) was 5.8 (95% CI: 4.6 to 7.0). Overall survival was evaluated by ten reviews, none of which found a significant difference. Three reviews provided a selective meta-analysis of time-to-event data. Previously published systematic reviews were poorly and highly selectively referenced (mean citation ratio 0.16, 95% CI: 0.093 to 0.22). Previously published trials were not comprehensively identified and cited (mean citation ratio 0.56, 95% CI: 0.46 to 0.65).

Conclusions: Numerous overlapping systematic reviews of laparoscopic and open surgery for colorectal cancer exist in the literature. Despite variable methods and quality, survival outcomes are congruent across reviews. A duplication of research efforts appears to exist in the literature. Further systematic reviews or meta-analyses are unlikely to be justified without specifying a significantly different research objective. This works lends support to the registration and updating of systematic reviews.

Show MeSH
Related in: MedlinePlus