Limits...
Reproducibility of Search Strategies Is Poor in Systematic Reviews Published in High-Impact Pediatrics, Cardiology and Surgery Journals: A Cross-Sectional Study

View Article: PubMed Central - PubMed

ABSTRACT

Background: A high-quality search strategy is considered an essential component of systematic reviews but many do not contain reproducible search strategies. It is unclear if low reproducibility spans medical disciplines, is affected by librarian/search specialist involvement or has improved with increased awareness of reporting guidelines.

Objectives: To examine the reporting of search strategies in systematic reviews published in Pediatrics, Surgery or Cardiology journals in 2012 and determine rates and predictors of including a reproducible search strategy.

Methods: We identified all systematic reviews published in 2012 in the ten highest impact factor journals in Pediatrics, Surgery and Cardiology. Each search strategy was coded to indicate what elements were reported and whether the overall search was reproducible. Reporting and reproducibility rates were compared across disciplines and we measured the influence of librarian/search specialist involvement, discipline or endorsement of a reporting guideline on search reproducibility.

Results: 272 articles from 25 journals were included. Reporting of search elements ranged widely from 91% of articles naming search terms to 33% providing a full search strategy and 22% indicating the date the search was executed. Only 22% of articles provided at least one reproducible search strategy and 13% provided a reproducible strategy for all databases searched in the article. Librarians or search specialists were reported as involved in 17% of articles. There were strong disciplinary differences on the reporting of search elements. In the multivariable analysis, only discipline (Pediatrics) was a significant predictor of the inclusion of a reproducible search strategy.

Conclusions: Despite recommendations to report full, reproducible search strategies, many articles still do not. In addition, authors often report a single strategy as covering all databases searched, further decreasing reproducibility. Further research is needed to determine how disciplinary culture may encourage reproducibility and the role that journal editors and peer reviewers could play.

No MeSH data available.


Flow Chart of Included Studies.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC5036875&req=5

pone.0163309.g001: Flow Chart of Included Studies.

Mentions: We searched PubMed on September 1, 2013 and retrieved 765 articles which were added to PubMed between January and December of 2012 from the candidate journals. We independently reviewed the abstract and full-text of each article and identified 272 systematic reviews (available at http://z.umn.edu/15qt) (Fig 1) published in 25 journals which met our inclusion criteria (Table 1). No systematic reviews could be identified from 2012 in 5 of the 30 original candidate journals. Only 12/25 journals (48%) identified or endorsed a reporting guideline in their instructions to authors. We were able to identify instructions to authors for 11 of the candidate journals using the Wayback Machine (9 of the 25 with included articles). In one case (American Journal of Transplantation), the instructions to authors were from 2011 and thus the Wayback Machine was not consulted. While in most cases the review of instructions to authors from November 2013 and the Wayback Machine agreed, in one case (Archives of Surgery/JAMA Surgery), we discovered that instructions to follow PRISMA/MOOSE reporting guidelines were added in 2013. In this case, we counted JAMA Surgery as not endorsing a reporting guideline in 2012.


Reproducibility of Search Strategies Is Poor in Systematic Reviews Published in High-Impact Pediatrics, Cardiology and Surgery Journals: A Cross-Sectional Study
Flow Chart of Included Studies.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC5036875&req=5

pone.0163309.g001: Flow Chart of Included Studies.
Mentions: We searched PubMed on September 1, 2013 and retrieved 765 articles which were added to PubMed between January and December of 2012 from the candidate journals. We independently reviewed the abstract and full-text of each article and identified 272 systematic reviews (available at http://z.umn.edu/15qt) (Fig 1) published in 25 journals which met our inclusion criteria (Table 1). No systematic reviews could be identified from 2012 in 5 of the 30 original candidate journals. Only 12/25 journals (48%) identified or endorsed a reporting guideline in their instructions to authors. We were able to identify instructions to authors for 11 of the candidate journals using the Wayback Machine (9 of the 25 with included articles). In one case (American Journal of Transplantation), the instructions to authors were from 2011 and thus the Wayback Machine was not consulted. While in most cases the review of instructions to authors from November 2013 and the Wayback Machine agreed, in one case (Archives of Surgery/JAMA Surgery), we discovered that instructions to follow PRISMA/MOOSE reporting guidelines were added in 2013. In this case, we counted JAMA Surgery as not endorsing a reporting guideline in 2012.

View Article: PubMed Central - PubMed

ABSTRACT

Background: A high-quality search strategy is considered an essential component of systematic reviews but many do not contain reproducible search strategies. It is unclear if low reproducibility spans medical disciplines, is affected by librarian/search specialist involvement or has improved with increased awareness of reporting guidelines.

Objectives: To examine the reporting of search strategies in systematic reviews published in Pediatrics, Surgery or Cardiology journals in 2012 and determine rates and predictors of including a reproducible search strategy.

Methods: We identified all systematic reviews published in 2012 in the ten highest impact factor journals in Pediatrics, Surgery and Cardiology. Each search strategy was coded to indicate what elements were reported and whether the overall search was reproducible. Reporting and reproducibility rates were compared across disciplines and we measured the influence of librarian/search specialist involvement, discipline or endorsement of a reporting guideline on search reproducibility.

Results: 272 articles from 25 journals were included. Reporting of search elements ranged widely from 91% of articles naming search terms to 33% providing a full search strategy and 22% indicating the date the search was executed. Only 22% of articles provided at least one reproducible search strategy and 13% provided a reproducible strategy for all databases searched in the article. Librarians or search specialists were reported as involved in 17% of articles. There were strong disciplinary differences on the reporting of search elements. In the multivariable analysis, only discipline (Pediatrics) was a significant predictor of the inclusion of a reproducible search strategy.

Conclusions: Despite recommendations to report full, reproducible search strategies, many articles still do not. In addition, authors often report a single strategy as covering all databases searched, further decreasing reproducibility. Further research is needed to determine how disciplinary culture may encourage reproducibility and the role that journal editors and peer reviewers could play.

No MeSH data available.