Limits...
A manually curated ChIP-seq benchmark demonstrates room for improvement in current peak-finder programs.

Rye MB, Sætrom P, Drabløs F - Nucleic Acids Res. (2010)

Bottom Line: Second, none of the programs returned peak-regions that corresponded to the actual resolution in ChIP-seq data.Our results showed that ChIP-seq peaks should be narrowed down to 100-400 bp, which is sufficient to identify unique peaks and binding sites.Based on these results, we propose a meta-approach that gives improved peak definitions.

View Article: PubMed Central - PubMed

Affiliation: Department of Cancer Research and Molecular Medicine, Norwegian University of Science and Technology, NO-7489 Trondheim, Norway. morten.rye@ntnu.no

ABSTRACT
Chromatin immunoprecipitation (ChIP) followed by high throughput sequencing (ChIP-seq) is rapidly becoming the method of choice for discovering cell-specific transcription factor binding locations genome wide. By aligning sequenced tags to the genome, binding locations appear as peaks in the tag profile. Several programs have been designed to identify such peaks, but program evaluation has been difficult due to the lack of benchmark data sets. We have created benchmark data sets for three transcription factors by manually evaluating a selection of potential binding regions that cover typical variation in peak size and appearance. Performance of five programs on this benchmark showed, first, that external control or background data was essential to limit the number of false positive peaks from the programs. However, >80% of these peaks could be manually filtered out by visual inspection alone, without using additional background data, showing that peak shape information is not fully exploited in the evaluated programs. Second, none of the programs returned peak-regions that corresponded to the actual resolution in ChIP-seq data. Our results showed that ChIP-seq peaks should be narrowed down to 100-400 bp, which is sufficient to identify unique peaks and binding sites. Based on these results, we propose a meta-approach that gives improved peak definitions.

Show MeSH

Related in: MedlinePlus

New peaks identified for (A) NRSF and (B) SRF in deeply sequenced sets, compared to random subsets. The bars show the number of positive, ambiguous and negative peaks found by MACS (M), MACS with background (Mc), SISSRs (S) and SISSRs with background (Sc), together with the manual evaluation reference (Ev).
© Copyright Policy - creative-commons
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC3045577&req=5

Figure 3: New peaks identified for (A) NRSF and (B) SRF in deeply sequenced sets, compared to random subsets. The bars show the number of positive, ambiguous and negative peaks found by MACS (M), MACS with background (Mc), SISSRs (S) and SISSRs with background (Sc), together with the manual evaluation reference (Ev).

Mentions: As high-throughput sequencing becomes more common, the number of deeply sequenced data sets together with an extensive use of replicates is expected to increase. However, how this may influence the identification of peaks and the performance of peak-finder programs is not clear (12,19). The idea behind using more sequence data is to identify more weak binding sites while making stronger sites appear more pronounced. Running the programs on the more deeply sequenced data sets did indeed produce more peaks compared to the smaller randomly sampled subsets, which is in accordance with previous studies (12,19). The question is, however, whether these additional peaks represent additional binding sites. Comparing the evaluation curves for the deeply sequenced sets to the randomly sampled subsets showed no obvious improvement in performance (with the exception of QuEST, which performed poorly on the randomly sampled subsets) (Supplementary Figure S10). To investigate this paradox further, we closely inspected a subset of the manually evaluated peaks. This subset included peaks with clearly improved visibility in the complete sets compared to the randomly sampled subsets. A total of 136 peaks satisfied this criterion from the NRSF and SRF data sets, where 33 of these were classified as true peaks, 29 as ambiguous peaks and 74 as noise features. When examining the program outputs on the selected regions, improved performance is observed in the more deeply sequenced sets only when an external background model is included (Figure 3). Without an external background model, the programs cannot identify more true peaks without including an intolerable number of false positives, and one is often better off using the random subset.Figure 3.


A manually curated ChIP-seq benchmark demonstrates room for improvement in current peak-finder programs.

Rye MB, Sætrom P, Drabløs F - Nucleic Acids Res. (2010)

New peaks identified for (A) NRSF and (B) SRF in deeply sequenced sets, compared to random subsets. The bars show the number of positive, ambiguous and negative peaks found by MACS (M), MACS with background (Mc), SISSRs (S) and SISSRs with background (Sc), together with the manual evaluation reference (Ev).
© Copyright Policy - creative-commons
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC3045577&req=5

Figure 3: New peaks identified for (A) NRSF and (B) SRF in deeply sequenced sets, compared to random subsets. The bars show the number of positive, ambiguous and negative peaks found by MACS (M), MACS with background (Mc), SISSRs (S) and SISSRs with background (Sc), together with the manual evaluation reference (Ev).
Mentions: As high-throughput sequencing becomes more common, the number of deeply sequenced data sets together with an extensive use of replicates is expected to increase. However, how this may influence the identification of peaks and the performance of peak-finder programs is not clear (12,19). The idea behind using more sequence data is to identify more weak binding sites while making stronger sites appear more pronounced. Running the programs on the more deeply sequenced data sets did indeed produce more peaks compared to the smaller randomly sampled subsets, which is in accordance with previous studies (12,19). The question is, however, whether these additional peaks represent additional binding sites. Comparing the evaluation curves for the deeply sequenced sets to the randomly sampled subsets showed no obvious improvement in performance (with the exception of QuEST, which performed poorly on the randomly sampled subsets) (Supplementary Figure S10). To investigate this paradox further, we closely inspected a subset of the manually evaluated peaks. This subset included peaks with clearly improved visibility in the complete sets compared to the randomly sampled subsets. A total of 136 peaks satisfied this criterion from the NRSF and SRF data sets, where 33 of these were classified as true peaks, 29 as ambiguous peaks and 74 as noise features. When examining the program outputs on the selected regions, improved performance is observed in the more deeply sequenced sets only when an external background model is included (Figure 3). Without an external background model, the programs cannot identify more true peaks without including an intolerable number of false positives, and one is often better off using the random subset.Figure 3.

Bottom Line: Second, none of the programs returned peak-regions that corresponded to the actual resolution in ChIP-seq data.Our results showed that ChIP-seq peaks should be narrowed down to 100-400 bp, which is sufficient to identify unique peaks and binding sites.Based on these results, we propose a meta-approach that gives improved peak definitions.

View Article: PubMed Central - PubMed

Affiliation: Department of Cancer Research and Molecular Medicine, Norwegian University of Science and Technology, NO-7489 Trondheim, Norway. morten.rye@ntnu.no

ABSTRACT
Chromatin immunoprecipitation (ChIP) followed by high throughput sequencing (ChIP-seq) is rapidly becoming the method of choice for discovering cell-specific transcription factor binding locations genome wide. By aligning sequenced tags to the genome, binding locations appear as peaks in the tag profile. Several programs have been designed to identify such peaks, but program evaluation has been difficult due to the lack of benchmark data sets. We have created benchmark data sets for three transcription factors by manually evaluating a selection of potential binding regions that cover typical variation in peak size and appearance. Performance of five programs on this benchmark showed, first, that external control or background data was essential to limit the number of false positive peaks from the programs. However, >80% of these peaks could be manually filtered out by visual inspection alone, without using additional background data, showing that peak shape information is not fully exploited in the evaluated programs. Second, none of the programs returned peak-regions that corresponded to the actual resolution in ChIP-seq data. Our results showed that ChIP-seq peaks should be narrowed down to 100-400 bp, which is sufficient to identify unique peaks and binding sites. Based on these results, we propose a meta-approach that gives improved peak definitions.

Show MeSH
Related in: MedlinePlus