Evaluation of demographic history and neutral parameterization on the performance of FST outlier tests.
Bottom Line: BayeScan was improved when the prior odds on neutrality was increased, regardless of the true odds in the data.In addition, default settings in FDIST2 and BayeScan resulted in many false positives suggesting balancing selection.We conclude that in species that exhibit IBD or have undergone range expansion, many of the published FST outliers based on FDIST2 and BayeScan are probably false positives, but FLK and Bayenv2 show great promise for accurately identifying loci under spatially divergent selection.
Affiliation: Department of Zoology, University of British Columbia, Vancouver, British Columbia, Canada, V6T 1Z4.Show MeSH
Mentions: We evaluated the effect of unwanted inclusion of selected loci by measuring the correlation between coancestry or covariance matrices estimated with all the data to those estimated with neutral data only. A higher correlation means that the selected loci in the data set did not bias the estimation of the coancestry/covariance matrix. Bayenv2 tended to be less sensitive to bias caused by inclusion of selected loci than FLK for more complex demographic histories, while FLK only outperformed Bayenv2 for the island model (Fig. 4). The effect of selection on the coancestry matrix from FLK and the covariance matrix from Bayenv2 are shown in Fig. S17 (Supporting information).
Affiliation: Department of Zoology, University of British Columbia, Vancouver, British Columbia, Canada, V6T 1Z4.