Limits...
CSAR benchmark exercise 2011-2012: evaluation of results from docking and relative ranking of blinded congeneric series.

Damm-Ganamet KL, Smith RD, Dunbar JB, Stuckey JA, Carlson HA - J Chem Inf Model (2013)

Bottom Line: Lastly, we found that minimizing the protein and correcting histidine tautomeric states positively trended with low RMSD for pose prediction but minimizing the ligand negatively trended.Optimizing docking parameters and pretraining with the native ligand had a positive effect on the docking performance as did using restraints, substructure fitting, and shape fitting.Here, by combining the results of many methods, we hope to provide a statistically relevant evaluation and elucidate specific shortcomings of docking methodology for the community.

View Article: PubMed Central - PubMed

Affiliation: Department of Medicinal Chemistry, University of Michigan, Ann Arbor, Michigan 48109-1065, USA.

ABSTRACT
The Community Structure-Activity Resource (CSAR) recently held its first blinded exercise based on data provided by Abbott, Vertex, and colleagues at the University of Michigan, Ann Arbor. A total of 20 research groups submitted results for the benchmark exercise where the goal was to compare different improvements for pose prediction, enrichment, and relative ranking of congeneric series of compounds. The exercise was built around blinded high-quality experimental data from four protein targets: LpxC, Urokinase, Chk1, and Erk2. Pose prediction proved to be the most straightforward task, and most methods were able to successfully reproduce binding poses when the crystal structure employed was co-crystallized with a ligand from the same chemical series. Multiple evaluation metrics were examined, and we found that RMSD and native contact metrics together provide a robust evaluation of the predicted poses. It was notable that most scoring functions underpredicted contacts between the hetero atoms (i.e., N, O, S, etc.) of the protein and ligand. Relative ranking was found to be the most difficult area for the methods, but many of the scoring functions were able to properly identify Urokinase actives from the inactives in the series. Lastly, we found that minimizing the protein and correcting histidine tautomeric states positively trended with low RMSD for pose prediction but minimizing the ligand negatively trended. Pregenerated ligand conformations performed better than those that were generated on the fly. Optimizing docking parameters and pretraining with the native ligand had a positive effect on the docking performance as did using restraints, substructure fitting, and shape fitting. Lastly, for both sampling and ranking scoring functions, the use of the empirical scoring function appeared to trend positively with the RMSD. Here, by combining the results of many methods, we hope to provide a statistically relevant evaluation and elucidate specific shortcomings of docking methodology for the community.

Show MeSH
RMSD is plotted againstthe percentage of inactive molecules rankedhigher than an active molecule for both Urokinase and Chk1 targets.The insert shows the percentage of ligands that fall with each RMSDbin for two groups: (1) active molecules that have no inactives rankedhigher (0%) and (2) active molecules that have one or more inactivesranked higher (all other).
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC3753884&req=5

fig13: RMSD is plotted againstthe percentage of inactive molecules rankedhigher than an active molecule for both Urokinase and Chk1 targets.The insert shows the percentage of ligands that fall with each RMSDbin for two groups: (1) active molecules that have no inactives rankedhigher (0%) and (2) active molecules that have one or more inactivesranked higher (all other).

Mentions: Docking programs are faced with two major tasks: (1)predicting the pose of the ligand in the presence of the protein and(2) scoring the predicted the pose. Although two separate tasks, theyshould be correlated, which brings about the question “Arescoring functions getting the rankings correct for the right reasons?”.One would assume that if the pose is ranked the highest, it shouldbe the pose seen in the crystal structure. In Figure 13, the RMSD of the top-ranking pose for molecule X is shownversus the percentage of inactive molecules rankedhigher than molecule X for both Urokinase and Chk1 targets (Was thescoring function able to rank the active molecule higher than theinactive molecules, and if not, how many inactive molecules were rankedhigher?). Again, only Urokinase and Chk1 were used because of thereasons stated above. While there is a spread in the data suggestingthat the scoring function is not always ranking for the correct reason,there also is a large group of molecules (13.6%) near the 0,0 pointon the graph. There are multiple reasons that scoring functions maybe ranking an incorrect pose higher than the correct pose. First,it may be due to incorrect capturing of the contacts being made betweenthe protein and ligand. Additionally, it may be because of the termsthat are not explicitly accounted for in the scoring function (e.g.,entropy and/or solvation are typically not included).


CSAR benchmark exercise 2011-2012: evaluation of results from docking and relative ranking of blinded congeneric series.

Damm-Ganamet KL, Smith RD, Dunbar JB, Stuckey JA, Carlson HA - J Chem Inf Model (2013)

RMSD is plotted againstthe percentage of inactive molecules rankedhigher than an active molecule for both Urokinase and Chk1 targets.The insert shows the percentage of ligands that fall with each RMSDbin for two groups: (1) active molecules that have no inactives rankedhigher (0%) and (2) active molecules that have one or more inactivesranked higher (all other).
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC3753884&req=5

fig13: RMSD is plotted againstthe percentage of inactive molecules rankedhigher than an active molecule for both Urokinase and Chk1 targets.The insert shows the percentage of ligands that fall with each RMSDbin for two groups: (1) active molecules that have no inactives rankedhigher (0%) and (2) active molecules that have one or more inactivesranked higher (all other).
Mentions: Docking programs are faced with two major tasks: (1)predicting the pose of the ligand in the presence of the protein and(2) scoring the predicted the pose. Although two separate tasks, theyshould be correlated, which brings about the question “Arescoring functions getting the rankings correct for the right reasons?”.One would assume that if the pose is ranked the highest, it shouldbe the pose seen in the crystal structure. In Figure 13, the RMSD of the top-ranking pose for molecule X is shownversus the percentage of inactive molecules rankedhigher than molecule X for both Urokinase and Chk1 targets (Was thescoring function able to rank the active molecule higher than theinactive molecules, and if not, how many inactive molecules were rankedhigher?). Again, only Urokinase and Chk1 were used because of thereasons stated above. While there is a spread in the data suggestingthat the scoring function is not always ranking for the correct reason,there also is a large group of molecules (13.6%) near the 0,0 pointon the graph. There are multiple reasons that scoring functions maybe ranking an incorrect pose higher than the correct pose. First,it may be due to incorrect capturing of the contacts being made betweenthe protein and ligand. Additionally, it may be because of the termsthat are not explicitly accounted for in the scoring function (e.g.,entropy and/or solvation are typically not included).

Bottom Line: Lastly, we found that minimizing the protein and correcting histidine tautomeric states positively trended with low RMSD for pose prediction but minimizing the ligand negatively trended.Optimizing docking parameters and pretraining with the native ligand had a positive effect on the docking performance as did using restraints, substructure fitting, and shape fitting.Here, by combining the results of many methods, we hope to provide a statistically relevant evaluation and elucidate specific shortcomings of docking methodology for the community.

View Article: PubMed Central - PubMed

Affiliation: Department of Medicinal Chemistry, University of Michigan, Ann Arbor, Michigan 48109-1065, USA.

ABSTRACT
The Community Structure-Activity Resource (CSAR) recently held its first blinded exercise based on data provided by Abbott, Vertex, and colleagues at the University of Michigan, Ann Arbor. A total of 20 research groups submitted results for the benchmark exercise where the goal was to compare different improvements for pose prediction, enrichment, and relative ranking of congeneric series of compounds. The exercise was built around blinded high-quality experimental data from four protein targets: LpxC, Urokinase, Chk1, and Erk2. Pose prediction proved to be the most straightforward task, and most methods were able to successfully reproduce binding poses when the crystal structure employed was co-crystallized with a ligand from the same chemical series. Multiple evaluation metrics were examined, and we found that RMSD and native contact metrics together provide a robust evaluation of the predicted poses. It was notable that most scoring functions underpredicted contacts between the hetero atoms (i.e., N, O, S, etc.) of the protein and ligand. Relative ranking was found to be the most difficult area for the methods, but many of the scoring functions were able to properly identify Urokinase actives from the inactives in the series. Lastly, we found that minimizing the protein and correcting histidine tautomeric states positively trended with low RMSD for pose prediction but minimizing the ligand negatively trended. Pregenerated ligand conformations performed better than those that were generated on the fly. Optimizing docking parameters and pretraining with the native ligand had a positive effect on the docking performance as did using restraints, substructure fitting, and shape fitting. Lastly, for both sampling and ranking scoring functions, the use of the empirical scoring function appeared to trend positively with the RMSD. Here, by combining the results of many methods, we hope to provide a statistically relevant evaluation and elucidate specific shortcomings of docking methodology for the community.

Show MeSH