Limits...
Validation of 3D EM Reconstructions: The Phantom in the Noise.

Heymann JB - AIMS Biophys (2015)

Bottom Line: This poses the risk of inappropriate data processing with dubious results.How can we test that a map is a coherent structure present in the images selected from the micrographs?Adopting such a test can aid the microscopist in assessing the usefulness of the micrographs taken before committing to lengthy processing with questionable outcomes.

View Article: PubMed Central - HTML - PubMed

Affiliation: National Institute of Arthritis and Musculoskeletal and Skin Diseases, National Institutes of Health, 50 South Dr, Bethesda, MD 20892, USA.

ABSTRACT

Validation is a necessity to trust the structures solved by electron microscopy by single particle techniques. The impressive achievements in single particle reconstruction fuel its expansion beyond a small community of image processing experts. This poses the risk of inappropriate data processing with dubious results. Nowhere is it more clearly illustrated than in the recovery of a reference density map from pure noise aligned to that map-a phantom in the noise. Appropriate use of existing validating methods such as resolution-limited alignment and the processing of independent data sets ("gold standard") avoid this pitfall. However, these methods can be undermined by biases introduced in various subtle ways. How can we test that a map is a coherent structure present in the images selected from the micrographs? In stead of viewing the phantom emerging from noise as a cautionary tale, it should be used as a defining baseline. Any map is always recoverable from noise images, provided a sufficient number of images are aligned and used in reconstruction. However, with smaller numbers of images, the expected coherence in the real particle images should yield better reconstructions than equivalent numbers of noise or background images, even without masking or imposing resolution limits as potential biases. The validation test proposed is therefore a simple alignment of a limited number of micrograph and noise images against the final reconstruction as reference, demonstrating that the micrograph images yield a better reconstruction. I examine synthetic cases to relate the resolution of a reconstruction to the alignment error as a function of the signal-to-noise ratio. I also administered the test to real cases of publicly available data. Adopting such a test can aid the microscopist in assessing the usefulness of the micrographs taken before committing to lengthy processing with questionable outcomes.

No MeSH data available.


(a) Reconstructions using images from the HIVGP micrographs (blue disks and green diamonds) compared with those from gaussian noise images (gray disks). Both positive (blue disks) and negative (green diamonds) density references were tested because the contrast direction was not evident from the images. When the alignment of the images was limited to 20 Å, the resolution estimation did not improve beyond 18 Å (red triangles). Each point is the average of resolution estimates of 10 reconstructions with standard deviations as indicated by the error bars. The images were of size 1282 at 1.49 Å/pixel, symmetry C3. (b) FSC curves for the reconstructions from 105 micrograph (blue) and noise (gray) images aligned to Nyquist, and micrograph images aligned to 20 Å (red).
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4440490&req=5

Figure 5: (a) Reconstructions using images from the HIVGP micrographs (blue disks and green diamonds) compared with those from gaussian noise images (gray disks). Both positive (blue disks) and negative (green diamonds) density references were tested because the contrast direction was not evident from the images. When the alignment of the images was limited to 20 Å, the resolution estimation did not improve beyond 18 Å (red triangles). Each point is the average of resolution estimates of 10 reconstructions with standard deviations as indicated by the error bars. The images were of size 1282 at 1.49 Å/pixel, symmetry C3. (b) FSC curves for the reconstructions from 105 micrograph (blue) and noise (gray) images aligned to Nyquist, and micrograph images aligned to 20 Å (red).

Mentions: A study reporting a structure for the HIV glycoprotein (HIVGP) generated a large amount of controversy [4,16–19]. The data for this study has now been released, and the boxed images were obtained from the EMDB. The first problem was that the images appear not to contain any recognizable structures (as noted in [19]), and the contrast direction in the images was unclear. The images were therefore aligned with both a positive and negative version of the final reconstruction [18]. A similar alignment of noise images produced reconstruction resolution estimates tracking those of the particle images aligned with both references, suggesting that they are indistinguishable from noise (Figure 5a). The only difference is at low numbers of images (< 100), where the noise-derived reconstructions show worse resolutions compared to the particle images. This may be due to the amplitude decrease with spatial frequency in the micrograph data, whereas the generated noise images have constant average amplitude over all frequencies. The FSC curves for reconstructions from both micrograph and noise images are indistinguishable but consistent with the claimed ~6 Å resolution (Figure 5b). The conclusion is that the images do not exhibit any alignable information as demonstrated in the other two cases.


Validation of 3D EM Reconstructions: The Phantom in the Noise.

Heymann JB - AIMS Biophys (2015)

(a) Reconstructions using images from the HIVGP micrographs (blue disks and green diamonds) compared with those from gaussian noise images (gray disks). Both positive (blue disks) and negative (green diamonds) density references were tested because the contrast direction was not evident from the images. When the alignment of the images was limited to 20 Å, the resolution estimation did not improve beyond 18 Å (red triangles). Each point is the average of resolution estimates of 10 reconstructions with standard deviations as indicated by the error bars. The images were of size 1282 at 1.49 Å/pixel, symmetry C3. (b) FSC curves for the reconstructions from 105 micrograph (blue) and noise (gray) images aligned to Nyquist, and micrograph images aligned to 20 Å (red).
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4440490&req=5

Figure 5: (a) Reconstructions using images from the HIVGP micrographs (blue disks and green diamonds) compared with those from gaussian noise images (gray disks). Both positive (blue disks) and negative (green diamonds) density references were tested because the contrast direction was not evident from the images. When the alignment of the images was limited to 20 Å, the resolution estimation did not improve beyond 18 Å (red triangles). Each point is the average of resolution estimates of 10 reconstructions with standard deviations as indicated by the error bars. The images were of size 1282 at 1.49 Å/pixel, symmetry C3. (b) FSC curves for the reconstructions from 105 micrograph (blue) and noise (gray) images aligned to Nyquist, and micrograph images aligned to 20 Å (red).
Mentions: A study reporting a structure for the HIV glycoprotein (HIVGP) generated a large amount of controversy [4,16–19]. The data for this study has now been released, and the boxed images were obtained from the EMDB. The first problem was that the images appear not to contain any recognizable structures (as noted in [19]), and the contrast direction in the images was unclear. The images were therefore aligned with both a positive and negative version of the final reconstruction [18]. A similar alignment of noise images produced reconstruction resolution estimates tracking those of the particle images aligned with both references, suggesting that they are indistinguishable from noise (Figure 5a). The only difference is at low numbers of images (< 100), where the noise-derived reconstructions show worse resolutions compared to the particle images. This may be due to the amplitude decrease with spatial frequency in the micrograph data, whereas the generated noise images have constant average amplitude over all frequencies. The FSC curves for reconstructions from both micrograph and noise images are indistinguishable but consistent with the claimed ~6 Å resolution (Figure 5b). The conclusion is that the images do not exhibit any alignable information as demonstrated in the other two cases.

Bottom Line: This poses the risk of inappropriate data processing with dubious results.How can we test that a map is a coherent structure present in the images selected from the micrographs?Adopting such a test can aid the microscopist in assessing the usefulness of the micrographs taken before committing to lengthy processing with questionable outcomes.

View Article: PubMed Central - HTML - PubMed

Affiliation: National Institute of Arthritis and Musculoskeletal and Skin Diseases, National Institutes of Health, 50 South Dr, Bethesda, MD 20892, USA.

ABSTRACT

Validation is a necessity to trust the structures solved by electron microscopy by single particle techniques. The impressive achievements in single particle reconstruction fuel its expansion beyond a small community of image processing experts. This poses the risk of inappropriate data processing with dubious results. Nowhere is it more clearly illustrated than in the recovery of a reference density map from pure noise aligned to that map-a phantom in the noise. Appropriate use of existing validating methods such as resolution-limited alignment and the processing of independent data sets ("gold standard") avoid this pitfall. However, these methods can be undermined by biases introduced in various subtle ways. How can we test that a map is a coherent structure present in the images selected from the micrographs? In stead of viewing the phantom emerging from noise as a cautionary tale, it should be used as a defining baseline. Any map is always recoverable from noise images, provided a sufficient number of images are aligned and used in reconstruction. However, with smaller numbers of images, the expected coherence in the real particle images should yield better reconstructions than equivalent numbers of noise or background images, even without masking or imposing resolution limits as potential biases. The validation test proposed is therefore a simple alignment of a limited number of micrograph and noise images against the final reconstruction as reference, demonstrating that the micrograph images yield a better reconstruction. I examine synthetic cases to relate the resolution of a reconstruction to the alignment error as a function of the signal-to-noise ratio. I also administered the test to real cases of publicly available data. Adopting such a test can aid the microscopist in assessing the usefulness of the micrographs taken before committing to lengthy processing with questionable outcomes.

No MeSH data available.