Limits...
Three-dimensional reconstruction of highly complex microscopic samples using scanning electron microscopy and optical flow estimation

View Article: PubMed Central - PubMed

ABSTRACT

Scanning Electron Microscope (SEM) as one of the major research and industrial equipment for imaging of micro-scale samples and surfaces has gained extensive attention from its emerge. However, the acquired micrographs still remain two-dimensional (2D). In the current work a novel and highly accurate approach is proposed to recover the hidden third-dimension by use of multi-view image acquisition of the microscopic samples combined with pre/post-processing steps including sparse feature-based stereo rectification, nonlocal-based optical flow estimation for dense matching and finally depth estimation. Employing the proposed approach, three-dimensional (3D) reconstructions of highly complex microscopic samples were achieved to facilitate the interpretation of topology and geometry of surface/shape attributes of the samples. As a byproduct of the proposed approach, high-definition 3D printed models of the samples can be generated as a tangible means of physical understanding. Extensive comparisons with the state-of-the-art reveal the strength and superiority of the proposed method in uncovering the details of the highly complex microscopic samples.

No MeSH data available.


Related in: MedlinePlus

Comparison of the results for Fly Ash: a) the overall as well as a zoomed region of the computed disparity map using the state-of-the-art method of [32] which uses sparse feature-based matching approach and a contrario RANSAC for outlier removal. The dense disparity map is created by scattered data interpolation of the sparse disparity values. b) the result of Horn/Schunck optical flow estimation [43], which provides a better estimation of the disparity map than that of [32]. c) the result of dense feature matching proposed in [68] which uses dense SIFT features as well as factor graph representation of the matching energy functional optimized by loopy belief propagation. Even though relatively better than [43], the result still suffers from blurred edges. The result of the proposed method is presented in (d). In comparison to the state-of-the-art, the proposed approach generates a sharper and more accurate disparity map.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC5383242&req=5

pone.0175078.g007: Comparison of the results for Fly Ash: a) the overall as well as a zoomed region of the computed disparity map using the state-of-the-art method of [32] which uses sparse feature-based matching approach and a contrario RANSAC for outlier removal. The dense disparity map is created by scattered data interpolation of the sparse disparity values. b) the result of Horn/Schunck optical flow estimation [43], which provides a better estimation of the disparity map than that of [32]. c) the result of dense feature matching proposed in [68] which uses dense SIFT features as well as factor graph representation of the matching energy functional optimized by loopy belief propagation. Even though relatively better than [43], the result still suffers from blurred edges. The result of the proposed method is presented in (d). In comparison to the state-of-the-art, the proposed approach generates a sharper and more accurate disparity map.

Mentions: For a more comprehensive analysis, the proposed dense matching approach is compared with several other methods previously used in the literature for dense matching and subsequently 3D reconstruction. Sparse feature-based approaches track the movements of distinct feature points in the input images in order to compute the fundamental matrix and projective transformation [32, 64]. To generate a dense disparity map, similar to that of created by our approach for a better comparison of the performance, the sparse disparity values are interpolated employing a Delaunay triangulation-based interpolation method. As for dense matching schemes, the works of Horn and Schunck [43] and Liu et al. [68] are good examples. While the first one works based on the pixels’ correspondence, the later extends a similar idea to matching of dense SIFT descriptors. Figs 6 and 7 display the disparity maps computed using the above-mentioned methods as well as the proposed approach for the Graphene and Fly Ash sample sets, respectively. In each figure, the left column shows the overall disparity map while the right column is a zoomed view for a better visual comparison of the various techniques. Close inspection of the provided results displays the superiority of the proposed approach. As expected, the outcome of the sparse feature-based approach is highly blurred near edges with significant loss of details presented in the images. Even though such techniques are mainly used with more than two input images, the performance is the same as evident from the results. In contrast, dense matching approaches produce more accurate results. In the results of the method of [43], more details are presented and discontinuities are better preserved. However, in cases of having larger displacements near the margins of the input images (left side of the Graphene results) the estimated optical flow is not as accurate as the sparse feature-based approach. Using the dense descriptor matching scheme in the work of Liu et al. [68], this is mostly resolved. In this technique, at first two 128-dimensional dense SIFT descriptor images of both the first and second micrograph in the pair are created. To compute the matching, a factor graph representation of the specifically defined energy functional is introduced and the process of optimization is done using loopy belief propagation. By employing the dense descriptor matching methodology more accurate results can be achieved. The last row in Figs 6 and 7 is the disparity results using the proposed approach. Employing the proposed approach, higher levels of details can be reached in the resulted disparity maps. With higher accuracy in preserving the discontinuities, a more truthful reconstruction can be made. This is more evident in the samples with higher complexity levels, Fly Ash sample set for example. As shown in Fig 7, the proposed approach can recover disparity values even for smaller objects in the images, while in contrast, the other methods presented here cannot, due to high amount of blur around edges and boundaries.


Three-dimensional reconstruction of highly complex microscopic samples using scanning electron microscopy and optical flow estimation
Comparison of the results for Fly Ash: a) the overall as well as a zoomed region of the computed disparity map using the state-of-the-art method of [32] which uses sparse feature-based matching approach and a contrario RANSAC for outlier removal. The dense disparity map is created by scattered data interpolation of the sparse disparity values. b) the result of Horn/Schunck optical flow estimation [43], which provides a better estimation of the disparity map than that of [32]. c) the result of dense feature matching proposed in [68] which uses dense SIFT features as well as factor graph representation of the matching energy functional optimized by loopy belief propagation. Even though relatively better than [43], the result still suffers from blurred edges. The result of the proposed method is presented in (d). In comparison to the state-of-the-art, the proposed approach generates a sharper and more accurate disparity map.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC5383242&req=5

pone.0175078.g007: Comparison of the results for Fly Ash: a) the overall as well as a zoomed region of the computed disparity map using the state-of-the-art method of [32] which uses sparse feature-based matching approach and a contrario RANSAC for outlier removal. The dense disparity map is created by scattered data interpolation of the sparse disparity values. b) the result of Horn/Schunck optical flow estimation [43], which provides a better estimation of the disparity map than that of [32]. c) the result of dense feature matching proposed in [68] which uses dense SIFT features as well as factor graph representation of the matching energy functional optimized by loopy belief propagation. Even though relatively better than [43], the result still suffers from blurred edges. The result of the proposed method is presented in (d). In comparison to the state-of-the-art, the proposed approach generates a sharper and more accurate disparity map.
Mentions: For a more comprehensive analysis, the proposed dense matching approach is compared with several other methods previously used in the literature for dense matching and subsequently 3D reconstruction. Sparse feature-based approaches track the movements of distinct feature points in the input images in order to compute the fundamental matrix and projective transformation [32, 64]. To generate a dense disparity map, similar to that of created by our approach for a better comparison of the performance, the sparse disparity values are interpolated employing a Delaunay triangulation-based interpolation method. As for dense matching schemes, the works of Horn and Schunck [43] and Liu et al. [68] are good examples. While the first one works based on the pixels’ correspondence, the later extends a similar idea to matching of dense SIFT descriptors. Figs 6 and 7 display the disparity maps computed using the above-mentioned methods as well as the proposed approach for the Graphene and Fly Ash sample sets, respectively. In each figure, the left column shows the overall disparity map while the right column is a zoomed view for a better visual comparison of the various techniques. Close inspection of the provided results displays the superiority of the proposed approach. As expected, the outcome of the sparse feature-based approach is highly blurred near edges with significant loss of details presented in the images. Even though such techniques are mainly used with more than two input images, the performance is the same as evident from the results. In contrast, dense matching approaches produce more accurate results. In the results of the method of [43], more details are presented and discontinuities are better preserved. However, in cases of having larger displacements near the margins of the input images (left side of the Graphene results) the estimated optical flow is not as accurate as the sparse feature-based approach. Using the dense descriptor matching scheme in the work of Liu et al. [68], this is mostly resolved. In this technique, at first two 128-dimensional dense SIFT descriptor images of both the first and second micrograph in the pair are created. To compute the matching, a factor graph representation of the specifically defined energy functional is introduced and the process of optimization is done using loopy belief propagation. By employing the dense descriptor matching methodology more accurate results can be achieved. The last row in Figs 6 and 7 is the disparity results using the proposed approach. Employing the proposed approach, higher levels of details can be reached in the resulted disparity maps. With higher accuracy in preserving the discontinuities, a more truthful reconstruction can be made. This is more evident in the samples with higher complexity levels, Fly Ash sample set for example. As shown in Fig 7, the proposed approach can recover disparity values even for smaller objects in the images, while in contrast, the other methods presented here cannot, due to high amount of blur around edges and boundaries.

View Article: PubMed Central - PubMed

ABSTRACT

Scanning Electron Microscope (SEM) as one of the major research and industrial equipment for imaging of micro-scale samples and surfaces has gained extensive attention from its emerge. However, the acquired micrographs still remain two-dimensional (2D). In the current work a novel and highly accurate approach is proposed to recover the hidden third-dimension by use of multi-view image acquisition of the microscopic samples combined with pre/post-processing steps including sparse feature-based stereo rectification, nonlocal-based optical flow estimation for dense matching and finally depth estimation. Employing the proposed approach, three-dimensional (3D) reconstructions of highly complex microscopic samples were achieved to facilitate the interpretation of topology and geometry of surface/shape attributes of the samples. As a byproduct of the proposed approach, high-definition 3D printed models of the samples can be generated as a tangible means of physical understanding. Extensive comparisons with the state-of-the-art reveal the strength and superiority of the proposed method in uncovering the details of the highly complex microscopic samples.

No MeSH data available.


Related in: MedlinePlus