Limits...
Computationally efficient locally adaptive demosaicing of color filter array images using the dual-tree complex wavelet packet transform.

Aelterman J, Goossens B, De Vylder J, Pižurica A, Philips W - PLoS ONE (2013)

Bottom Line: By using a novel locally adaptive approach for demosaicing (complex) wavelet coefficients, we show that many of the common demosaicing artifacts can be avoided in an efficient way.Results demonstrate that the proposed method is competitive with respect to the current state of the art, but incurs a lower computational cost.The wavelet approach also allows for computationally effective denoising or deblurring approaches.

View Article: PubMed Central - PubMed

Affiliation: IPI-TELIN-IMINDS, Ghent University, Ghent, Belgium. jaelterm@telin.ugent.be

ABSTRACT
Most digital cameras use an array of alternating color filters to capture the varied colors in a scene with a single sensor chip. Reconstruction of a full color image from such a color mosaic is what constitutes demosaicing. In this paper, a technique is proposed that performs this demosaicing in a way that incurs a very low computational cost. This is done through a (dual-tree complex) wavelet interpretation of the demosaicing problem. By using a novel locally adaptive approach for demosaicing (complex) wavelet coefficients, we show that many of the common demosaicing artifacts can be avoided in an efficient way. Results demonstrate that the proposed method is competitive with respect to the current state of the art, but incurs a lower computational cost. The wavelet approach also allows for computationally effective denoising or deblurring approaches.

Show MeSH
Effect of postprocessing on the proposed demosaicing algorithm.Note the negligible visual difference, but the large difference in PSNR due to the data fidelity property.
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC3643977&req=5

pone-0061846-g015: Effect of postprocessing on the proposed demosaicing algorithm.Note the negligible visual difference, but the large difference in PSNR due to the data fidelity property.

Mentions: In this section, we compare the demosaicing performance of the proposed algorithm with several other algorithms. In our comparison we will use the non-adaptive wavelet demosaicing algorithm from [14], the DLMMSE method from [2], the POCS method from [3] (set to a fixed number of 5 iterations), the hybrid [15] (wavelet detection and pixel based reconstruction), the linear filter scheme from [10] and [5], which is the qualitative state-of-the-art at the moment of writing, to the knowledge of the author. The implementations used here are publicly available from http://www.csee.wvu.edu/xinl/source.html, except for [10] which we implemented based on the suggested filters in [10]. While the proposed approach for demosaicing has significant advantages, there is a drawback when it comes to objective comparison. The algorithm makes hard assumptions on the chrominance bandwidth, any small error thus introduced in a wavelet coefficient will cause a hue shift across all pixels in the wavelet's support. Upon visual inspection, these small errors are hardly noticeable, but they can result in a significant MSE. In order to decrease the MSE, we could simply insert the measured pixel intensities from the mosaic into the reconstructed image, however, this gives rise to very noticeable zipper artifacts (see for an example Figure 6). This indicates that MSE or PSNR have severe drawbacks as a measure for visual quality. We still choose to use it here, as it remains the most popular choice of comparison in literature. In order to have a fairer comparison with respect to visual quality, we apply the demosaicing post-processing technique from [31]. This algorithm exploits the spectral correlations between the color components and the luminance bandwidth (i.e. sharpness) that was introduced in the proposed demosaicing algorithm to estimate only the missing pixel intensities, starting from the measured pixel intensities and preliminary interpolations. Hence, it retains the artifact-reducing power and high luminance bandwidth advantages of the proposed algorithm, as we will use this as preliminary interpolation, but it increases PSNR. The visual quality is related to the artifact-reduction and, as a result, is not improved. This effect is demonstrated in Figure 15: even though there is an increase of more than 1 dB in PSNR, there is hardly any visual difference, even in these difficult demosaicing experiments. In the remainder of this paper, we will compare PSNR results of the proposed algorithm with post-processing enabled. It is important to note that other demosaicing algorithms (such as [2], [5]) already have data fidelity: They do not modify measured pixel values from the input grid. For these algorithms, as they already have data fidelity, it makes no sense to apply this post-processing and we repeatedly found it only reduces their respective performance. The proposed algorithm was tested on the 24 512×768 images of the classic Kodak test image data set (http://r0k.us/graphics/kodak/). Table 8 shows PSNR comparison for the different algorithms, compared with the proposed algorithm. We also list the results for the proposed algorithm when the dual-tree complex wavelet transform is not used. This demonstrates that the use of complex wavelets has a significant impact on the result with respect to aliasing reduction in the result, on average it means an increase of 1 dB in PSNR. The PSNR comparison shows that the proposed algorithm holds itself quite well, with respect to the state of the art in demosaicing algorithms. It shows how the wavelet-based methods, due to the crude assumptions made on the transition bandwidths, are outperformed by the pure linear filter scheme from [10], which has finer control over the transition bandwidth, when the local adaptivity of wavelets is not exploited. The local adaptivity however, is shown to be a significant improvement over non-adaptive wavelet schemes (e.g. from [14]) as well as the purely linear scheme in [10]. While some of the pixel-based demosaicing algorithms achieve a significantly higher PSNR in some experiments, this is mainly due to chroma shifts in the proposed algorithm. These small shifts are not as visually disturbing as structural demosaicing artifacts, which manifest more often in other algorithms, as large high frequency luminance+chroma errors. To demonstrate this point, we also include a qualitative comparison. Figure 16 shows the chrominance bandwidth problem in the most problematic image, i.e. the one with the highest PSNR difference between LPA-ICI and the proposed algorithm. Here we see a lot of high frequency edges between black areas (low chrominance) and red areas (very high chrominance), hence we have high chrominance bandwidth locally. Here, the low chrominance bandwidth assumption fails, and artifacts are introduced. However, we remark that these artifacts are not as visually disturbing as other demosaicing artifacts, which the proposed algorithm handles very well. One example is the classic lighthouse1 image from the same Kodak dataset. A comparison is shown in Figure 17. All demosaicing algorithms have difficulties reconstructing the white fence and the rocky river bank, because of its very high luminance bandwidth. The proposed algorithm is capable of reconstructing the luminance to the original Nyquist bandwidth, as discussed in Section 2.2, which leads to better reconstruction performance in comparison with other demosaicing algorithms in areas where the luminance exhibits a high bandwidth. A comparison of local PSNR, for these artifact-sensitive regions is shown in Table 9. Another big advantage of the proposed method lies in its computational simplicity. We compared the available Matlab implementations of the different demosaicing algorithms with respect to their average execution times on the 24 images of the Kodak set. The result is shown in Table 10, the proposed algorithm, in its naively implemented state, is significantly faster than the existing state of the art in demosaicing, while achieving a roughly equivalent qualitative, and in some respects (luminance bandwidth) better, demosaicing result. For POCS, we also mention the estimate that takes the speedup (factor 8.5) into account of the accelerated version of the POCS algorithm presented in [4]. The addition in between brackets expresses the time it takes to perform the wavelet transform, we make a distinction here as the wavelet transform can immediately be made use of to perform other restoration tasks than demosaicing, such as denoising, sharpening, etc. Combining wavelet based demosaicing with more general restoration has already been demonstrated in [32], which highlights the relevance of the proposed technique.


Computationally efficient locally adaptive demosaicing of color filter array images using the dual-tree complex wavelet packet transform.

Aelterman J, Goossens B, De Vylder J, Pižurica A, Philips W - PLoS ONE (2013)

Effect of postprocessing on the proposed demosaicing algorithm.Note the negligible visual difference, but the large difference in PSNR due to the data fidelity property.
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC3643977&req=5

pone-0061846-g015: Effect of postprocessing on the proposed demosaicing algorithm.Note the negligible visual difference, but the large difference in PSNR due to the data fidelity property.
Mentions: In this section, we compare the demosaicing performance of the proposed algorithm with several other algorithms. In our comparison we will use the non-adaptive wavelet demosaicing algorithm from [14], the DLMMSE method from [2], the POCS method from [3] (set to a fixed number of 5 iterations), the hybrid [15] (wavelet detection and pixel based reconstruction), the linear filter scheme from [10] and [5], which is the qualitative state-of-the-art at the moment of writing, to the knowledge of the author. The implementations used here are publicly available from http://www.csee.wvu.edu/xinl/source.html, except for [10] which we implemented based on the suggested filters in [10]. While the proposed approach for demosaicing has significant advantages, there is a drawback when it comes to objective comparison. The algorithm makes hard assumptions on the chrominance bandwidth, any small error thus introduced in a wavelet coefficient will cause a hue shift across all pixels in the wavelet's support. Upon visual inspection, these small errors are hardly noticeable, but they can result in a significant MSE. In order to decrease the MSE, we could simply insert the measured pixel intensities from the mosaic into the reconstructed image, however, this gives rise to very noticeable zipper artifacts (see for an example Figure 6). This indicates that MSE or PSNR have severe drawbacks as a measure for visual quality. We still choose to use it here, as it remains the most popular choice of comparison in literature. In order to have a fairer comparison with respect to visual quality, we apply the demosaicing post-processing technique from [31]. This algorithm exploits the spectral correlations between the color components and the luminance bandwidth (i.e. sharpness) that was introduced in the proposed demosaicing algorithm to estimate only the missing pixel intensities, starting from the measured pixel intensities and preliminary interpolations. Hence, it retains the artifact-reducing power and high luminance bandwidth advantages of the proposed algorithm, as we will use this as preliminary interpolation, but it increases PSNR. The visual quality is related to the artifact-reduction and, as a result, is not improved. This effect is demonstrated in Figure 15: even though there is an increase of more than 1 dB in PSNR, there is hardly any visual difference, even in these difficult demosaicing experiments. In the remainder of this paper, we will compare PSNR results of the proposed algorithm with post-processing enabled. It is important to note that other demosaicing algorithms (such as [2], [5]) already have data fidelity: They do not modify measured pixel values from the input grid. For these algorithms, as they already have data fidelity, it makes no sense to apply this post-processing and we repeatedly found it only reduces their respective performance. The proposed algorithm was tested on the 24 512×768 images of the classic Kodak test image data set (http://r0k.us/graphics/kodak/). Table 8 shows PSNR comparison for the different algorithms, compared with the proposed algorithm. We also list the results for the proposed algorithm when the dual-tree complex wavelet transform is not used. This demonstrates that the use of complex wavelets has a significant impact on the result with respect to aliasing reduction in the result, on average it means an increase of 1 dB in PSNR. The PSNR comparison shows that the proposed algorithm holds itself quite well, with respect to the state of the art in demosaicing algorithms. It shows how the wavelet-based methods, due to the crude assumptions made on the transition bandwidths, are outperformed by the pure linear filter scheme from [10], which has finer control over the transition bandwidth, when the local adaptivity of wavelets is not exploited. The local adaptivity however, is shown to be a significant improvement over non-adaptive wavelet schemes (e.g. from [14]) as well as the purely linear scheme in [10]. While some of the pixel-based demosaicing algorithms achieve a significantly higher PSNR in some experiments, this is mainly due to chroma shifts in the proposed algorithm. These small shifts are not as visually disturbing as structural demosaicing artifacts, which manifest more often in other algorithms, as large high frequency luminance+chroma errors. To demonstrate this point, we also include a qualitative comparison. Figure 16 shows the chrominance bandwidth problem in the most problematic image, i.e. the one with the highest PSNR difference between LPA-ICI and the proposed algorithm. Here we see a lot of high frequency edges between black areas (low chrominance) and red areas (very high chrominance), hence we have high chrominance bandwidth locally. Here, the low chrominance bandwidth assumption fails, and artifacts are introduced. However, we remark that these artifacts are not as visually disturbing as other demosaicing artifacts, which the proposed algorithm handles very well. One example is the classic lighthouse1 image from the same Kodak dataset. A comparison is shown in Figure 17. All demosaicing algorithms have difficulties reconstructing the white fence and the rocky river bank, because of its very high luminance bandwidth. The proposed algorithm is capable of reconstructing the luminance to the original Nyquist bandwidth, as discussed in Section 2.2, which leads to better reconstruction performance in comparison with other demosaicing algorithms in areas where the luminance exhibits a high bandwidth. A comparison of local PSNR, for these artifact-sensitive regions is shown in Table 9. Another big advantage of the proposed method lies in its computational simplicity. We compared the available Matlab implementations of the different demosaicing algorithms with respect to their average execution times on the 24 images of the Kodak set. The result is shown in Table 10, the proposed algorithm, in its naively implemented state, is significantly faster than the existing state of the art in demosaicing, while achieving a roughly equivalent qualitative, and in some respects (luminance bandwidth) better, demosaicing result. For POCS, we also mention the estimate that takes the speedup (factor 8.5) into account of the accelerated version of the POCS algorithm presented in [4]. The addition in between brackets expresses the time it takes to perform the wavelet transform, we make a distinction here as the wavelet transform can immediately be made use of to perform other restoration tasks than demosaicing, such as denoising, sharpening, etc. Combining wavelet based demosaicing with more general restoration has already been demonstrated in [32], which highlights the relevance of the proposed technique.

Bottom Line: By using a novel locally adaptive approach for demosaicing (complex) wavelet coefficients, we show that many of the common demosaicing artifacts can be avoided in an efficient way.Results demonstrate that the proposed method is competitive with respect to the current state of the art, but incurs a lower computational cost.The wavelet approach also allows for computationally effective denoising or deblurring approaches.

View Article: PubMed Central - PubMed

Affiliation: IPI-TELIN-IMINDS, Ghent University, Ghent, Belgium. jaelterm@telin.ugent.be

ABSTRACT
Most digital cameras use an array of alternating color filters to capture the varied colors in a scene with a single sensor chip. Reconstruction of a full color image from such a color mosaic is what constitutes demosaicing. In this paper, a technique is proposed that performs this demosaicing in a way that incurs a very low computational cost. This is done through a (dual-tree complex) wavelet interpretation of the demosaicing problem. By using a novel locally adaptive approach for demosaicing (complex) wavelet coefficients, we show that many of the common demosaicing artifacts can be avoided in an efficient way. Results demonstrate that the proposed method is competitive with respect to the current state of the art, but incurs a lower computational cost. The wavelet approach also allows for computationally effective denoising or deblurring approaches.

Show MeSH