A Multi-Cues Fusion Method for Low Illumination Image Enhancement

1. Introduction

It is difficult to obtain good visual effects in a complex illumination environment with low illumination, uneven illumination, or interference from multiple light sources. Limited by equipment, ordinary color cameras just have poor photoelectric conversion effect in low illumination environments at night. The images or videos obtained from that scene are not only blurring, but also having a low signal-to-noise ratio. Therefore, image enhancement of low-illumination images is usually well required to achieve a better visual perception of various targets in the original image.

When it comes to image enhancement, according to state-of-the-art categories [1], there are generally two main methods, namely spatial and frequency domain enhancement. The spatial domain enhancement will directly process the pixel value of the image while the frequency domain enhancement can convert the pixel of the image to a specific transformation domain for processing. These two methods have their own unique advantages and disadvantages. So far, there is no unified framework on how to enhance low-illumination visual perception.

Histogram equalization, which belongs to the spatial method, is a global image contrast enhancement technique. Veluchamy and Subramani [2] applied histogram equalization technique in his experiment and combined it with adaptive gramma correction, then finally achieved good results by subjective and objective index tests. However, the experiment was conducted only on the artificial synthetic TID2008 dataset, which did not provide image samples under low illumination.

The luminance component of the incident light is removed on the basis of the Retinex enhancement method, so that the reflection component is not affected by the brightness. The estimation methods of luminance component in images fall roughly into two categories. One is the path-based luminance component estimation method. To address low illumination and large color difference in underwater images, Li et al. [3] analyzed the defects of the classic path-based McCann Retinex algorithm and proposed an improved scheme, which achieved good visual effects. Another is the most used method, namely the brightness of component estimation method based on the center-surrounding [4] function. But both Retinex-based enhancement methods are prone to halo [5] and color distortion [6].

Wavelet transform is one of the frequency domain enhancement techniques. Sharma and Jain [7] proposed a dual-tree complex wavelet transform to improve the visual effect of low-illumination medical image, which had been resulting in good results subsequently.

In fact, homomorphic filtering (HF) and guided filtering (GF) are often used as two important image enhancement methods in various low-illumination environments. For the purpose of improving the quality of degraded image in the underwater images obtained, Luo et al. [8] added the contrast-limited adaptive histogram equalization (CLAHE) before HF. By the experiment, a result is found that it cannot only increases the details of the image, but also suppresses the noise caused by the process of image enhancement. In view of the real scene under the condition of insufficient light, Huang et al. [9] proposed a weighted GF-based tone mapping method which can adopt different weighted ratios to enhance the image according to different spectral frequencies. The method not only effectively reduces the artifacts but also adds more details to the image.

However, like the previous wavelet transform method, HF and GF just only enhance the image in the frequency domain for no comparing or fusing with other images based on spatial methods. Therefore, the obtained enhancement cannot fully demonstrate the comprehensiveness of image enhancement technology. In view of this, the author address a fusion method which can integrate these better techniques to achieve a more comprehensive enhancement effect.

In addition, in recent years, some deep learning-based methods [10,11] have also been applied to low-illumination color image enhancement. However, all of these deep learning methods have two defects that are difficult to overcome. One reason is the interpretability of the parameters of those deep learning. There are many bias parameters that need to be adjusted in the deep learning algorithm, and these parameters can get a better enhancement effect even when they cannot be explained. Another, the deep learning-based image enhancement algorithm needs many samples for training and testing. If the provided sample scene is different from the actual scene, there will be a large illumination and color deviation. Taking into account the above defects, the deep learning method is not used for enhancement processing under low illumination in this paper.

Aiming at enhancing low-illumination images under complex environmental conditions, there are two important assumptions. One is that different areas will have different luminance, and the other is that similar patches have close similarity. Obeying on the above assumptions, Wu et al. [12] designed an improved Retinex method with the help of non-local similarity decomposition. The experimental results show that the method they used has brought a good enhancement result for low-light images. At the same time, color constancy is also ensured by keeping hue and saturation unchanged.

Based on the above analysis, the author and his team present presented a method for low-illumination image enhancement, the specific framework is shown in Fig. 1. In view of reducing the color distortion of the enhanced image, we only enhance the luminance component in the low-illumination image. The luminance component separated from the color space is respectively enhanced by HF, GF, and Retinex enhancement. Then, the results obtained by the above three enhancement methods are transformed by discrete wavelet under the multi-cues fusion (MCF) framework, and then the principal component analysis (PCA) fusion [13] is performed according to their frequency domain attributes. Finally, the enhanced luminance component is output by an inverse discrete wavelet transform (DWT). The image enhanced by the proposed method has good validity, high robustness, and high color fidelity.

The rest of the paper is organized as follows. Section 2 introduces the involved related methods and Section 3 demonstrates the details of the MCF image enhancement. In Section 4, the experimental results of MCF are given and the author compares it with those results of some present popular methods. Finally, the author summarizes the whole paper and draw conclusions in Section 5.

2. Related Methods

2.1 Homomorphic Filtering

The premise of HF is to assume that the gray level of the image is composed of the irradiated and the reflected components. The gray image f(x,y) is taken first to separate the illuminance component i(x,y) and the reflection component r(x,y). For a convenient expression, the author makes z(x,y) equal to the value of the logarithm of the gray image f(x,y):

Taking the Fourier transform of both sides of Eq. (1), the author gets

where [TeX:] $$z(x, y) \Leftrightarrow Z(u, v), \ln i(x, y) \Leftrightarrow F_i(u, v),$$ and [TeX:] $$z(x, y) \Leftrightarrow Z(u, v), \ln i(x, y) \Leftrightarrow F_i(u, v),$$ The notation & leftrightarrow; says that the left-to-right arrow is the forward Fourier transform, and the right-to-left arrow is the inverse Fourier transform. Then, Z(u,v) is filtered by using a Gaussian high-pass filter,

where H(u,v) is the transfer function of the Gaussian high-pass filter and the distance between the frequency point (u,v) and filter center [TeX:] $$\left(u_0, v_0\right)$$ is [TeX:] $$D(u, v)=\sqrt[2]{\left(u-u_0\right)^2+\left(v-v_0\right)^2}.$$ [TeX:] $$G_h \text { and } G_l$$ are the high-frequency and low-frequency gain, respectively. [TeX:] $$D_0 \text { and } c$$ are the cutoff frequency and the constant controlling the sharpening of inclined plane, respectively. At the end of HF, a Fourier inverse transform is performed in the obtained results.

The enhanced grayscale image can be obtained by performing an exponential operation on the both sides of Eq. (5).

2.2 Guide Filtering

Let [TeX:] $$q \text { and } g_{g u i}$$ be the source image and the guide image, respectively. Then, for the i-th pixel of the output image, the calculation method is

where j is the pixel label of the local window and the filter kernel function is presented as follows:

where [TeX:] $$w_k$$ is the k-th kernel function window and [TeX:] $$\left|\omega_k\right|$$ is the number of pixels in [TeX:] $$w_k, \mu_\lambda \text{ and } \sigma_\lambda^2$$ is the mean and variance of [TeX:] $$g_{g u i},$$ respectively, and [TeX:] $$\varepsilon$$ is the smoothing factor.

2.3 Center-Surround Retinex

When adopting the center-surrounding Retinex enhancement algorithm, the two-dimensional Gaussian function is used as a corresponding convolution function usually, that is,

where c is the scale parameter of the Gaussian function, and the default value is 200. & lambda; is the scale constant, and the default value is 1. The above two-dimensional Gaussian center-surrounding function needs to meet the normalization conditions, namely,

where, both i and j are integers, then the output of the reflection component is

where * is a convolution operation symbol. At this point, the enhanced luminance component can be obtained after the data of the reflection component is normalized to [0,255].

3. Multi-Cues Fusion Image Enhancement

3.1 Multi-Cues Fusion Framework Structure

In this paper, low-light image enhancement is not limited to spatial or frequency domain enhancement, but if we combine combines them to one whole framework, we finally get better fusion results to improve people's perception ability in a low-light environment. The framework structure of the proposed method is shown in Fig. 1.

Firstly, after obtaining the low-illumination color image captured by camera, it will be processed in two ways, respectively. The first way is to gray the low-illumination color image directly, and then use HF to enhance the gray image on frequency domain. Then the enhanced gray image (named as [TeX:] $$g_1$$) will act as the guidance image for the later GF. The second way is to convert the low-illuminance color image to HSV color space and preserve the hue and saturation part. Then the proposed algorithm will use [TeX:] $$g_1$$ to guide the luminance component to result in the enhanced image (named as [TeX:] $$g_2$$). Then, the enhanced image (named as [TeX:] $$g_3$$) is obtained by using single-scale Retinex (SSR) to enhance [TeX:] $$g_2$$ based on center-surround Gaussian filter. The three enhanced images [TeX:] $$g_1, g_2, text{ and } g_3$$ are decomposed by DWT, respectively. The reconstructed image is fused by PCA, and then the enhanced luminance component based on MCF is obtained after inverse DWT (the detailed process is shown in Fig. 2). Finally, it is merged with the previous hue and saturation and converted to the RGB color coordinate to obtain the enhanced color image.

Fig. 1.

The framework structure of multi-cues fusion.

The content in the dashed box in Fig. 1 is the core method of MCF, and the specific algorithm steps are given below.

3.2 Multi-Cues Fusion Steps

Step 1: The original low-illuminance color image is grayed to [TeX:] $$I_{gray }$$ by the mean value of the red, green, and blue components of the original low-illuminance color image. The first step is a necessary preprocess step of HF whose ability is to separate multiplicative noise and remove it.

Step 2: The gray image [TeX:] $$I_{gray }$$ is homomorphic filtered by the method described in Section 2.1 and then the enhanced gray image [TeX:] $$g_1$$ is obtained by using Eq. (6). Although HF can remove a large amount of multiplicative noise, edge details will also be reduced in the process of HF. Therefore, GF is considered to enhance the edge details of [TeX:] $$g_1$$.

Step 3: Transferring the RGB color space of the original low-illumination color image to the HSV color space to obtain the Hue, Saturation and Luminance component, respectively. Specific steps can be referred to the literature [14]. This step is to prepare for the subsequent enhancement of the luminance component alone.

Step 4: Setting the gray image [TeX:] $$g_1$$ obtained in the second step as the guided image, the luminance component obtained in the third step is guided and filtered by Eq. (7) and the enhanced image [TeX:] $$g_2$$ is obtained at the end of this step. Edge details are enhanced through GF. But the second and the fourth steps have limited luminance increments for low illumination images. Therefore, we adopt the-state-of-the-art Retinex algorithm to enhance [TeX:] $$g_2$$.

Step 5: By using Eq. 11, the enhanced image [TeX:] $$g_3$$ based on the single-scale Retinex is obtained. At this point, we own the enhanced images from Steps 2, 4, and 5. For a better enhancement, they can be fused at a pixel level.

Step 6: The three enhanced images [TeX:] $$g_1, g_2 \text { and } g_3$$ are decomposed by discrete wavelet, respectively and fused by PCA. This sub-module is detailed in the fusion algorithm section of Section 3.3.

Step 7: When the fusion enhanced image obtained in the sixth step is gotten, the MCF algorithm will substitute it for the luminance component obtained from Step 3. So, the new luminance component will be merged with the Hue and Saturation components retained in the third step and converted into the RGB color space. Thus, an enhanced low-illumination color image will finally be generated.

3.3 Core Algorithm of Multi-Cues Fusion

To specifically illustrate the fusion algorithm in this paper, the dashed box of Fig. 1 is taken out separately. The processing details of the DWT decomposition redrawing and the PCA function are show in Fig. 2.

In the dashed box of Fig. 1, three gray-level enhanced images obtained after the three enhancement techniques are applied to the original image. However, their results are different and not very expected. So, they are integrated at a pixel level to obtain better enhancement effects. In recent years, there has been much research on the fusion technology of DWT decomposition and principal component decomposition. It is very effective for us to apply it to our MCF fusion algorithm in this way.

Here, the core steps of the fusion algorithm are given as follows:

Step 1: The three gray-enhanced images [TeX:] $$g_1, g_2 \text { and } g_3$$ gotten in the sixth step of Section 3.2 are firstly subjected to wavelet forward transformation. To accelerate the efficiency of the algorithm, only one layer of decomposition is carried out here to obtain the horizontal low frequency, horizontal high frequency, vertical low frequency, and diagonal high frequency of the three images, respectively.

Step 2: Pairwise fusion of PCA is performed among the obtained components with similar attributes (refer to the literature [15] for the specific process). For example, the horizontal low frequency [TeX:] $$L L_n^1 \text { of } g_1 \text { and } L L_n^2 \text { of } g_2$$ are fused, and the obtained fusion results are fused with the horizontal low frequency [TeX:] $$L L_n^3 \text { of } g_3$$, and so on.

Step 3: After obtaining the fused results from Step 2, image reconstruction is carried out by the inverse wavelet transform, and finally the fused enhanced image is obtained.

Fig. 2.

Schematic diagram of the fusion process.

4. Experimental Results and Analysis>

To verify the effectiveness and consistency of the MCF algorithm for low-illuminance image enhancement, the Phos dataset [16] is used. Each scene in the dataset contains five images with different shapes, colors and textures captured under different intensifications of light, and their resolution is all 511 × 340 pixels. In addition, for the convenience of testing the color restoration and the running time of the algorithm, we also used the image set provided by Kodak Company and shot by KODAK EKTAR25 camera with 35-mm focal length to test the consistency of the data. All with 512 × 768 resolution of those images are selected to ensure the consistency of the data. Compared to the MCF algorithm, there are seven common image enhancement algorithms, and all algorithms are implemented on the MATLAB 2022a platform. The configuration of the computer is Intel Core i7-10700k @3.8 GHz CPU, 32 G RAM, and the operating system is Windows 10. There are many parameters involved in MCF, including the HF kernel, GF, Retinex, and wavelet parameter. he specific configuration of the experimental environment is shown in Table 1.

Table 1.

Parameters setup in multi-cues fusion

4.1 Robust Consistency of Enhancement Effectiveness

The quality of low-illuminance images is related to ambient light. The Phos dataset contains many images with inconsistent ambient light, which are divided into five levels according to the illuminance level. The image marked with standard light in the dataset is the ideal image, whereas the other four images are successively divided into four different levels. Here, we first apply the MCF algorithm to four levels of low illumination image for enhancement, then a result is gotten that they can be employed to detect the relationship between illumination change and image enhancement effect.

In Fig. 3, it is seen that as the illuminance level gradually reduces, the visual perception becomes worse. But after processing it by the MCF algorithm, the falling degree of the enhanced images is not obvious, and the perception of the target of four prospects has been less decreased. It indicates the adaptability to ambient brightness and consistency of the proposed enhancement algorithm. Lining with avoiding bias in subjective judgment, we adopt many image quality indexes including mean absolute error (MAE), peak signal-to-noise ratio (PSNR), structural similarity image measurement (SSIM), and edge preserve index (EPI) which are the common evaluation indexes of full reference image quality [16] and average gradient (AG), Mean, Variance, Entropy, BRISQUE, and NIQE which are the common no reference image quality evaluation indexes [17,18].

Fig. 3.

Comparison results of the subject effect of the enhanced low-illumination images with different illuminance levels. The first row of (a) is a standard image in normal light and (b)–(e) columns are, respectively, noted as the original low illumination images with illuminance level 1 to 4 and the second row (f)–(i) are their corresponding enhanced images.

MAE is the average of absolute values between the standard image and the enhanced image. It is to be mentioned here that a higher value of the MAE implies a more obvious the brightness enhancement effect. PSNR is the ratio between the energy of the peak signal and the average energy of the noise, which can quantitatively describe the distortion degree of the enhanced image. The higher the value, the smaller the image distortion. SSIM is an index of the similarity between standard image and enhanced image, and it returns a value between 0 and 1. The higher the value, the better the enhancement effect. EPI is the ratio of edge details in the enhanced image to the standard image, and the value is also within the range of [0,1]. A value of EPI towards 1 for some enhanced images implies that the enhanced images have better contour detail with their reference images. AG is the average gradient reflecting the change of image gray level. The larger the value, the higher the gray level of the enhanced image and the clearer the image. Mean is the average gray value of the entire image. Variance is used to describe the dispersion degree of the gray value of the pixel, which reflects the richness of the image details. The greater the value, the richer the details. Entropy is used to measure the richness of information in an image. Entropy is a positive indicator. The larger the value, the richer the information contained in the image. NIQE is a no-reference image quality index and can compute image statistics based on normalized luminance values. The higher the value, the worse the quality of the image. BRISQUE is also a no-reference image quality index, which mainly includes three parts: extracting natural scene statistics, calculating feature vector, and predicting image quality score. The higher the final score, the worse the image quality.

In Fig. 4, the MAE change rate of the enhanced image is only 1.47% during the change from illuminance level 1 to 3. The change rate of illuminance from level 3 to 4 becomes 38.93%, which indicates that the enhancement effect of this algorithm is worse than that of the previous brightness level in the extremely low brightness environment. However, the change rates of the other three indicators during the change of illuminance from level 1 to level 4 are 11%, 19.84% and 26.21%, respectively. It proved that the proposed algorithm still has a good enhancement effect.

In Fig. 5, the AG difference between the enhanced image and the standard image almost remains unchanged, showing that the enhanced image does not decrease with the decrease in brightness. So, the expression ability of the image for detail comparison is consistent. From the perspective of the rate of change, the speed of mean, variance, and BRISQUE index is the fastest between illuminance 3 and illuminance 4, indicating that the enhancement of the proposed algorithm has a certain divergence under a very low illuminance. But, overall, there is little difference from the corresponding index of the standard image. It is worth noting that with the decrease of illumination, the image entropy and NIQE values of the enhanced image are better than those of the standard image, demonstrating that the enhancement processing which can make the image have more average information and more in line with the natural perception characteristics of human vision.

Fig. 4.

Comparison results of low-illumination images with different illuminance levels detected by full-reference image quality index: (a) MAE, (b) PSNR, (c) SSIM, and (d) EPI.

Fig. 5.

Comparison results of low-illumination images with different illuminance levels detected by no-reference image quality index: (a) AG, (b) mean, (c) variance, (d) entropy, (e) BRISQUE, and (f) NIQE.

4.2 Enhancement Effectiveness of Different Algorithms

Low-illumination enhancement, especially very low-illumination image, is a very important sample for testing the effectiveness of the image enhancement algorithm. The MCF algorithm and the common image enhancement algorithms such as CLAHE, McCann Retinex (McCann), SSR, multi-scale Retinex (MSR), multi-scale Retinex with color restoration (MSRCR), DWT, and HF are applied to the lowest-level illumination image by means of making a full comparison to obtain a more comprehensive understanding of the MCF algorithm.

Though it can be seen from the results of Fig. 6(b) and 6(d) that the image brightness can increase by using CLAHE algorithm or SSR algorithm, the edge of the target in the image appears over-bright phenomenon, and the color appears to show large distortion. Through the comparison among Fig. 6(a), 6(c), and 6(j), the McCann algorithm has limited brightness enhancement to those which are low illumination image, small contrast between target and background, and large color distortion. The common feature of the results brought by Fig. 6(g) and 6(h) is that the brightness enhancement is limited, while the difference is that the results of the DWT enhancement have better color restoration. When Fig. 6(e) is compared with Fig. 6(g) and 6(h), although the brightness of the MSR enhancement is higher than that of the DWT and the HF enhancement, the contrast is still lower. With the increase of brightness in Fig. 6(f), the color distortion is relatively small. However, there are false outlines on the edge of the foreground target, which hinders people's perception and understanding of the target. By comparing Fig. 6(i) and 6(j), the image enhanced by the MCF algorithm is more realistic than the standard image in color performance. The enhancement of the foreground target and background highlights the characteristics of better contrast, and the enhanced image is more natural than those obtained by other algorithms. Considering that simply using some image samples in the PHO image database cannot fully reflect the effectiveness of the method presented in this paper, we conduct objectivity index tests on all image samples in the image database and achieve the corresponding statistical data. The division of the standard image and the test image is described in detail in Section 4.1.

Fig. 6.

Comparison results after enhancing the level 4 low-illuminant image by different algorithms. (a)–(j) are, respectively, noted as the original image, the enhanced image by CLAHE, and the enhanced image by McCann, SSR, MSR, MSRCR, DWT, HF, and the enhanced one by our proposed algorithm, and the standard image whose illuminance is level 0.

As it can be seen from the MAE index in Table 2 that the MCF method has the smallest average absolute error from the standard image, it has the highest increase in the pixel value for low-illumination image. Compared to other algorithms, its PSNR index is the largest one, numerically exceeding the HF index by 54.7%. The structural similarity index is the highest one, indicating that MCF can keep the image structure unchanged and close to the standard image structure under the premise of increasing the value of the image pixel. Compared to MSRCR algorithm, the edge preservation index of the MCF algorithm is only 6.82% lower and 5.7% higher than that of the third CLAHE algorithm, so it has good edge preservation ability. In terms of numerical value, MSRCR has the highest average gradient. Compared to the subjective visual effect of MSRCR, due to the appearance of many halos, and due to the fact that the increase of the halo edge can also increases the overall level of its gradient, so the image quality cannot be simply considered as the good one with a high average gradient. Meanwhile, the average gradient of the MCF algorithm is only 9.8% lower than that of the MSRCR, but its subjective enhancement effect is obviously better. It can be seen the MCF algorithm has the highest image mean and image entropy, which also shows that the enhancement effect is obvious and the enhanced image contains the most information. And the variance of the image is slightly lower than that of the SSR algorithm, showing that the enhanced pixel values have a sharp difference and the numerical distribution was relatively discrete. These three indicators are close to the image mean, variance, and entropy of standard images, respectively, which also proves that the MCF algorithm is not a blind enhancement, but closer to the image obtained by ideal illumination in the process of enhancement, which reflects the adaptability to environmental illumination. The last two indicators in Table 1, namely BRISQUE and NIQE, are key indicators of whether the enhancement effect is in accordance with human visual perception. From the data in Table 1, the value obtained by MCF is 40.5% lower than the average BRISQUE value of the seven enhancement algorithms and 15.2% lower than the lowest SSR. In addition, the NIQE index of the MCF algorithm is 1.36 times lower than the average NIQE of the seven enhancement algorithms and 37.9% lower than the image obtained by ideal illumination, which proves the truth of the strong effectiveness of the MCF algorithm for low-illumination image.

Table 2.

Comparison table of image quality index after enhancing the low illumination image

4.3 Color Fidelity and Run-Time

In Fig. 7, the first row is the color image obtained in a complex environment, and the second row is the image obtained after being enhanced by the MCF algorithm. In terms of subjective visual perception, the enhanced image not only owns improved brightness, but also owns small color distortion and full detail performance.

Fig. 7.

The enhancement results of the proposed algorithm for the Kodak image library. The first row is, respectively, noted as the original low illumination images and the second row are their corresponding enhanced images.

In Fig. 8(a), it shows the running-time comparison diagram of various algorithms when the statue image in the first row and the first column of Fig. 7 (the scale is 768 × 512) is enhanced. The running time of the MCF algorithm is the longest among the eight algorithms, which is 32.4% longer than that of the McCann algorithm. To avoid the random nature of the sample collection, we count all the image enhancement times in the Kodak image dataset, and the distribution characteristics of the running time are represented by a box diagram in Fig. 8(b).

Fig. 8.

Comparison of the run-time of algorithms: (a) statue image enhancement and (b) Kodak image library enhancement.

5. Conclusion

This paper presents a low illumination image enhancement method based on MCF. The method owns high validity and consistency for low illumination images especially a very low illumination image. Compared to other popular methods, the MCF method usually has some expected competitive advantages, especially in the natural image perception quality index. For the purpose of achieving a better performance image enhancement quality, the algorithm complexity of the proposed method will be more, but the time consumption will not be much more. Comparative experiments show that the MCF algorithm is a robust and effective low illumination image enhancement algorithm. The subsequent research in this paper will continue in the angles of two aspects: one is to optimize the algorithm to accelerate the real-time performance of image enhancement, and the other is to apply the MCF method to the real world to obtain more test data and application requirements.

Conflict of Interest

The authors declare that they have no competing interests.

Funding

This work was supported in part by University Level Scientific Research Project of Anhui Polytechnic University (No. Xjky072019A02) and University Synergy Innovation Program of Anhui Province (No. GXXT-2019-007).