Hue–saturation–intensity and texture feature-based cloud detection algorithm for unmanned aerial vehicle images

Abstract

There are a large number of cloud-covered areas in most unmanned aerial vehicle images and lead to the loss of information in the image and affect image post procession such as image fusion and target identification. Finding the cloud-occluded area in an image is a key step in image processing. Based on the differences of color and texture characteristics between cloud and ground, a cloud detection algorithm for the unmanned aerial vehicle images is proposed. Simulation results show that the proposed algorithm is better than the classical cloud detection algorithms in accuracy rate, false-positive rate, and kappa coefficient.

Keywords

Cloud detection multi-feature utilization color and texture characteristics image brightness image saturation

Introduction

Based on the long-term observations of International Satellite Cloud Climatology Project in Li et al.,¹ 66.7% of the Earth is covered by clouds. So, there are clouds in most unmanned aerial vehicle (UAV) images that lead to the loss of information in the image and affect image post procession such as image fusion and target identification. So, cloud detection evokes much attention and research result. The difference of blue value between cloud and non-cloud area is utilized to detect the thick cloud in Wang and Huang.² Digital elevation model is combined with F-mask algorithm in Qiu et al.³ to decrease the interference of the cloudlike area on the detection results. Multiple detection charts of multiple thresholds are used to distinguish cloud and shadow of cloud in Zhong et al.⁴ The above methods fit for specific sensors and scenes as relying on the spectrum and physical characteristics. In Zhang and Xiao,⁵ the bilateral filter in edge-preserving filter is used to decompose the image at multiple scales, and the texture is obtained to distinguish the cloud from the non-cloud target. Texture feature is used in Shi et al.⁶ to merge pixel blocks with similar characteristics after super-pixel segmentation and establishes a corresponding segmentation map model, which can better distinguish cloud and non-cloud regions. With the development of artificial intelligence, fully convolutional networks for pixel division is combined with network for edge detection in Yuan et al.⁷ Super-pixel division method is combined with convolutional neural networks (CNN) to distinguish thick and thin clouds in Liu et al.⁸ The thin and thick clouds are detected from remote sensing images using 10 bands of spectral data based on the multiscale eeatures-CNN in Shao et al.⁹ In Zhiyuan et al.,¹⁰ the cloud and cloud shadows are detected in remote sensing images based on multilevel feature fused segmentation network. But the above pattern recognition methods need a lot of training datum, and there is no high precision training datum available.

The relative characteristics of cloud and ground are as follows: that the cloud texture hopping and structural level difference is little, there are rich details in the non-cloud area and the textures of high reflective ground area are complex. By synthesizing color and detail characteristics, a general cloud detection algorithm for high resolution UAV images is proposed with higher accuracy and lower false detection rate for image cloud detection.

The paper is organized as follows: In the second section, color and texture features of high resolution UAV images are introduced. The cloud detection algorithm is described in the third section. Simulation results are shown in the fourth section. Finally, the conclusion of this work is summarized in the fifth section.

Color and texture features of high-resolution unmanned aerial vehicle images

Generally, there are two contrast features of the cloud and non-cloud area in UAV images.

According to Zhang and Xiao,⁵ The reflection rate of cloud area is larger than the non-cloud area, so the brightness of cloud area is greater than that of non cloud area, while the hue and saturation of the cloud area is low.

According to Zhang and Xiao,⁵ the areas covered by clouds usually have less detail information than those without clouds in color images. The cloud area is smooth while the texture information of the non-cloud area is richer.

According to the first feature, we can extract significance map of the cloud area in hue–saturation–intensity (HSI) space of a UAV image. Based on the second feature, we can diminish non-cloud area with high reflection rate lying in the significance map by edge detection using the texture feature. So, from the features above, we can get preliminary division of cloud and non-cloud area by means of brightness, hue, and texture of the UAV images.

Algorithm flow

Based on the color and texture features of high resolution UAV images, we propose a cloud detection algorithm in combination with the common detail detection algorithms. The detailed flow is shown in Figure 1.

1. Extract the significance map based on HSI.

i. Transfer the image from RGB color space to HSI

Figure 1.

Flowchart of HSI and texture feature based cloud detection algorithm. HSI: hue–saturation–intensity.

The brightness of cloud area in UAV image is generally higher, and cloud area can be preliminarily screened by brightness. The mean of R, G, and B components are taken as the intensity value of brightness in Zhang and Xiao.⁵ But it cannot reflect the brightness well owing to the RGB color model is one of the non-uniform color models. HSI color space is designed according to the visual characteristics of human eyes and it can extract cloud region preliminarily. Transfer the image from RGB color space to HSI according to formula (1)

\begin{array}{l} H = {\begin{matrix} θ, G \geq B \\ 2 π - θ, G < B \end{matrix} \\ I = 0.30 R + 0.59 G + 0.11 B \end{array}

where H is tonal component, I is brightness component, and θ can be calculated by

\begin{array}{l} θ = arccos (\frac{(R - G) - (R - B)}{2 (\sqrt{{(R - G)}^{2} - (R - B) (G - B)}}) \end{array}

ii. Extract the significance map according to formula (2)

\begin{array}{l} Smap = \frac{I^{'} + e}{2 \times (H^{'} + e)} \times 255 \end{array}

where

\begin{array}{l} I^{'} = \frac{I}{255} \\ e = 1 \\ H^{'} = \frac{H}{255} \end{array}

and the symbol Smap in formula (2) means significance map.

iii. Threshold segmentation of significance map using the method in Otsu¹¹ and get X ₁.

2. Construct detail image and take preliminary detection

Cloud areas are generally smooth and have fewer textures in UAV images. On the contrary, non-cloud areas have complex details and many textures. Therefore, non-cloud areas can be extracted from texture features.

i. Take X ₁ as the input image, extract edge details using Sobel algorithm in Conzalez and Woods,¹² get the horizontal G ₁ and vertical gratitude G ₂, get detail image X_d according to formula (3)

\begin{array}{l} X_{d} = \sqrt{G_{1} ​^{2} + G_{2} ​^{2}} \end{array}

ii. It is just a regular culling operation. Using OTSU method for detail image X_d , detail binary graph ${X^{'}}_{d}$ is obtained. Exclude the area with rich details from X ₁ and get rough detection result X ₂

\begin{array}{l} X_{2} = X_{1} \land {X^{'}}_{d} \oplus X_{1} \end{array}

Through ∧ operation, we can get the rich details of terrain in X ₁. Through ⊕ operation, these abundant details can be removed from X ₁, and X ₂ can be obtained.

3. Detailed detection and refinement process

i. Extract and exclude the connected area little than s using the connected region notation in morphological image processing,¹³ where $s \in [200, 800]$ .

ii. After removing the connected area whose number of pixels is less than s by the first step, we get ${X^{'}}_{2}$ . Then, use 3 × 3 filter window to median filter ${X^{'}}_{2}$ , as follows

\begin{array}{l} X_{3} (M, N) = median ({X^{'}}_{2} (m, n)) \end{array}

where

\begin{matrix} \begin{matrix} m \in [M - 1, M + 1] \end{matrix} \\ n \in [N - 1, N + 1] \\ M \geq 1 \\ N \geq 1 \end{matrix}

where $(M, N)$ is the two-dimensional coordinates of the pixels in image X ₃, $(m, n)$ is the two-dimensional coordinates of the pixels in image ${X^{'}}_{2}$ , and $median$ is a median operation. That is to say, the pixels contained in the 3 × 3 filtering window in ${X^{'}}_{2}$ are arranged from small to large. Then, the pixel values in the middle position are assigned to $X_{3} (M, N)$ . When the boundary pixels of ${X^{'}}_{2}$ are filtered, the area covered by the filter window will exceed the actual area of ${X^{'}}_{2}$ . Therefore, the outermost pixel value of ${X^{'}}_{2}$ is copied to fill the boundary. By means of median filtering, each pixel in X ₃ can be close to the pixel values of the surrounding pixels. The edge details of the image are preserved while the scattered salt and pepper noise is removed.

iii. Take fine segmentation of the edge of the pending cloud area in X ₃ using guided image filtering in He et al.¹⁴

\begin{array}{l} X_{guided} = F (X_{3}, X, r, ε) \end{array}

where F is the guided image filter, X is the input raw image, X ₃ is the image to be filtered, $X_{guided}$ is the filtered output image, r is the window radius, ε is the regularization parameter and $ε = 0.000001$ . Under different spatial resolutions, the ambiguity of cloud edges is different, and the average width of translucent edge areas is also different. Hence, the window radius should be selected according to the spatial resolution of the remote sensing image to be detected. Generally, $r \in [35, 120]$ .

iv. Take binarization of $X_{guided}$ using minimum cross entropy threshold in Li and Lee¹⁵ and get the last detection result $X_{result}$ .

Through the above steps, the difference in brightness, saturation, and detail information between cloud and ground can be comprehensively utilized. For the case that the brightness of non-cloud area and cloud area is basically the same due to reflection and other reasons, it can be effectively distinguished by detail processing.

Simulation results

To test the detection effectiveness and superiority of the proposed algorithm, on the one hand, four different raw UAV images will be detected, as shown in Figure 2. The images used include UAV images with simple, complex, and man-made terrains. And the manual detection results for cloud areas in four raw images are also shown in Figure 2. They will be used as ideal detection results to judge the accuracy of cloud detection algorithms. On the other hand, we take K-means in Lin et al.,¹⁶ mean-shift in Comaniciu and Meer,¹⁷ Chan–Vese in Vese and Chan¹⁸ as compared algorithms and take subjective and objective assessment.

Figure 2.

(a) Raw and (b) manually labeled cloud area images.

Accuracy rate, false-positive rate, and kappa coefficient are used to assess the detection result quantitatively. Kappa coefficient k is between 0 and 1 and is the indicator for measuring classification accuracy. Detailed correspondence is as follows:

if $k \in [0.21, 0.40]$ , it indicates that the result generally consistent to the truth;

if $k \in [0.41, 0.60]$ , it indicates that the result medium consistent to the truth;

if $k \in [0.61, 0.80]$ , it indicates that the result highly consistent to the truth;

if $k \in [0.81, 1]$ , it indicates that the result is almost identical to the truth.

Accuracy rate A is defined as

\begin{array}{l} A = \frac{T P + T N}{P + N} \end{array}

where $T P$ (true positive) is the number of the real cloud pixels detected as cloud, $T N$ (true negative) is the number of the non-cloud pixels detected as non-cloud pixel, P is the number of real cloud pixels, and N is the real non-cloud pixels.

False positive rate F is defined as

\begin{array}{l} F = \frac{F P}{T P + F P} \end{array}

where $F P$ (false positive) is the number of the non-cloud pixels detected as cloud pixel.

And kappa coefficient is defined as

\begin{array}{l} Kappa = \frac{p_{a} - p_{e}}{1 - p_{e}} \end{array}

where

\begin{array}{l} p_{a} = \frac{T P + T N}{T P + T N + F P + F N} \\ ​ \\ p_{e} = \frac{P (T P + F P) + N (F N + T N)}{{(P + N)}^{2}} \end{array}

Figure 3 shows the first original image, the manual labeling cloud area and its cloud detection results using K-means, mean shift, Chan–Vese, and the algorithm proposed in this paper. It is shown in Figure 3 that the K-means, mean-shift, and Chan–Vese algorithms often misjudge the non-cloud area with similar brightness to the cloud, while the test result of the proposed algorithm is closest to the manual labeled image and it takes much less misjudge as utilization both the brightness and texture characters. The values of accuracy rate, false positive rate, and kappa coefficient of K-means algorithm are 70.9, 73.2, and 0.23, respectively. The values of accuracy rate, false-positive rate, and kappa coefficient of mean shift algorithm are 74.1, 71.9, and 0.25, respectively. The values of accuracy rate, false-positive rate, and kappa coefficient of Chan–Vese algorithm are 67.1, 75.7, and 0.20, respectively. In contrast, the proposed algorithm only includes the detected cloud areas in the detection image. The values of accuracy rate, false-positive rate, and kappa coefficient of proposed algorithm are 95.3, 0.2, and 0.95, respectively. Obviously, the detection indexes of the proposed algorithm are the best.

Figure 3.

Detection results contrast of the first raw image: (a) raw, (b) manual, (c) k-means, (d) mean shift, (e) Chan–Vese, and (f) proposed.

Figure 4 shows the second original image, the manual labeling cloud area and its cloud detection results using K-means, mean shift, Chan–Vese, and the algorithm proposed in this paper. It is shown in Figure 4 that the K-means, mean shift, and Chan–Vese algorithms often misjudge the non-cloud area with similar brightness to the cloud, while the test result of the proposed algorithm is closest to the manual labeled image and it takes much less misjudge as utilization both brightness and texture characters. The values of accuracy rate, false-positive rate, and kappa coefficient of K-means algorithm are 73.5, 68.2, and 0.34, respectively. The values of accuracy rate, false-positive rate, and kappa coefficient of mean shift algorithm are 67.1, 71.9, and 0.29, respectively. The values of accuracy rate, false-positive rate, and kappa coefficient of Chan–Vese algorithm are 76.7, 64.9, and 0.39, respectively. In contrast, the proposed algorithm only includes the detected cloud areas in the detection image. The values of accuracy rate, false-positive rate, and kappa coefficient of proposed algorithm are 93.6, 29.4, and 0.72, respectively. Obviously, the detection indexes of the proposed algorithm are the best.

Figure 4.

Detection results contrast of the second raw image: (a) raw, (b) manual, (c) k-means, (d) mean shift, (e) Chan–Vese, and (f) proposed.

Figure 5 shows the third original image, the manual labeling cloud area and its cloud detection results using K-means, mean shift, Chan–Vese, and the algorithm proposed in this paper. It is shown in Figure 5 that the K-means, mean shift, and Chan–Vese algorithms often misjudge the non-cloud area with similar brightness to the cloud, while the test result of the proposed algorithm is closest to the manual labeled image and it takes much less misjudge as utilization both brightness and texture characters. The values of accuracy rate, false-positive rate, and kappa coefficient of K-means algorithm are 91.1, 17.3, and 0.80, respectively. The values of accuracy rate, false-positive rate, and kappa coefficient of mean shift algorithm are 90.3, 22.8, and 0.79, respectively. The values of accuracy rate, false-positive rate, and kappa coefficient of Chan–Vese algorithm are 91.9, 15.8, and 0.82, respectively. In contrast, the proposed algorithm only includes the detected cloud areas in the detection image. The values of accuracy rate, false-positive rate, and kappa coefficient of proposed algorithm are 97.4, 0.4, and 0.94, respectively. Obviously, the detection indexes of the proposed algorithm are the best.

Figure 5.

Detection results contrast of the third raw image: (a) raw, (b) manual, (c) k-means, (d) mean shift, (e) Chan–Vese, and (f) proposed.

Figure 6 shows the fourth original image, the manual labeling cloud area and its cloud detection results using K-means, mean shift, Chan–Vese, and the algorithm proposed in this paper. It is shown in Figure 6 that the K-means, mean shift, and Chan–Vese algorithms often misjudge the non-cloud area with similar brightness to the cloud, while the test result of the proposed algorithm is closest to the manual labeled image and it takes much less misjudge as utilization both brightness and texture characters. But as some part of the Figure 6 is smooth with high reflection rate which is similar to the features of the cloud area, so all the algorithms take it as cloud. The values of accuracy rate, false-positive rate, and kappa coefficient of K-means algorithm are 91.8, 9.9, and 0.80, respectively. The values of accuracy rate, false-positive rate, and kappa coefficient of mean shift algorithm are 70.7, 48.9, and 0.45, respectively. The values of accuracy rate, false-positive rate, and kappa coefficient of Chan–Vese algorithm are 93.1, 5.6, and 0.83, respectively. In contrast, the proposed algorithm only includes the detected cloud areas in the detection image. The values of accuracy rate, false-positive rate, and kappa coefficient of the proposed algorithm are 95.1, 2.9, and 0.88, respectively. Obviously, the detection indexes of the proposed algorithm are the best.

Figure 6.

Detection results contrast of the fourth raw image: (a) raw, (b) manual, (c) k-means, (d) mean shift, (e) Chan–Vese, and (f) proposed.

Tables 1 to 3 present the comparison of detection indexes of different algorithms for different images.

Table 1.

Accuracy rate.

	Accuracy rate (%)
Figures	K-means	Mean shift	Chan–Vese	Proposed
Figure 3	70.9	74.1	67.1	95.3
Figure 4	73.5	67.1	76.7	93.6
Figure 5	91.1	90.3	91.9	97.4
Figure 6	91.8	70.7	93.1	95.1
Mean	81.8	75.6	82.2	95.4

Table 2.

False-positive rate.

	False-positive rate (%)
Figures	K-means	Mean-shift	Chan–Vese	Proposed
Figure 3	73.2	71.9	75.7	0.2
Figure 4	68.2	71.9	64.9	29.4
Figure 5	17.3	22.8	15.8	0.4
Figure 6	9.9	48.9	5.6	2.9
Mean	42.2	53.9	40.5	8.2

Table 3.

Kappa coefficient.

	Kappa (%)
Figures	K-means	Mean shift	Chan–Vese	Proposed
Figure 3	0.23	0.25	0.20	0.95
Figure 4	0.34	0.29	0.39	0.72
Figure 5	0.80	0.79	0.82	0.94
Figure 6	0.80	0.45	0.83	0.88
Mean	0.54	0.45	0.56	0.87

From Tables 1 to 3, we can see that the accuracy rate of the proposed algorithm is much higher and the false-positive rate is much lower than the other algorithms. Taking the kappa coefficient, we can see that the detection result of the proposed method is almost identical to the man judged and better than the other three obviously.

Conclusion

In this article, the brightness, hue, and texture difference of cloud and non-cloud area in high-resolution UAV images are utilized to put forward a general cloud detection algorithm. The simulation results show the high accuracy, low false-positive rate of the algorithm, and its high consistency to the manual judgment.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Wang Yi

References

, et al. Spatial distribution and seasonal variation of cloud over China based on ISCCP data and surface observations. J Meteorol Soc Jpn Ser II 2004; 82(2): 761–773.

Wang

Huang

Cloud detection for landsat images by fusion of multi-temporal data. In: International conference on audio, language and image processing (ICALIP), Shanghai, China, 16–17 July, 2018, pp. 333–337. IEEE.

Qiu

Zhu

, et al. Improving Fmask cloud and cloud shadow detection in mountainous area for Landsats 4–8 images. Remote Sens Environ 2017; 199: 107–119.

Zhong

Chen

, et al. A cloud detection method based on relationship between objects of cloud and cloud-shadow for Chinese moderate to high resolution satellite imagery. IEEE J Sel Top Appl Earth Obs Remote Sens 2017; 10(11): 4898–4908.

Zhang

Xiao

. Cloud detection of RGB color aerial photographs by progressive refinement scheme. IEEE T Geosci Remote Sens 2014; 52(11): 7264–7275.

Shi

Wang

, et al. Ground-based cloud detection using graph model built upon superpixels. IEEE Geosci Remote Sens Lett 2017; 14(5): 719–723.

Yuan

Meng

Cheng

, et al. Efficient cloud detection in remote sensing images using edge-aware segmentation network and easy-to-hard training strategy. In: IEEE international conference on image processing (ICIP), Beijing, China, 17–20 September, 2017, pp. 61–65. IEEE.

Liu

Zeng

Tian

Super-pixel cloud detection using hierarchical fusion CNN. In: IEEE fourth international conference on multimedia big data (BigMM), Xi’an, China, 13-16 September, 2018, pp. 1–6. IEEE.

Shao

Pan

Diao

, et al. Cloud detection in remote sensing images based on multiscale features-convolutional neural network. IEEE T Geosci Remote Sens 2019; 57(6): 4062–4076.

10.

Zhiyuan

Menglong

Hao

, et al. Cloud and cloud shadow detection using multilevel feature fused segmentation network. IEEE Geosci Remote Sens Lett 2018; 15(10): 1600–1604.

11.

Otsu

. A threshold selection method from gray-histogram. IEEE Trans Syst Man Cybern 1979; 9(1): 62–66.

12.

Conzalez

Woods

. Digital image processing. 3rd ed. Beijing: Publishing House of Electronics Industry, 2017.

13.

Wood

Field

. The distribution of cloud horizontal sizes. J Climate 2011; 24(18): 4800–4816.

14.

Sun

Tang

. Guided image filtering. IEEE Trans Pattern Anal Mach Intell 2013; 35(6): 1397–1409.

15.

C H

Lee

. Minimum cross entropy thresholding. Pattern Recognit 1993; 26(4): 617–625.

16.

Lin

, et al. Image segmentation using the K-means algorithm for texture features. World Acad Sci Eng Technol 2010; 4(5): 945–948.

17.

Comaniciu

Meer

. Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 2002; 24(5): 603–619.

18.

Vese

Chan

. A multiphase level set framework for image segmentation using the Mumford and Shah model. Int J Comput Vision 2002; 50(3): 271–293.