Sage Journals: Discover world-class research

Abstract

Background

X-ray is an effective measure in the diagnosis of coronavirus disease 2019. However, it suffers from low visibility and poor details. A plausible solution is to decompose the captured images and enhance the details. The bilateral weighted least square model can be an effective tool for this task. However, it is highly computationally expensive.

Method

In this article, we propose an efficient algorithm for the bilateral weighted least square model. We approximate the bilateral weight with the bilateral grid and then incorporate it into the optimization model. This significantly reduces the number of variables in the linear system. Therefore, the model can be efficiently solved. We employ the proposed algorithm to decompose the input X-rays into base and detail layers. The detail layers are then boosted and added back to the input to derive the detail-enhanced results.

Results

The subjective results indicate that our method achieves higher contrast than the best-performing method ( $442.30 > 410.09$ , $426.40 > 403.34$ , $564.51 > 531.38$ ). Furthermore, our method is highly efficient. It takes 0.92 s to process a 720P color image on an Intel i7-6700 CPU. The objective results derive from the chi-square test indicate that subjects hold more positive attitudes toward our detail-enhanced images than the original X-ray images ( $3.53 > 2.72$ , $3.42 > 2.61$ , $3.5 > 2.56$ ).

Conclusion

We have conducted extensive experiments to evaluate the proposed image detail enhancement method. It can be concluded that (1) our method could significantly improve the visibility of the X-ray images. (2) our method is fast and effective, thus facilitating real applications.

Keywords

X-ray image detail enhancement coronavirus disease 2019 bilateral weighted least square

Introduction

The coronavirus disease 2019 (COVID-19) is extremely contagious due to the high basic reproduction number (R0).^1–4 During the last three years, it has spread rapidly across the world. The booming development of COVID-19 has imposed a heavy burden on the control of the disease. Even though the impact of the virus is being alleviated, the number of reported reinfections is increasing.^5–7 A timely diagnosis of COVID-19 could prevent the spread of the virus. However, the dominant variants^8,9 of COVID-19 are characterized by a range of clinical signs of asymptomatic infection,^10,11 which complicates the diagnosis.

Chest radiography (X-ray) is an essential measure for human experts to diagnose COVID-19 disease.^12,13 Moreover, researchers have also attempted to investigate deep learning-based approaches for the diagnosis of COVID-19. Comprehensive studies of these methods can be found in existing surveys.^14,15 However, due to the low dynamic range and contrast of the X-ray images, subtle but meaningful details may be challenging to identify for both humans and machines. Therefore, it is critical to enhance the details so that the visibility of the X-ray images can be improved.

Detail enhancement refers to the process to enhance the details of images. It can be leveraged to improve the clarity of X-ray images. The early attempts on this task are based on the histogram equalization and its variants, such as the adaptive histogram equalization,¹⁶ contrast limited adaptive histogram equalization,^17,18 and normalized contrast limited adaptive histogram equalization.¹⁹ The Histogram equalization-based methods could promote the image details, but they may also increase the contrast of noise and suppress the contrast of meaningful structures.

Recent methods for detail enhancement mostly rely on image decomposition with edge-aware smoothing filters, such as the bilateral filter,^20,21 anisotropic diffusion filter,²² guided filter,^23,24 soft clustering filter,²⁵ $L_{1}$ reconstruction filter,²⁶ $L_{0}$ filter,²⁷ generalized smoothing filter,²⁸ and weighted least square filter.²⁹ The decomposition-based methods consider the pixels in the neighborhood during the enhancement process, thus could alleviate the problems of the histogram equalization-based methods.

It is worth noting that, among the decomposition-based methods, the weighted least square (WLS) filter²⁹ is shown to be particularly useful for detail manipulation, that is, it can significantly enhance the details without causing the gradient reversal or halo artifacts. However, the weighted design only involves the range, making it less edge-aware. Moreover, similar to many methods based on global optimization, it is highly time-consuming, as it requires to solve a linear system with a large inhomogenous Laplacian coefficient matrix.

With the booming advancement of the deep learning technique, we are seeing significant progress in the field of medical data/image processing, such as COVID-19 prediction,³⁰ segmentation of cervical cytology,³¹ thyroid³² and iris,³³ pulse extraction,³⁴ and breast cancer detection.³⁵ Different from these researches, our objective is to enhance the details of the COVID-19 chest X-ray images, thus aiding the diagnosis. We intend to develop a method that is not learning-based, and can be more stable and interpretable.

To facilitate our objective, we develop a novel decomposition-based image detail enhancement method, where we extract the detail layer, boost it, and add it back to the input image. The main contributions of this article are:

We propose a bilateral WLS model for the task of the X-ray image detail enhancement. The bilateral weighted scheme involves not only the range, but also the domain, thus can be more edge-aware.

We propose a fast solution to our bilateral WLS model, where we approximate the bilateral weight with the bilateral grid. This significantly reduces the size of the problem, thus can be highly efficient.

We have conducted subjective and objective experiments to evaluate our method. The subjective results show that our method achieves better contrast than existing methods. The objective results indicate that our method can improve the visibility of COVID-19 chest X-ray images.

The rest of the article is organized as follows. In the “Literature review” section, we summarize relevant methods. The “Method” section covers the details of the proposed method. Both objective and subjective experiments are presented in the “Experimental settings” section and both objective and subjective evaluations are presented in the “Experimental results” section. We discuss the proposed model in the “Discussion” section. Finally, we conclude the article and present future directions of research in the “Conclusion and future work” section.

Literature review

The most important step in our detail enhancement method is image decomposition. In this section, we will focus on various smoothing filters for image decomposition, which can be mainly divided into two categories, namely local and global methods. The former ones are based on explicitly define filter kernels, while the latter ones are based on sophisticated global optimization models.

Local methods

The bilateral filter³⁶ is perhaps the most prevalent non-linear filter for edge-preserving smoothing. It consists of a spatial kernel and a range kernel. Both of the two kernels are mostly Gaussian functions that are used to penalize pixels that are distant in space and range. Even though there are various artifacts, such as halos and gradient reversals, the bilateral filter is still effective in practice. The main impediment is the high computational cost.

Over the last decade, researchers have made various attempts to improve the efficiency of the bilateral filter. Most of the methods are built upon range quantization,^37,38 range decomposition,^39,40 or spatial subsampling.²⁷ These strategies significantly accelerate the bilateral filter, but they may not be able to improve the quality of image decomposition over the naive bilateral filter.

Another popular local method is the guided image filter.²³ The basic assumption in the work is that there exist local linear relations between the corresponding patches of the output and guidance images. Based on this assumption, the authors build a set of local linear regression problems that have closed-form solutions. The guided image filter can be computed efficiently. However, it suffers from severe halo artifacts in image decomposition.

In order to alleviate the halo artifacts that haunt the guided image filter, various strategies have been proposed, such as the anisotropic guided filter,²⁴ the weighted guided filter with steering kernel,⁴¹ and the soft clustering guided filter.⁴² These strategies all focus on the weighted averaging operation in the final step. These variants of guided filter significantly suppress the halo artifacts. However, the improvement is at the cost of sacrificing efficiency.

Global methods

The WLS filter²⁹ is the pioneering work for global image decomposition. It seeks to minimize the $L_{2}$ loss between the input and output regularized by the WLS of the output gradients. The model is shown to be particularly useful for detail enhancement, as it alleviates the halo and gradient reversal artifacts. However, the absence of the domain in the weighted design limits its edge-aware power. Furthermore, it can be time-consuming to solve. Though there exist separable 1D approximations,^43,44 they may introduce severe streak artifacts.

The $L_{1}$ reconstruction model²⁶ optimizes the $L_{2}$ fidelity term regularized by the $L_{1}$ gradient reconstruction loss. The $L_{1}$ regularization enforces the sparsity of the output gradients, thus better facilitating edge awareness. The authors propose a novel mapping function to compress the small gradients while preserving large gradients, which further promote the edge awareness. However, the lack of the weighted scheme limits its performance.

The iterative least square filter⁴⁵ explores a non-convex penalty for regularization. It optimizes the $L_{2}$ fidelity term regularized by the generalized Charbonnier penalty on the gradients. The non-convex penalty function accommodates the gradient distribution of natural images. Therefore, the filter achieves promising performance on various tasks. Furthermore, it is highly efficient. However, it is inclined to the compartmentalization artifacts in detail enhancement.

The generalized smoothing filter²⁸ is designed to provide a framework for various edge-aware image processing tasks. The model is based on a novel truncated Huber penalty function, which is friendly to weak edges. Therefore, the resultant filter is strongly edge-aware. However, it tends to sharpen the salient edges, thus may be subject to the gradient reversal artifacts. Moreover, the piece-wise objective function can be computationally expensive to solve.

Different from existing optimization-based image decomposition methods that impose gradient regularization, the soft clustering filter²⁵ models the task as image clustering similar to super-pixel segmentation. The partition matrix derived from the clustering process is then leveraged to construct the affinity matrix for image decomposition. The method accommodates a variety of applications, yet it may not be able to protect thin structures.

From the above discussion, we see that the local methods are mostly fast, but they suffer from poor quality. Moreover, the global methods suffer from artifacts in detail enhancement except for the WLS filter, which on the other hand, is time-consuming to solve. In this article, we propose a global method for image decomposition based on the bilateral WLS model. We consider not only the range, but also the domain in the weighted design, thus can be more edge-aware. Furthermore, we propose a fast solution to our model based on the bilateral grid.³⁸

Proposed method

Bilateral WLS model

The bilateral WLS model seeks to minimize the summed squared error regularized by the bilateral weighted $L_{2}$ norm of the gradients. The objective function of the model is

min_{X} \sum_{i} (X_{i} - P_{i})^{2} + \frac{λ}{2} \sum_{i} \sum_{j \in Ω (i)} W_{i, j} (X_{i} - X_{j})^{2}

(1)

where

W_{i, j}

is the bilateral affinity between pixels

i

and

j

and it is formally defined as

W_{i, j} = e^{- \frac{‖ j - i ‖_{2}^{2}}{2 σ_{s}^{2}} - \frac{{‖ P_{j} - P_{i} ‖}_{2}^{2}}{2 σ_{r}^{2}}},

(2)

where

σ_{s}

and

σ_{r}

are the spatial and range bandwidths, respectively, which are used to control the amount of smoothing. Different from the weighted scheme in the original WLS model²⁹ that considers only the range, the bilateral weight scheme incorporates both the range and domain, thus is deemed to be more edge-aware.

Rewriting equation (1) into the matrix form, we have

min_{X} (X - P)^{T} (X - P) + λ X^{T} (H - W) X

(3)

where

H

is the degree matrix derived from

W

. Specifically,

H

is a diagonal matrix, where the

i

th diagonal entry is the sum of the entries in the

i

th row of

W

H - W

is the inhomogeneous Laplacian matrix. The fidelity term

(X - P)^{T} (X - P)

ensures that the filtering output

X

does not deviate too far from the input

P

. The regularization term

X^{T} (H - W) X

enforces that the output

X

must be smoothed. The parameter

λ

balances the two seemingly contradicting terms.

Taking the derivative of equation (3) to $X$ , and let it be zero, we have

(I + λ (H - W)) X = P

(4)

where

I

is the identity matrix. Equation (4) is computationally expensive to solve, as it is a linear system with a large sparse coefficient matrix.

Efficient solver

Directly solving equation (4) can be time-consuming. Inspired by the bilateral grid,³⁸ which is essentially a fast approximation of the bilateral filter, we propose a fast solver to the bilateral WLS model. According to the idea of the bilateral grid, the bilateral weight can be approximated as:

W \approx S^{T} B S

(5)

where

S

stands for the splat matrix, and it is for hard quantization, where the entries can be either 0 or 1.

S

is a

n \times k

matrix, where

n

is the number of pixels and

k

is the number of grids.

S_{i, j} = 1

indicates that pixel

i

belongs to the grid

j

and

S_{i, j} = 0

indicates otherwise. The range and domain values are quantized into grids. The matrix

S

can then be derived by checking which grid the domain and range values of each pixel fall in.

B

is the matrix for the high-dimensional spatial-invariant Gaussian filtering.

S^{T}

is the slice matrix, and it is the transpose of

S

Based on equation (5), equation (3) can be approximated as

min_{X} (X - P)^{T} (X - P) + λ X^{T} (\tilde{H} - S^{T} B S) X

(6)

where

\tilde{H}

is the degree matrix derived from

S^{T} B S

. Specifically,

\tilde{H}

is a diagonal matrix, where the

i

th diagonal entry is the sum of the entries in the

i

th row of

S^{T} B S

We introduce an auxiliary variable $Y$ , and let

X = S^{T} Y

(7)

Assume the number of pixels and grids are

n

and

k

(

n >> k

Y

(

k \times 1

column vector) is in fact the grid representation of

X

(

n \times 1

column vector), then equation (6) becomes

min_{Y} (S^{T} Y - P)^{T} (S^{T} Y - P) + λ Y^{T} S (\tilde{H} - S^{T} B S) S^{T} Y

(8)

To solve the optimization problem shown in equation (8), we take the partial derivative to

Y

and let it be zero, then we have

(S S^{T} + λ S (\tilde{H} - S^{T} B S) S^{T}) Y = S P

(9)

where

S S^{T}

is a

k \times k

diagonal matrix, where each diagonal entry is the number of pixels in the corresponding grid, thus it can be evaluated efficiently. Note that equation (9) is also a linear system akin to equation (3). However, since

Y

contains much fewer unknowns than

X

, it can be much easier to solve, namely

Y = {(S S^{T} + λ S (\tilde{H} - S^{T} B S) S^{T})}^{- 1} S P

(10)

In this article, we leverage the preconditioned conjugate gradient algorithm⁴⁶ to solve the linear system.

When $Y$ is solved, we can derive $X$ by a simple matrix–vector multiplication $X = S^{T} Y$ according to equation (7), namely

X = S^{T} {(S S^{T} + λ S (\tilde{H} - S^{T} B S) S^{T})}^{- 1} S P

(11)

Since the matrix

S

is for hard quantization, the output may exhibit blocky artifacts. In order to eliminate the artifacts, we further apply the domain transform filter⁴⁷ on the results derived from the optimization model. The domain transform filter is extremely fast (

O (n)

), so it has a neglectable influence on the running time.

System model

To achieve detail enhancement, the input X-ray image is firstly decomposed into the base and detail layers with our fast bilateral WLS filter. The detail layer is then boosted ( $4 \times$ ) and added back to the input image to derive the detail-enhanced result. The flowchart of our enhancement method is shown in Figure 1.

Figure 1.

The flowchart of the proposed X-ray image detail enhancement method.

Our detail enhancement process can be mainly divided into three steps, which are detailed as follows:

Derive the base layer $X$ with the bilateral weighted least filter described in the previous sections.

Extract the detail layer $D$ by subtracting the base layer from the input image $P$ , as shown below:

D = P - X

(12)

The detail layer is then boosted and added back to the input image to derive the detail-enhanced image, that is,

R = P + t D

(13)

where

t

is a scalar specifying the extent of boost. Throughout this article, we set

t = 4

. Our detail enhancement method can be sketched as Algorithm 1.

Algorithm 1.

X-ray image detail enhancement

Input: image

P

, parameters

λ

σ_{s}

σ_{r}

t = 4

;

1: Solve equation (11) for the base layer

X

2: Derive the detail layer

D

with equation (12);

3: Derive the enhanced image

R

with equation (13);

Output: enhanced image

R

;

Our detail enhancement method is highly efficient. Assume that the number of pixels and grids are $n$ and $k$ , and the number of iterations of the preconditioned conjugate gradient algorithm⁴⁶ is $m$ , deriving the base layers by solving equation (11) takes $O (m k^{2} + n k)$ time. Deriving, boosting, and adding back the detail layer are element-wise operations, so the running time is $O (n)$ . Therefore, the overall time complexity of our detail enhancement method is $O (m k^{2} + n k)$ . On the other hand, if we derive the base layer by solving equation (4), it would take $O (m n^{2})$ . Considering that in practice $k << n$ , our method could significantly accelerate the detail enhancement process.

Experimental settings

We have conducted both objective and subjective experiments to evaluate the proposed X-ray image detail enhancement method. In this section, we will detail the experimental settings from these two aspects.

Objective evaluation

Metrics

For objective evaluation, we leverage the contrast measure as the objective metric to evaluate the quality of the detail-enhanced X-ray images, as shown below

C = \sum_{δ} δ (P_{i}, P_{j})^{2} E_{δ} (P_{i}, P_{j})

(14)

where

P_{i}

and

P_{j}

are the intensities of neighboring pixels.

δ (P_{i}, P_{j})

is the absolute difference between

P_{i}

and

P_{j}

, that is,

δ (P_{i}, P_{j}) = | P_{i} - P_{j} |

(15)

E_{δ} (P_{i}, P_{j})

is the probability distribution of the absolute difference

δ

. The contrast measure is a nice indicator of the visibility and the amount of details in an X-ray image.

Furthermore, we record and compare the running time (measured in seconds) of various methods on images with varying resolutions. Note that the running time of all methods is collected on a modern desktop with an Intel i7-6700 CPU, an Nvidia GTX 1070, and 16 GB memory. The running time could be an essential criterion to validate the practicability of the proposed method.

Dataset

The objective evaluation is conducted on the COVID-19 chest X-ray image dataset.⁴⁸ The dataset is initially developed to advance the detection of COVID-19 using chest X-rays. It contains 317 X-ray images in total, which can be mainly divided into three categories, that is, COVID-19, viral pneumonia, and normal chest X-rays. Our method is not learning-based and it does not require training. We test our method on the 61 images in the test dataset.

Methods

We compare the proposed method to a variety of prevalent methods. The full names, abbreviations, and references of all the methods involved in this experiment can be found in Table 1. We use the official implementations of these methods, which are provided by the authors. For the parameter settings, we best tune the parameters based on the recommended values for all methods.

Table 1.

Brief of the methods in our experiment.

Method	Reference	Abbreviation
Fast bilateral filter	Paris and Durand³⁸	FBF
Guided image filter	He et al.²³	GIF
Domain transform filter	Gastal and Oliveira⁴⁷	DTF
Mutual structure filter	Shen et al.⁴⁹	MSF
Weighted least square	Farbman et al.²⁹	WLS
Anistropic diffusion	Liu and Gong⁵⁰	AD
Smoothing sharpening filter	Deng et al.⁵¹	SSF
Iterative least square	Liu et al.⁴⁵	ILS
$L_{1}$ reconstruction filter	Yang et al.²⁶	L1RF
General smoothing filter	Liu et al.²⁸	GSF
Soft clustering filter	Liu et al.²⁵	SCF
Ours	-	Ours

Subjective evaluation

Procedure

For the subjective evaluation, we conduct a user study to validate the effectiveness of the proposed method. We invite a total of 36 doctors to participate in our user study. We randomly choose 20 X-ray images from the COVID-19 chest X-ray image dataset⁴⁸ and then enhance the detail of the images with our method. The doctors are required to observe and analyze 10 original X-ray images, and the other 10 detail-enhanced X-ray images, through a 32-inches monitor with 2K resolution.

After carefully observing all of the 20 images, the doctors are required to provide responses to a questionnaire, which includes the following questions:

I think the images are clear for diagnosis.

I think the images contain adequate details.

I am satisfied with the presented images.

We adopt a five-point Likert scale⁵² for the answers to the questions. The doctors need to choose responses from the following: strongly disagree (SD); disagree (D); neutral (N); agree (A); and strongly agree (SA). The ratings for the choices are 1, 2, 3, 4, and 5, respectively.

Grouping information

The 36 doctors are from the departments of respiratory, emergency, and geriatrics. There are 15 males and 21 females. They are randomly divided into two groups, namely the control and the treatment groups. The doctors in the control group are presented with the original X-ray images, and the doctors in the treatment group are presented with the detail-enhanced X-ray images derived from our method. We divide the 20 pairs of X-ray images into two image sets, that is, S1 and S2, with 10 pairs in each set. The control group observing image set S1 is the treatment group observing image set S2. The treatment group observing image set S2 is the control group observing image set S1. In this way, all subjects are exposed to both the original X-ray images and the detail-enhanced image. The detailed arrangement can be found in Table 2.

Table 2.

Grouping information.

Set	Group	Mode	No. of subjects $^{a}$
rowspan="2" align="*"S1	Control $^{b}$	X-ray images	18
	Treatment $^{c}$	Enhanced X-rays	18
rowspan="2" align="*"S2	Control $^{c}$	X-ray images	18
	Treatment $^{b}$	Enhanced X-rays	18

$^{a}$ The number of subjects.

$^{b}$ The control group observing image set S1 is the treatment group observing image set S2.

$^{c}$ The treatment group observing image set S1 is the control group observing image set S2.

Research hypotheses

Through the user study, we want to find out the answers to the following research questions:

What were the perceived clarity of the original X-ray images and the detail-enhanced X-ray images?

What were the perceived details of the original X-ray images and the detail-enhanced X-ray images?

What were the satisfaction levels provided by the original X-ray images and the detail-enhanced X-rays?

The corresponding research hypotheses are as follows:

H1.

The detail-enhanced X-ray images derived from our method are clearer than the original X-ray images.

H2.

The detail-enhanced X-ray images derived from our method contain more details than the original X-ray images.

H3.

The detail-enhanced X-ray images derived from our method improve the satisfaction levels over the original X-ray images.

Data collection and analysis

The doctors’ responses to the questionnaire are used to validate hypotheses H1, H2, and H3 in our user study. We printed the questions on paper so that it would be convenient for collecting the responses. The doctors are required to provide their responses by filling in their answers on the papers.

We record the number of subjects choosing each answer for each of the questions in the questionnaire. We leverage the chi-square test to study hypotheses H1, H2, and H3, as the measurements are discrete variables, and the purpose is to test whether the number of individuals in different categories fits a null hypothesis.

Experimental results

Objective evaluation

Quality

Table 3 shows the contrast measures of the original X-ray images and the enhanced images derived with various methods. The contrast measures are averaged over the entire testing dataset. It can be observed that enhancing the X-ray images with any one of the methods could significantly promote the contrast measures. However, among all the methods involved in this experiment, our method achieves the highest contrast for all the three categories of X-ray images. Therefore, our method could better improve the clarity and details in the X-ray images.

Table 3.

Contrast measures of the detail-enhanced X-ray images derived with various methods.

Method	COVID-19	Normal	Viral pneumonia
Input	75.54	32.66	55.34
FBF	408.97	320.34	496.48
GIF	326.63	275.17	393.82
DTF	384.84	310.56	471.71
MSF	401.63	395.47	479.14
WLS	346.86	274.67	409.78
AD	349.83	287.25	461.92
SSF	407.93	327.70	498.09
ILS	427.23	358.15	484.33
L1RF	389.88	312.62	459.09
GSF	395.00	313.63	475.17
SCF	410.09	403.34	531.38
Ours	442.30	426.40	564.51

FBF: fast bilateral filter; GIF: guided image gilter; DTF: domain transform filter; MSF: mutual structure filter; WLS: weighted least square; AD: anistropic diffusion; SSF: smoothing sharpening filter; ILS: iterative least square; L1RF: $L_{1}$ reconstruction filter; GSF: general smoothing filter; SCF: soft clustering filter.

The superiority of the proposed method can be also validated from the qualitative comparisons in Figure 2. We observe that enhancing the images with any method could improve the clarity, as any one of the enhanced images has a clearer appearance than the input X-ray image. However, the image derived with our method demonstrates even more details than that of fast bilateral filter (FBF), guided image gilter (GIF), domain transform filter (DTF), and iterative least square (ILS). Furthermore, our filter tends to better preserve the intensity of the original X-ray image compared to WLS, soft clustering filter (SCF), and $L_{1}$ reconstruction filter (L1RF) (see the black areas).

Figure 2.

Qualitative comparisons on medical image detail enhancement: (a) input, (b) FBF, (c) GIF, (d) DTF, (e) WLS, (f) ILS, (g) L1RF, (h) SCF, and (i) ours. FBF: fast bilateral filter; GIF: guided image gilter; DTF: domain transform filter; WLS: weighted least square; ILS: iterative least square; L1RF: $L_{1}$ reconstruction filter; SCF: soft clustering filter.

Efficiency

To further validate the practicality of our method, we compare its running time (measured in seconds) to that of other methods. Note that all the recorded time is averaged over 10 runs. As shown in Table 4, our method is highly efficient. It takes 0.92 seconds to process a 720P image, enabling interactive image processing. Through comparison, we can see that our method is faster than most of the methods involved in our experiments. DTF, FBF, GIF, and SCF achieve better efficiency than our method. However, our method delivers much better image quality, as demonstrated in the previous subsection. Finally, our method significantly improves the efficiency of WLS.

Table 4.

Running time of various methods under different resolutions (measured in seconds).

Method	$320 \times 240$	$640 \times 480$	$1280 \times 720$
FBF	0.04	0.25	0.85
GIF	0.05	0.26	0.87
DTF	0.01	0.06	0.15
MSF	0.30	1.34	4.35
WLS	0.45	1.92	6.03
AD	0.92	4.19	15.05
SSF	0.08	0.18	0.42
ILS	0.15	0.55	1.63
L1RF	0.07	0.32	1.08
GSF	2.46	9.65	31.51
SCF	0.06	0.27	0.88
Ours	0.06	0.30	0.92

Subjective evaluation

Image clarity

Figure 3 demonstrates the subjects’ responses to the question “I think the images are clear for diagnosis.” Their ratings from 1 to 5 are used to evaluate the clarity of the images. The results are analyzed with the chi-square test as shown in Table 5. We observe that the subjects in the treatment group (presented with the detail-enhanced images) hold a more positive attitude ( $3.53 > 2.72$ ) toward the clarity of the images than that of the control group (presented with the original X-ray images). Furthermore, we observe that the difference between the perceived image clarity of the two groups is significant ( $χ^{2} (4) = 9.81$ , $P = 0.044 < 0.05$ , $P$ is the probability of the difference being insignificant). The above findings support hypothesis H1, that is, our detail-enhanced X-ray images show better clarity than the original X-ray images.

Figure 3.

Subjects’ responses to the question “I think the images are clear for diagnosis.”

Table 5.

Chi-square test result for the perceived image clarity (degree of freedom = 4).

	Control	Treatment	Expect	$χ^{2} (4)$
SD (1)	8	2	5	3.6
D (2)	7	5	6	0.33
N (3)	12	8	10	0.8
A (4)	5	14	9.5	4.26
SA (5)	4	7	5.5	0.82
Total	36	36	36	9.81
Mean rating	2.72	3.53

SD: strongly disagree; D: disagree; N: neutral; A: agree; SA: strongly agree.

Image details

Figure 4 illustrates the subjects’ responses to the question “I think the images contain adequate details.” Their ratings from 1 to 5 are used to evaluate the details of the images. The results are analyzed with the chi-square test as shown in Table 6. It can be observed that the average ratings about the details of the image in the treatment group (presented with the detail-enhanced images) are higher ( $3.42 > 2.61$ ) than that of the control group (presented with the original X-ray images). Moreover, we observe that the difference between the perceived image details of the two groups is significant ( $χ^{2} (4) = 10.78$ , $P = 0.029 < 0.05$ ). The above findings support hypothesis H2, that is, our detail-enhanced X-ray images contain more adequate details than the original X-ray images.

Figure 4.

Subjects’ responses to the question “I think the images contain adequate details.”

Table 6.

Chi-square test result for the perceived image details (degree of freedom = 4).

	Control	Treatment	Expect	$χ^{2} (4)$
SD (1)	9	4	6.5	1.92
D (2)	7	5	6	0.33
N (3)	13	7	10	1.8
A (4)	3	12	7.5	5.4
SA (5)	4	8	6	1.33
Total	36	36	36	10.78
Mean rating	2.61	3.42

SD: strongly disagree; D: disagree; N: neutral; A: agree; SA: strongly agree.

Satisfaction level

The subjects’ responses to the question “I am satisfied with the presented images” are shown in Figure 5. The subjects’ ratings from 1 to 5 are used to evaluate the satisfaction level provided by the images. The results are studied with the chi-square test as demonstrated in Table 7. It can be observed that the satisfaction levels of the subjects in the treatment group (presented with the detail-enhanced images) are higher ( $3.5 > 2.56$ ) than that of the control group (presented with the original X-ray images). Furthermore, we observe that the difference between the satisfaction levels of the two groups is significant ( $χ^{2} (4) = 10.69$ , $P = 0.03 < 0.05$ ). The above findings support hypothesis H3, that is, the satisfaction levels provided with our detail-enhanced X-ray images are better than that provided with the original X-ray images.

Figure 5.

Subjects’ responses to the question “I am satisfied with the presented images.”

Table 7.

Chi-square test result for the perceived satisfaction levels (degree of freedom = 4).

	Control	Treatment	Expect	$χ^{2} (4)$
SD (1)	8	3	5.5	2.27
D (2)	11	4	7.5	3.27
N (3)	9	9	9	0
A (4)	5	12	8	2.88
SA (5)	3	8	5.5	2.27
Total	36	36	36	10.69
Mean rating	2.56	3.5

SD: strongly disagree; D: disagree; N: neutral; A: agree; SA: strongly agree.

Discussion

In this article, we propose a novel and efficient method to enhance the details of images and explore the COVID-19 chest X-ray image dataset⁴⁸ as an example to demonstrate the effectiveness of the proposed method. In fact, our method essentially defines a general framework for image detail enhancement, that is, besides the X-ray images, our method can be potentially applied to an even broader range of images, such as computed tomography images, magnetic resonance imaging images, or even natural images. More studies need to be conducted to evaluate our method in enhancing other types of images. Our method is implemented on the CPU, which processes 720P images at interactive rates. It would be beneficial to implement the proposed method on GPU so that it can achieve real-time enhancement for 720P images.

Due to the lack of ground truth labels, we leverage the contrast measure for objective evaluation of the proposed image detail enhancement method. The contrast measure is a good indicator of the details and clarity. However, it does not reflect artifacts or degradation generated in the enhancing process. More efforts need to be spent to build a dataset with ground truth labels for X-ray image detail enhancement. The ground truth labels can be obtained with more extensive user studies, where human experts are invited to select the best-looking images from a set of detail-enhanced images derived from a variety of methods. In this way, one can study the quality of the detail-enhanced images in a more objective sense, using measures such as peak noise-to-signal ratio and structure similarity.

The proposed method can be used to enhance the details of the X-ray images. It can be beneficial to down-streaming diagnosis tasks conducted by either machines or human experts. However, the tones of images are not involved in this article. The X-ray images suffer from low dynamic range. Therefore, the tone can be also important to the visibility of the X-ray images. More studies need to be done to consider the tones of the X-ray images in order to further improve the clarity. Possible directions can be the HDR tone mapping operator²⁰ and the tone manipulation method.⁵³ It would be interesting to investigate the proposed fast WLS filter in the tasks of tone mapping and manipulation to see how it can be applied to further improve the visibility of the X-ray images.

The proposed solution to the optimization model shown in equation (11) is highly efficient. However, it is based on the approximation with the bilateral grid, which is essentially a hard quantization technique. Therefore, the results of our filter may exhibit blocky artifacts. Even though the artifacts can be eliminated by post-processing with the domain transform filter, degradation in the quality of the results may be introduced. It would be also interesting to investigate soft quantization or clustering techniques to approximate or define the affinity matrix in the weighted scheme, so that the results can be free of blocky artifacts without post-processing. Possible ways to achieve this objective can be the approximation of the bilateral affinity with spatial subsampling²¹ or the soft clustering affinity.²⁵

Conclusion and future work

X-ray images can be essential to the diagnosis of COVID-19, which has brought significant impacts on our lives. However, the original X-ray images may suffer from low dynamic range and low contrast, which would hinder the diagnosis by either human experts or machine learning methods. Aiming at improving the clarity of the X-ray images, we propose an efficient method to enhance the details of the X-ray images. The proposed enhanced method is based on image decomposition with the bilateral WLS model. In order to efficiently solve the model, we propose to approximate the bilateral weight matrix with the bilateral grid so that the large sparse linear system would be simplified into a linear system with much fewer unknowns.

We have conducted extensive experiments to evaluate the proposed X-ray image detail enhancement method. Both objective and subjective evaluations are leveraged in our experiment. For the objective evaluation, we analyze the contrasts of the enhanced images derived with different methods. It is shown that our method could achieve higher contrast than the methods being compared. Furthermore, we compare the running time of different methods in deriving the enhanced images. Our method can process 720P 3-channel images at interactive rates on an i7-6700 CPU. For the subjective evaluation, we conduct a user study that involves 36 doctors. The chi-square test results indicate that the ratings of our detail-enhanced X-ray images are significantly higher than that of the original X-ray images. Therefore, we draw that our method could significantly improve the visibility of the X-ray images. Moreover, it can be applied to real practice, as it is efficient and effective.

Based on the discussion in the previous section, we summarize the future directions of research as follows:

We only apply our detail enhancement method in X-ray images, more studies need to be conducted to evaluate our method on a broader range of images.

In order for a more objective evaluation, more efforts need to be spent to build a dataset with ground truth labels for X-ray image detail enhancement.

The proposed method only deals with the details, more studies need to be carried out to investigate the tone manipulation techniques^20,53 to further improve the clarity.

Our fast solution is based on a hard quantization technique, that is, the bilateral grid, more studies need to be conducted to explore affinity designs with soft quantization²¹ or clustering.²⁵

Footnotes

Acknowledgements

We would like to thank all participants for their time and efforts in our user study.

Contributorship

Conceptualization: YY and WB; data curation: WB and YY; methodology: WB and YY; formal analysis: WB and YY; supervision: YY; validation: YY; writing–original draft: WB; writing–review and editing: YY.

Consent statement

Patient consent was not required for the present manuscript, as the COVID-19 image dataset in this study is publicly available.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Ethical approval

Not applicable.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was funded by National Natural Science Foundation of China (Ref: 61402205), China Postdoctoral Science Foundation (Ref: 2015M571688), Jiangsu University (Ref: 13JDG085), and Jiangsu Province Elderly Health Project (Ref: LD2021039).

Guarantor YY.

ORCID iD

Yang Yang

References

Alimohamadi

Taghdir

Sepandi

. The estimate of the basic reproduction number for novel coronavirus disease (Covid-19): a systematic review and meta-analysis. J Prev Med Public Health 2020; 53: 151–157.

Elsaid

Ayman

Nguyen

. R0 of Covid-19 and its impact on vaccination coverage: compared with previous outbreaks. Hum Vaccin Immunother 2021; 17: 1–5.

Prada

Maag

Siegmund

et al. Estimation of r0 for the spread of SARS-CoV-2 in germany from excess mortality. Sci Rep 2022; 12: 17221.

Chinazzi

Davis

Ajelli

et al. The effect of travel restrictions on the spread of the 2019 novel coronavirus (Covid-19) outbreak. Science 2020; 368: 395–400.

Tenforde

Olson

Self

et al. Effectiveness of Pfizer-biontech and moderna vaccines against Covid-19 among hospitalized adults aged greater or equal to 65 years – United States, January–March 2021. MMWR Morb Mortal Wkly Rep 2021; 70: 674–679.

Antonelli

Pujol

Spector

et al. Risk of long Covid associated with delta versus omicron variants of sars-cov-2. Lancet 2022; 399: 2263–2264.

Notarte

Catahay

Velasco

et al. Impact of Covid-19 vaccination on the risk of developing long-Covid and on existing long-Covid symptoms: a systematic review. EClinicalMedicine 2022; 53: 101624.

Organization

. Tracking SARS-CoV-2 variants. https://www.who.int/activities/tracking-SARS-CoV-2-variants/. Accessed March 25, 2023.

Organization

. Summary of variant surveillance. https://covid.cdc.gov/covid-data-tracker/variant-summary. Accessed March 25, 2023.

10.

Wang

Andraweera

Elliott

et al. Asymptomatic SARS-CoV-2 infection by age: a global systematic review and meta-analysis. Pediatr Infect Dis J 2023; 42: 232–239.

11.

Kronbichler

Kresse

Yoon

et al. Asymptomatic patients as a source of Covid-19 infections: a systematic review and meta-analysis. Int J Infect Dis 2020; 180–186.

12.

Habas

Said

Ramzee Abdul Aziz

et al. Chest X-ray findings and hyponatremia in Covid-19 pneumonia patients. Qatar Med J 2022; 2022: 34.

13.

Cozzi

Albanesi

Cavigli

et al. Chest x-ray in new coronavirus disease 2019 (Covid-19) infection: findings and correlation with clinical outcome. Radiol Med 2020; 125: 730–737.

14.

Nayak

Sinha

et al. Application of deep learning techniques for detection of Covid-19 cases using chest X-ray images: a comprehensive study. Biomed Signal Process Control 2021; 64: 102365.

15.

Zhou

Liu

et al. A review of deep learning imaging diagnostic methods for Covid-19. Electronics 2023; 12: 1167.

16.

Öktem

Egiazarian

Niittylahti

et al. An approach to adaptive enhancement of diagnostic X-ray images. EURASIP J Adv Signal Process 2003; 2003: 430–436.

17.

Qiu

Zhang

et al. Automatic X-ray image contrast enhancement based on parameter auto-optimization. J Appl Clin Med Phys 2017; 18: 218–223.

18.

Zhu

et al. Multiscale X-ray image contrast enhancement based on limited adaptive histogram equalization. In International Conference on Internet Multimedia Computing and Service, 2013, pp. 231–236. DOI:10.1145/2499788.2499793.

19.

Koonsanit

Thongvigitmanee

Pongnapang

et al. Image enhancement on digital X-ray images using n-clahe. In Biomedical Engineering International Conference, 2017, pp. 1–4. DOI:10.1109/BMEiCON.2017.8229130.

20.

Durand

Dorsey

. Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans Graph 2002; 21: 257–266.

21.

Yang

Xiong

Cao

et al. Fast bilateral filter with spatial subsampling. Multim Syst 2022; 29: 435–446.

22.

Septiana

Lin

. X-ray image enhancement using a modified anisotropic diffusion. In International Symposium on Bioelectronics and Bioinformatics, 2014, pp. 1–4. DOI:10.1109/ISBB.2014.6820940.

23.

Sun

Tang

. Guided image filtering. IEEE Trans Pattern Anal Mach Intell 2013; 35: 1397–1409.

24.

Ochotorena

Yamashita

. Anisotropic guided filtering. IEEE Trans Image Process 2020; 29: 1397–1412.

25.

Yang

Hui

Zeng

et al. Edge-preserving image filtering based on soft clustering. IEEE Trans Circuits Syst Video Technol 2022; 32: 4150–4162.

26.

Yang

Zheng

Zeng

et al. L1-regularized reconstruction model for edge-preserving filtering. IEEE Trans Multimedia 2022. DOI: 10.1109/TMM.2022.3171686.

27.

Yang

Tang

Zeng

et al. L0 image smoothing via iterating truncated L1 gradient regularization. J Electron Imaging 2022; 31: 053016.

28.

Liu

Zhang

Lei

et al. A generalized framework for edge-preserving and structure-preserving image smoothing. IEEE Trans Pattern Anal Mach Intell 2022; 44: 6631–6648.

29.

Farbman

Fattal

Lischinski

et al. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans Graph 2008; 27: 67.

30.

Liao

Lan

Fan

et al. SIRVD-DL: a COVID-19 deep learning prediction model based on time-dependent SIRVD. Comput Biol Medicine 2021; 138: 104868.

31.

Liu

Ding

Luo

et al. Cx22: a new publicly available dataset for deep learning-based segmentation of cervical cytology images. Comput Biol Medicine 2022; 150: 106194.

32.

Han

et al. Adaptive soft erasure with edge self-attention for weakly supervised semantic segmentation: thyroid ultrasound image case study. Comput Biol Medicine 2022; 144: 105347.

33.

Chen

Gan

Chen

et al. Accurate iris segmentation and recognition using an end-to-end unified framework based on madnet and dsanet. Neurocomputing 2023; 517: 264–278.

34.

Zhao

Wang

Chen

et al. Jamsnet: a remote pulse extraction network based on joint attention and multi-scale fusion. IEEE Trans Circuits Syst Video Technol 2023; 33: 2783–2797.

35.

Din

NMU

Dar

Rasool

et al. Breast cancer detection using deep learning: datasets, methods, and challenges ahead. Comput Biol Medicine 2022; 149: 106073.

36.

Tomasi

Manduchi

. Bilateral filtering for gray and color images. In International Conference on Computer Vision, 1998, pp. 839–846. DOI:10.1109/ICCV.1998.710815.

37.

Gastal

ESL

Oliveira

. Adaptive manifolds for real-time high-dimensional filtering. ACM Trans Graph 2012; 31: 33:1–33:13.

38.

Paris

Durand

. A fast approximation of the bilateral filter using a signal processing approach. Int J Comput Vis 2009; 81: 24–52.

39.

Nair

Chaudhury

. Fast high-dimensional bilateral and nonlocal means filtering. IEEE Trans Image Process 2019; 28: 1470–1481.

40.

Nair

Gavaskar

Chaudhury

. Compressive adaptive bilateral filtering. In International Conference on Acoustics, Speech and Signal Processing, 2020, pp. 2078–2082. DOI:10.1109/ICASSP40776.2020.9053275.

41.

Sun

Han

et al. Weighted guided image filtering with steering kernel. IEEE Trans Image Process 2020; 29: 500–508.

42.

Guo

Feng

et al. Soft clustering guided image smoothing. In International Conference on Multimedia and Expo, 2018, pp. 1–6. DOI:10.1109/ICME.2018.8486448.

43.

Min

Choi

et al. Fast global image smoothing based on weighted least squares. IEEE Trans Image Process 2014; 23: 5638–5653.

44.

Liu

Chen

Shen

et al. Semi-global weighted least squares in image filtering. In International Conference on Computer Vision, 2017, pp. 5862–5870. DOI:10.1109/ICCV.2017.624.

45.

Liu

Zhang

Huang

et al. Real-time image smoothing via iterative least squares. ACM Trans Graph 2020; 39: 28:1–28:24.

46.

Shewchuk

. An introduction to the conjugate gradient method without the agonizing pain. Technical report, USA, 1994. DOI:10.5555/865018.

47.

Gastal

ESL

Oliveira

. Domain transform for edge-aware image and video processing. ACM Trans Graph 2011; 30: 69.

48.

RAIKOTE

. Covid-19 image dataset. https://www.kaggle.com/datasets/pranavraikokte/covid19-image-dataset, 2023.

49.

Shen

Zhou

et al. Mutual-structure for joint filtering. Int J Comput Vis 2017; 125: 19–33.

50.

Liu

Gong

. Guided depth enhancement via anisotropic diffusion. In Pacific-Rim Conference on Multimedia, 2013, pp. 408–417. DOI:10.1007/978-3-319-03731-8_38.

51.

Deng

Galetto

Al-nasrawi

et al. A guided edge-aware smoothing-sharpening filter based on patch interpolation model and generalized gamma distribution. IEEE Open J Signal Process 2021; 2: 119–135.

52.

Likert

. A technique for the measurement of attitudes. Arch Psychol 1932; 140: 1–55.

53.

Guo

Ling

. LIME: low-light image enhancement via illumination map estimation. IEEE Trans Image Process 2017; 26: 982–993.

Fast bilateral weighted least square for the detail enhancement of COVID-19 chest X-rays

Abstract

Background

Method

Results

Conclusion

Keywords

Introduction

Literature review

Local methods

Global methods

Proposed method

Bilateral WLS model

Efficient solver

System model

Experimental settings

Objective evaluation

Metrics

Dataset

Methods

Subjective evaluation

Procedure

Grouping information

Research hypotheses

Data collection and analysis

Experimental results

Objective evaluation

Quality

Efficiency

Subjective evaluation

Image clarity

Image details

Satisfaction level

Discussion

Conclusion and future work

Footnotes

Acknowledgements

Contributorship

Consent statement

Declaration of conflicting interests

Ethical approval

Funding

ORCID iD

References