Sage Journals: Discover world-class research

Abstract

Timely detection of building damage after a disaster can provide support and help in saving lives and reducing losses. The emergence of transfer learning can solve the problem of difficulty in obtaining several labeled samples to train deep models. However, some degree of differences exists among different scenarios, which may affect the transfer performance. Furthermore, in reality, data can be collected from multiple historical scenarios but cannot be directly combined using single-source domain adaptation methods. Therefore, this study proposes a multi-source variational domain adaptation (MVDA) method to complete the task of post-disaster building assessment. The MVDA method consists of two stages: first, the distributions of each pair of source and target domains in specific feature spaces are aligned separately; second, the outputs of the pre-trained classifiers are aligned using domain-specific decision boundaries. This method maximizes the relevant information in the historical scene, solves the problem of inconsistent image classification in the current scene, and improves the migration efficiency from the history to the current disaster scene. The proposed approach is validated by two challenging multi-source transfer tasks using the post-disaster hurricane datasets. The average accuracy rate of 83.3% for the two tasks is achieved, obtaining an improvement of 0.9% compared with the state-of-the-art methods.

Keywords

Building damage detection domain adaptation multi-source domain transfer learning remote sensing

1 Introduction

Since the 21st century, natural disasters have claimed many lives and caused damage to many people worldwide [1]. The large damage caused by disasters is mainly manifested in the damage to buildings. The relevant information on buildings can evaluate the location and quantity distribution of the affected population and can guide rescuers to where they need to be in a short time [2]. The traditional damage assessment of buildings requires professional field surveys [3]. This process is time-consuming and labor-intensive, which is not desirable. However, the remote sensing data obtained by electronic equipment, such as radar, are very useful. Electronic equipment can obtain accurate images of building damage in a short time and has the characteristics of low cost and wide field of view [4]. Hence, the automatic identification of damaged buildings based on remote sensing data is the current mainstream method [5, 6].

Owing to its powerful feature learning ability, deep learning has gradually become a research hotspot in remote sensing image classification [7]. In addition, the detection of building damage after a disaster has gradually transitioned from handcrafted features combined with shallow machine learning methods to those based on convolutional neural networks (CNN) methods. Cao et al. [8] used a Lenet-5-inspired CNN to classify the images of post-hurricane buildings extracted from satellite images into two categories, damaged or undamaged. Ji et al. [9] optimized the data balancing strategy, using CNN to improve the overall accuracy of detecting collapsed buildings from satellite images. Duarte et al. [10] also improved the overall satellite image classification accuracy of damaged buildings using convolutional residual connections with extended receptive fields. However, CNN-based deep learning models have complex structures and large-scale parameters and require several labeled samples to ensure good performance [11]. However, in the post-disaster detection of damaged buildings, the image content is complex, and the buildings are in various forms. Generating several labeled samples containing various situations only from the current post-disaster images is difficult and cannot meet the training requirements [12].

The emergence of transfer learning can overcome the above difficulties. Researchers have made preliminary explorations and attempts by using post-disaster images of historical scenes to make up for the shortage of training samples during current disasters. Although these methods [10 , 14] have made some progress, they fail to consider the differences among different post-disaster scenarios, greatly affecting the transfer efficiency. This study solves this problem by using a domain adaptation method to reduce the differences among domains. In addition, many historical post-disaster scenarios data are available [15], which can help transfer learning. Multi-source processing is a common and straightforward approach to merge all-source domains into one source domain and align the distributions, such as the traditional single-source domain adaptation method. However, owing to data expansion, these methods do not improve performance significantly [16]. Therefore, finding a great way to fully utilize multiple source domains is necessary.

This study proposes a multi-source variational domain adaptation (MVDA) method to address the above problems. The first stage is to align domain-specific distributions, map each pair of source and target domains to different feature spaces, and align domain-specific distributions to learn multiple domain-invariant representations. Then, domain-specific classifiers are trained using multiple domain-invariant representations. The second stage is to align domain-specific classifiers. The target samples near the domain-specific decision boundary predicted by different classifiers may get different labels [17]. Therefore, the classifier outputs of the target samples are aligned with domain-specific decision boundaries.

The research contributions of this study are summarized as follows: (1) An unsupervised MVDA method is proposed. This method utilizes a Variational AutoEncoder (VAE) to model the target domain and aligns the feature distributions of the target domain and each source domain separately. In addition, the method exploits a Gaussian mixture model and the Bayesian formula to obtain the decision boundary of the target domain samples and weight the output of each source domain classifier to obtain the prediction of the target domain samples. (2) The MVDA method is applied to post-disaster building damage detection, and relevant experiments are carried out on the hurricane data set to verify the effectiveness of the model.

2 Related work

Building damage detection: The purpose of building damage detection is to evaluate the damage degree of the buildings in the area after the disaster, provide reference and help for the follow-up rescue work, and provide data support for the reconstruction of the disaster area. Owing to the rapid development of deep learning in recent years, deep learning using optical remote sensing images has become an extensive research topic to solve post-disaster building damage detection. Wheeler et al. [18] detected the damage degree of different buildings by evaluating mainstream CNNs and using multi-temporal satellite imagery. Ghaffarian et al. [19] proposed an efficient method to update the building database by training a CNN using pre-disaster satellite images and building footprints, and fine-tuning using post-disaster images. Li et al. [20] adopted a single-shot multi-box detector with pre-trained and data augmentation for damaged building evaluation. Abdollahi et al. [21] used an end-to-end CNN with Generative Adversarial Networks (GAN) architecture to extract building footprints from high-resolution aerial images. Kim et al. [22] proposed a CNN structure to detect building areas in large images and classify the roof material in the detected area to obtain the damage situation. However, CNN-based deep learning models have complex structures and large-scale parameters and require several labeled samples to ensure good performance; otherwise, model overfitting may occur. However, in the post-disaster detection of damaged buildings, the image content is complex, and the building forms are diverse. Generating several labeled samples, including various situations, only by the current post-disaster single-phase image is difficult. Moreover, the number of samples cannot meet the training requirements of the deep learning model. Therefore, the recognition accuracy is greatly affected.

Domain adaptation: Domain adaptation is a research branch of transfer learning [23, 24]. It mainly solves the problem of migration when the feature distribution of the training set as the source domain and the test set as the target domain are different, but the tasks are the same. Deep network structure can obtain deep semantic information about samples, which can help the network complete its tasks. The deep learning method applied in domain adaptation problems is called deep domain adaptation, and its central idea is to use deep neural networks to align the data distribution between the source and target domains. Compared with traditional methods, the features obtained by deep domain adaptation methods not only have stronger generalization ability but also better transferability. Most domain adaptation methods employ a joint architecture with two branches, jointly training a traditional task loss based on labeled source data and another loss that measures the difference between the source and target domains, including disparity, adversarial, and reconstruction loss, thereby reducing inter-domain differences. Disparity loss-based methods explicitly measure the difference between two domains, such as Deep Adaptation Network (DAN) using Maximum Mean Discrepancy (MMD) and Correlation Alignment [25]. The adversarial loss comes from the adversarial generative model or the discriminative model. The former combines the domain discriminator and the generative component to generate false source samples based on GAN and its variants, such as Cycle-Consistent Adversarial Networks [26] and Cycle-Consistent Adversarial Domain Adaptation [27]. Meanwhile, the latter typically employs adversarial objectives against the domain discriminator to encourage domain confusion, such as Domain-Adversarial Neural Network [28] and Adversarial Discriminative Domain Adaptation (ADDA) [29]. In addition to the above methods, Deep Reconstruction-Classification Networks (DRCN) [30] converts the extracted features back to the image and calculates the reconstruction loss between it and the original image, ensuring that the two maintain a high similarity.

Traditional domain adaptation methods mainly focus on a single-source scenario, assuming that the labeled source data come from the same distribution. In practice, however, labeled data can be collected from multiple source domains with different distributions. However, as varying degrees of inter-domain differences also exist, the above domain adaptation method cannot be used simply by combining all-source domains into a single-source domain. Early multi-source domain adaptation (MDA) methods mainly focus on shallow models [31], learn latent feature spaces from different domains [32], or combine pre-trained source classifiers [33]. More recently, the focus of MDA has shifted to deep learning architectures. Mansour et al. [34] assumed that the target distribution can be approximated by mixing source distributions. Hence, weighted combinations of source classifiers are widely used for MDA. Furthermore, tight cross-domain generalization bounds and precise domain dissimilarity measures can provide an intuitive basis for deriving efficient MDA algorithms. Reference [35] derived a new bound using DC programming and computed precise combined weights. Reference [36] extended the generalization of the pioneering theoretical model to multiple sources in the classification and regression setting. In addition to the domain difference between the target and each source, Li et al. [37] also considered the relationship between paired sources and derived a tight bound on the weighted multi-source difference, based on which one can pick additional relevant source domains. The above works focused on extracting common domain invariant representations for all domains; however, this learning style is difficult.

Training a deep model with good performance is difficult owing to the small number of labeled samples after the disaster. Hence, the idea of transfer learning can be used to solve the problem, that is, using the post-disaster images taken in the history to train the model and then perform the classification task under the current disaster to complete the damage to the building assessment of the situation. For example, Andugula et al. [13] used pre-disaster images to pre-train the classifier and then used a small number of post-disaster labeled samples to fine-tune the classifier to improve the classification accuracy. However, such a migration strategy does not consider the differences among different scenarios and does not take measures to reduce the differences, resulting in unsatisfactory classification accuracy. The above-mentioned domain adaptation method brings inspiration to solve this problem and can be used as an effective means to reduce the differences among various fields. Furthermore, traditional domain adaptation methods mainly focus on single-source scenarios, assuming that the labeled source data come from the same distribution. However, in practical post-disaster scenarios, labeled data can be collected from multiple source domains with different distributions. Simply combining multiple source domains into a single-source domain and directly using a domain adaptation method with a single-source domain may lead to suboptimal solutions because data from different sources interfere with one another during learning. Therefore, this study chooses the extended MDA method and improves its current shortcomings. The study proposes the MVDA algorithm to finally complete the post-disaster building damage detection. The method is divided into two stages. First, each pair of source and target domains are mapped to different feature spaces to extract multiple domain-invariant representations; then, the output of each source classifier for the target sample is analyzed by using the decision boundary of a specific domain.

3 Methodology

In the multi-source unsupervised domain adaptation method, N different source distributions are included, denoted as ${p_{sk} (x, y)}_{k = 1}^{N}$ , and each labeled source data ${(X_{sk}, Y_{sk})}_{k = 1}^{N}$ are sampled from these distributions, where $X_{sk} = {x_{i}^{sk}}_{i = 1}^{| X_{sk} |}$ represents the data from the source domain k, and $Y_{sk} = {y_{i}^{sk}}_{i = 1}^{| X_{sk} |}$ is the corresponding real label. Furthermore, the method has a target distribution p_t (x, y), from which the target domain data $X_{t} = {x_{i}^{t}}_{i = 1}^{| X_{t} |}$ are sampled but have no label Y_t. Domain-invariant representations are learned in the common feature space of all domains by minimizing the distance loss between each pair of source and target domains, without considering domain-specific decision boundaries between classes, resulting in pairs of different source classifiers close to each other. The label results predicted by the target samples of the specific domain decision boundary are inconsistent, which affects the classification accuracy.

The MVDA algorithm proposed in this study makes up for the shortcomings of the above MDA methods. This algorithm extracts domain-invariant representations for each pair of source and target domains. Then, the MVDA algorithm uses the decision boundary of a specific domain to align the label predictions of the source classifiers for the target samples. Specifically, this algorithm consists of two stages, and Fig. 1 shows its frame structure.

Fig. 1

MVDA algorithm architecture. In the first stage, encoder E is constantly affected by classification loss, maximum mean difference loss, and reconstruction loss during the training process. Ultimately, the source domain feature Z_s extracted by encoder E and the mean Z_t _ mean and variance Z_t _ log _ var output that can represent the distribution of target domain features will become increasingly accurate. In the second stage, the results of two classifiers are combined with the weight p (c_S1,S2|z_t) to output the final label.

As shown in Fig. 1, E₁ and E₂ are encoders for source domains 1 and 2, respectively, to extract deep features of the image. Z_s1 and Z_s2 are the features extracted by the encoder. C₁ and C₂ are classifiers for two source domains, respectively, to classify images in source domains 1 and 2. DE₁ and DE₂ are decoders for two source domains used to reconstruct images with different source domain styles for target domain samples. $X_{t}^{'}$ is the reconstructed images from the decoder.

The first stage mainly aligns the domain-specific distributions of each pair of source and target domains. Usually, multiple domain-invariant representations for each pair of source and target domains are extracted by mapping each of them to a specific feature space and matching its distribution. To achieve this, weights are not shared between these networks. Furthermore, we use a VAE to reconstruct target domain features to learn domain-invariant representations to align each pair of source and target domains, which can also provide more useful information for cross-domain target recognition.

Each source learns a domain-specific classifier. The output of domain-specific classifiers is aligned in the second stage to ensure consistent label information predicted by different classifiers for target samples close to the domain-specific decision boundary. However, most of the current methods are highly complicated for this step. In this study, the probability distribution of the target sample is calculated using the Bayesian formula to determine the weight applied to the classifier, which effectively solves the above problem.

3.1 Domain-specific distribution alignment

In this study, a common network E_k (·) is used to extract common representations for all domains, mapping images from the original feature space to a common feature space. Furthermore, each pair of source and target domain data needs to be mapped into their respective common feature spaces. Given batch image x^sk from the source domain (X_sk, Y_sk) and batch image x^t from target domain X_t, source domain features E_k (x^sk) and target domain features E_k (x^t) are extracted through N non-shared domain-specific networks.

For the processing of target domain data, this study introduces a VAE model. We consider E_k (·) as an inference network, infer the variational inference of the target domain data x^t, and then generate the variational probability distribution q_φ (z^t|x^t) of the latent variable Z^t. Here, q_φ (z^t|x^t) satisfies the Gaussian distribution, so the mean Z_t _ mean and variance z_t _ log _ var of the Gaussian distribution are the real output of the E_k (·) network. Through the re-parameterization technique, formula (3-1) is used to obtain the hidden variable z_t, where ɛ is the standard Gaussian distribution. $z_{t} = \exp (z_{t_var}) \times ɛ + z_{t_mean} .$ (3-1)

The generative network DE_k (·) takes the latent variable Z^t as the input, and the output is the probability distribution p_θ (x^t|z^t) of the target data. The generated data are similar to the input data only when q_φ (z^t|x^t) is close to the real posterior probability p_θ (z^t|x^t), that is, the reconstruction loss $L_{rec}$ of the target data x^t is the smallest. Specifically, it is calculated by formula (3-2): $\begin{matrix} L_{rec} = & | | X_{t} - {DE}_{1} (E_{1} (X_{t})) | |_{2}^{2} \\ + | | X_{t} - {DE}_{2} (E_{2} (X_{t})) | |_{2}^{2} . \end{matrix}$ (3-2)

This study also chooses MMD as the criterion to measure the distance between two domains to reduce the inter-domain distribution differences to achieve the alignment of domain-specific distributions in the first stage. MMD is a two-sample test that rejects or accepts the null hypothesis p = q based on the observed sample. The basic idea is that if the generating distributions are the same, then all statistics are the same. Formally, MMD is defined as follows: $D_{H} (p, q) = | | E_{P} [Φ (x^{s})] - E_{q} [Φ (x^{t})] | |_{H}^{2},$ (3-3) where H is the regenerated kernel Hilbert space (RKHS) with characteristic kernel k. The probability distributions of p and q agree if and only if D_H (p, q) = 0. Therefore, the estimation of the difference between each pair of source and target domains in this study can be expressed as follows: $\begin{matrix} L_{mmd} = & D_{H} (E_{1} (X_{S 1}), E_{1} (X_{t})) \\ + D_{H} (E_{2} (X_{S 2}), E_{2} (X_{t})) . \end{matrix}$ (3-4)

A domain-invariant representation for each pair of source and target can be learned by minimizing $L_{mmd}$ and $L_{rec}$ , enabling the alignment of domain-specific distributions in the first stage.

Algorithm 1. Summary of the whole process of training the MVDA model.

Algorithm 1 MVDA algorithm framework
Stage 1:
enter: Source domain data X_S1, X_S2, source domain labels Y_S1, Y_S2, target domain data X_t;
Encoder E₁, E₂, Decoder DE₁, DE₂, Classifier C₁, C₂.
1: for t in 100 do:
2: Update the parameters of E₁, DE₁, C₁ by formula (3-8);
3: Update the parameters of E₂, DE₂, C₂ by formula (3-8);
4: end for
output: Trained E₁, E₂ and C₁, C₂, and, z_{t_mean}, z_{t_var}
Stage 2:
enter: source domain data X_S1, X_S2, source domain label Y_S1, Y_S2, target domain data z_{t_mean}, z_{t_var}, GMM, trained E₁, E₂, C₁, C₂.
1: fort in (1, max iterations) do:
2: Extract π_s₁, s₂, μ_s₁, s₂, and Σ_s₁, s₂ by GMM;
3: Update p (c_S1\|z_t) and p (c_S2\|z_t) by Equations (3–6);
4: Calculate P by Equations (3–7) and output the target label;
5: end for
Output: MVDA model

3.2 Classifier alignment

The classifier C consists of N source domain predictors ${C_{k}}_{k = 1}^{N}$ , where each C_k is a Softmax classifier that receives domain-specific features after the feature extractor E_k (·) of the k_th source domain. For each classifier, a classification loss is added using cross-entropy: $\begin{matrix} L_{cls} = & E_{x_{s 1} \sim X_{S 1}} J (C_{1} (E_{1} (x_{s 1}), y_{s 1}) \\ + E_{x_{s 2} \sim X_{S 2}} J (C_{2} (E_{2} (X_{s 2}), y_{s 2})) . \end{matrix}$ (3-5)

The source domain data can be accurately classified by minimizing $L_{cls}$ .

In the second stage of the proposed MVDA algorithm, the target samples near the class boundary are easily misclassified by the classifier learned from the source samples. Classifiers are trained on different source domains. Thus, their predictions on target samples, particularly those near class boundaries, may diverge. In theory, the same target samples predicted by different classifiers should obtain consistent results. Therefore, the goal of the second alignment stage is to minimize the differences among all classifiers, as shown in Fig. 2.

Fig. 2

Schematic diagram of the classifier alignment stage. The final prediction result is output by combining the predicted results of two classifiers C₁ and C₂ with the weights obtained from GMM.

As shown in Fig. 2, Z_S1 and Z_S2 are the output results of E₁ and E₂ extracted from the source domain in Fig. 1. After modeling with a Gaussian mixture model, weights π_s₁, s₂, means μ_s₁, s₂, and variances rmSigma_s₁, s₂ can be obtained. The mean z_{t_mean} and variance z_{t_var} are obtained by extracting features from the target domain using encoders 1 and 2 trained in the first stage. We use Gaussian mixture models (GMM) for the two source domains, combined with the mean z_{t_mean} and variance z_{t_var} of the target domain features extracted by encoders 1 and 2, according to the Bayesian formula (3-6) to calculate the probability distribution that the target domain samples belong to a certain source domain:

$p (c | z) = \frac{p (c) p (z | c)}{\sum_{t}^{K + 1} p (t) p (z | t)} .$ (3-6)

After p (c_S1|z_t) and p (c_S2|z_t) are obtained, they are respectively applied as weights to the classification results of the two classifiers to determine the final category of the target domain samples. The probability value P of the final category is obtained from formula (3-7), where C₁ and P₂ are the probability values predicted by classifiers C₁ and C₂ for the target domain samples. $P = p (c_{S 1} | z_{t}) P_{1} + p (c_{S 2} | z_{t}) P_{2} .$ (3-7)

The loss function of our proposed algorithm consists of three parts: classification loss $L_{cls}$ , reconstruction loss $L_{rec}$ , and MMD loss $L_{mmd}$ . The total loss is expressed as follows: $L = L_{cls} + L_{rec} + L_{mmd} .$ (3-8)

4 Experiments

4.1 Datasets

The post-disaster building damage assessment includes two stages: image segmentation and image classification. In the first stage, the original image is segmented to generate training and test samples. In the second stage, the MVDA model is trained using the training samples, and then, the test samples are divided into different classes. This section uses the hurricane Sandy, Irma, and Maria datasets to verify the classification performance of the proposed method. In 2012, Hurricane Sandy [38] caused varying degrees of damage to buildings in most parts of the United States. Shortly after Hurricane Sandy, a team from Drexel University used an aerial camera to capture 10,000 RGB images of the New Jersey coastline at their original size of 1920×1080 pixels. In September 2017, when Hurricanes Irma [39] and Maria [40] hit Florida and Puerto Rico, respectively, NOAA collected post-disaster aerial images with an original size of 18681×18681 pixels. The Sandy, Irma, and Maria dataset images were converted into 200×200 pixel image patches using a Simple Linear Iterative Clustering (SLIC) [41] algorithm. In the following experiments, two experimental tasks were established using different combinations of three datasets: (1) using datasets Sandy and Irma as the source domain and dataset Maria as the target domain; (2) using dataset Sandy and Maria as the source domain, using the dataset Irma as the target domain.

Each dataset contains 10,000 initial aerial images, which are processed into 200×200 pixel blocks using the SLIC algorithm. Each dataset includes the same three categories: undamaged buildings, damaged buildings, and other objects. The undamaged category represents buildings that have not changed in appearance after the disaster. The damaged category includes buildings that are completely or partially damaged. The last category is the images that do not include buildings, most of which are roads, trees, or other objects. Figure 3 shows parts of the samples from the three datasets.

Fig. 3

Sample examples from three datasets. (a) undamaged, (b) damaged, (c) other objects.

4.2 Experimental preparation and implementation details

In this study, all experiments are conducted using Python 0.4.1 on the Windows system, and the graphics card is NVIDIA GTX 1080.

Our goal is to train a classifier that can accurately predict the target domain only through image data from the source domain. Therefore, we use all labeled source domain data and all unlabeled target domain data. Our task, network architecture, and parameters for training are summarized below.

For all tasks, we use ResNet-50 [47] as the backbone network. The domain adaptation process consists of two stages. In stage 1, the training encoder, classifier, and decoder output the mean z_{t_mean} and the variance z_{t_log_var} using Formula (3–7) to classify target domain images in stage 2.

Stage 1: The structure of the two lines in source domains 1 and 2, as shown in Fig. 1, is the same, but they do not share weights. Train encoders can extract common features from source and target domains, classifiers can accurately classify source domain features, and decoders can use target domain features to reconstruct target domain images. The encoders all use the original Resnet-50 structure as the backbone network, which is used to extract image features. After the Resnet-50 structure in the encoder, two linear structures do not change the size of the dimensions. One linear is used to output the mean representing the probability distribution of target domain features, and the other linear is used to output the variance representing the probability distribution of target domain features. Figure 4 shows a classifier for three classification tasks and a decoder after the encoder structure. Among them, the classifier classifies the source domain image based on the source domain features extracted from the backbone network. The decoder uses the target domain image features extracted by the backbone network to reconstruct the target domain. The entire model uses the Adam optimizer. Although source domains 1 and 2 do not share weights, the parameter settings of the optimizer are the same. The learning rate is 0.0001, and in the optimizer, the value of β1 is 0.5, the value of β2 is 0.999, and the batch size is 8.

Fig. 4

Network structure of decoders 1 and 2.

Stage 2: The images of the three categories (i.e., undamaged, damaged, and others) of the source domain dataset are mixed, and the two source domains are regarded as different categories. Then, we calculate the class probability p (c_S1|z_t) and p (c_S2|z_t) of the target domain feature belonging to the two source domains according to the Bayesian formula (3–7) to determine the weight of each classifier.

In the following experiments, two experimental tasks were established using different combinations of three datasets: (1) using dataset Sandy and Irma as the source domain and dataset Maria as the target domain; (2) using dataset Sandy and Maria as the source domain, using the datasets Irma as the target domain.

4.3 Benchmark algorithms

Only a small amount of MDA work exists in post-disaster building damage detection. Five recently proposed MDA methods, namely, DCTN, MDMN, MDAN, M3SDA, and MADAN, were compared in this experiment to demonstrate the effectiveness of the proposed MVDA method. The DCTN method utilizes a classifier combined with confusion scores to first label the target sample with false labels and then update the encoder and classifier together with the target sample with false labels and the source domain sample. MDMN embeds all data into a shared feature space while learning which domains share strong statistical relationships. MDAN achieves domain adaptation by optimizing task-adaptive generalization boundaries. The M3SDA method utilizes moment matching for multi-source domain transfer learning, completing the MDA task by aligning the source domain with the target domain and simultaneously aligning the source domains with each other. MADAN attempts to solve the pixel-level alignment between sources and targets or the misalignment across different sources by using dynamic semantic information. These five methods all performed with an average accuracy of 84% in MDA tasks on the office dataset.

In addition, we introduced three single-source domain methods, namely, DAN [25], DRCN [30], and ADDA [29], for comparison to further demonstrate the effectiveness of the multi-source domain method. DAN measures the distance between two domains through maximum mean difference loss; DRCN verifies whether the model can correctly extract common features among different sources by reconstructing the target domain image; ADDA is a domain adaptation classical method that introduces adversarial thinking. These three methods all performed well on the domain-adapted classic dataset Office-31, with ADDA performing the best, reaching an average level of 83.0% among all single-source migration tasks in Office-31. Due to the execution of these methods in a single-source setting, we need to verify whether the multi-source method is superior to the single-source method. Therefore, we have introduced three MDA criteria: (1) single best: displays the best single-source migration results across multiple source domains; (2) source combine: all-source domains are combined to form the traditional single-source domain and target domain settings; (3) multi-source: the results of the MDA method. The first criterion proves whether multiple source domains are valuable. Then, the second criterion evaluates whether the best single-source domain method can be further improved by introducing other source domains. The third criterion demonstrates the effectiveness of the proposed MVDA method.

4.4 Experimental results and analysis

This study uses the above hurricane data sets to conduct MVDA and each reference experiment and sets up two migration tasks. Table 1 shows the final experimental results.

Table 1
Comparison of the accuracy (%) between MVDA and the current mainstream\\ domain adaptation algorithms

Standard Method Task 1 Task 2 Average

Single best ResNet-50 [47] 64.2 77.8 71.0

DAN [25] 65.5 78.5 72.0

DRCN [30] 65.3 79.0 72.1

ADDA [29] 67.1 81.8 74.4

Source combine ResNet-50 67.3 84.2 75.8

DAN 68.8 84.7 76.7

DRCN 69.1 84.7 76.9

ADDA 71.1 85.9 78.5

Multi-source DCTN [42] 73.0 87.4 80.2

MDMN [44] 72.6 88.4 80.5

MDAN [45] 73.2 87.4 80.3

M3SDA [43] 74.5 88.9 81.7

MADAN [46] 75.6 89.1 82.4

MVDA(ours) 76.5 90.2 83.3

Standard	Method	Task 1	Task 2	Average
Single best	ResNet-50 [47]	64.2	77.8	71.0
	DAN [25]	65.5	78.5	72.0
	DRCN [30]	65.3	79.0	72.1
	ADDA [29]	67.1	81.8	74.4
Source combine	ResNet-50	67.3	84.2	75.8
	DAN	68.8	84.7	76.7
	DRCN	69.1	84.7	76.9
	ADDA	71.1	85.9	78.5
Multi-source	DCTN [42]	73.0	87.4	80.2
	MDMN [44]	72.6	88.4	80.5
	MDAN [45]	73.2	87.4	80.3
	M3SDA [43]	74.5	88.9	81.7
	MADAN [46]	75.6	89.1	82.4
	MVDA(ours)	76.5	90.2	83.3

Resnet-50 is the base network used by each method in Table 1. Among them, DAN, DRCN, and ADDA are the domain adaptation methods for single-source and target domains. In multi-source domain tasks, owing to the different collection methods among different sources, significant differences exist in the sample data distribution among different sources. During the training process, data of the same type with significant differences in data distribution will interfere with one another. Therefore, single-source domain methods are generally not as effective as multi-source domain methods in tasks with multiple sources.

Table 1 shows the following: (1) The result of source combine is better than single best, which indicates that combining all-source domains into a single-source domain is helpful for most transfer tasks owing to abundant data; (2) Comparing MVDA with DAN (source combine), the only difference is that MVDA extracts multiple domain-invariant representations in multiple feature spaces, whereas DAN extracts common domain-invariant representations in a common feature space. The results of MVDA outperform DAN, indicating that extracting a common domain-invariant representation for all domains is difficult. (3) MVDA outperforms all comparison methods in most multi-source transfer tasks, which verifies that considering domain-specific class boundaries to narrow the gap among all classifiers can help each classifier learn knowledge from other classifiers.

4.5 Analysis of image reconstruction results

Figures 5 and 6 show the reconstructed image of the target domain obtained by decoding and reconstructing the target domain features after domain adaptation in the experiment of task 2 (Sandy+Maria⟶Irma). Among them, the reconstructed image of the target domain corresponds to the label of the original image one-to-one, and the style is similar to that of the source domain image. The reason is that the target domain features have already approached the source domain features after domain adaptation, and the classifiers pre-trained in the source domain can be used to perform image classification tasks in the target domain.

Fig. 5

Comparison of source domain, target domain, and reconstructed images. Panel (a) shows the source domain Sandy image, (b) shows the target domain Irma reconstruction image, and (c) shows the target domain Irma image.

Fig. 6

Comparison of source domain, target domain, and reconstructed images. Panel (a) shows the source domain Maria image, (b) shows the target domain Irma reconstructed image, and (c) shows the target domain Irma image.

4.6 Ablation studies and discussions

In addition, this section conducts the following ablation experiments to evaluate several variants of the MVDA algorithm: (1) MVDA_cls, which preserves the classifier loss and does not consider the MMD loss; (2) MVDA_mmd, which preserves the MMD loss and does not consider the classifier loss; (3) MVDA, which considers classifier loss and MMD loss. Table 2 shows the comparison results. Experimental results show that MVDA outperforms all compared methods on multi-source transfer tasks, showing that multiple domain-invariant representations for each pair of source and target domains should be learned and domain-specific class boundaries must be considered.

Table 2
Comparison of the accuracy (%) of the MVDA algorithm and its variants

Standard Method Task 1 Task 2 Average

Multi-source MVDA_cls 72.9 88.3 80.6

MVDA_mmd 74.6 90.5 82.5

MVDA 76.5 90.2 83.3

Standard	Method	Task 1	Task 2	Average
Multi-source	MVDA_cls	72.9	88.3	80.6
	MVDA_mmd	74.6	90.5	82.5
	MVDA	76.5	90.2	83.3

This group of comparative experiments was set up to further verify the effectiveness of the MVDA classification method: (1) MVDA_mmd + S1_cls: Regardless of the classifier loss, the classifier pre-trained in the source domain is directly used to classify the target domain data after domain adaptation; (2) MVDA_mmd + S2_cls: The classifier loss is not considered, and the classifier pre-trained in the source domain is directly used to classify the target domain data after domain adaptation; (3) MVDA: The classifier loss is introduced while considering the MMD loss. Table 3 shows the comparison results. The experimental results show that the accuracy of using pre-trained classifier 1 or 2 as the target domain data classification is not as good as MVDA, and even a large gap exists. Introducing classifier loss can effectively narrow the gap among all classifiers to achieve classifier alignment.

Table 3

Comparison of MVDA algorithm accuracy (%) with and without classifier loss

Standard	Method	Task 1	Task 2	Average
Multi-source	MVDA_mmd+S1_cls	74.4	89.6	82.0
	MVDA_mmd+S2_cls	76.1	89.8	82.9
	MVDA	76.5	90.2	83.3

Relying solely on the data in Table 3 can only prove that our classification method is overall superior to a single classifier, but a good classifier should meet the standard of having a high true positive rate under the same false positive rate. The receiver operating characteristic and area under the curve (ROC-AUC) [48] line chart precisely reflects this standard, so we draw ROC line charts for three classification methods under two tasks to further validate the effectiveness of our method. As shown in Fig. 7, the red line represents the final classification results of our proposed MVDA method, the green line corresponds to the results of MVDA_mmd + S1_cls in Table 3, and the blue line corresponds to the results of MVDA_mmd + S2_cls in Table 3.

The results in Fig. 7 indicate that our classification method is effective.

Fig. 7

Performance of three classifiers. (a) task 1, (b) task 2.

We set up this group of comparative experiments to further explore the MMD loss imposed in the domain adaptation stage: (1) MVDA: calculate the MMD loss between the source domain S1 and the target domain and between the source domain S2 and the target domain, and add them up; (2) MVDA+S1S2_mmd: based on the MMD loss in MVDA, the MMD loss is also calculated between the two source domains so that the three are added together to participate in the optimization of the model. Table 4 shows the comparison results.

Table 4

Comparison of MVDA algorithm accuracy (%) with MMD loss between passive domains

Standard	Method	Task 1	Task 2	Average
Multi-source	MVDA	76.5	90.2	83.3
	MVDA+S1S2_mmd	74.8	90.3	82.5

The experimental results show that introducing MMD loss between source domains S1 and S2 did not bring better results, and the final classification accuracy was not even as good as the MVDA method. This may be because reducing the distance between S1 and target domains, as well as between S2 and target domains, and then reducing the distance between S1 and S2, actually increases the distance between the two source domains and target domains, affecting the learning of encoder 1 and encoder 2, resulting in negative migration compared to the MVDA method. Therefore, in the related experiments of MVDA, the distance between two source domains is no longer considered.

4.7 K-fold cross validation

The K-fold cross validation is a statistical method used to evaluate the performance of machine learning models, which relies on statistical analysis results to evaluate whether the model can be extended to independent datasets. In practical applications, the images to be predicted often have significant differences from the training datasets. The MVDA method proposed in this article simulates training on labeled source domain datasets and making predictions on unlabeled target domain datasets. Although the target domain image is unlabeled, it is still used for training. Therefore, using K-fold cross-validation on the target domain is to avoid over-fitting during training on the target domain.

The deployment of K-fold cross validation experiments is shown in Fig. 8. All gray parts in Fig. 8 are training data, among which the dark gray parts are data that participated in the training but did not have labels. The green part serves as the part of the validation datasets.

Fig. 8

Deployment of K-fold cross validation experiments.

The K-fold cross-validation experiments are set as follows: (1) For each task, divide each category of the target domain equally into five parts. For the Maria datasets, each section contains 684 undamaged building category images, 616 undamaged building category images, and 700 other object category images; For the Irma datasets, each section contains 600 undamaged building category images, 600 damaged building category images, and 800 other object category images. (2) For each task, use all source domains as the training set, a part of the target domain as the validation set (not participating in training, only testing), and the remaining four parts as the domain adaptation test set (the images of the four parts are trained without labels). (3) The results of using each part of the target domain as a validation set are shown in Table 5.

Table 5

K-fold cross-validation results for each part

Method	Validation Fold	Task 1	Task 2	Average
MVDA	Fold1	75.6	90.2	82.9
	Fold2	76.8	90.3	83.6
	Fold3	77.2	90.1	83.7
	Fold4	77.3	89.8	83.6
	Fold5	76.5	90.5	83.5
Average of Fold1-5	76.7	90.2	83.5

The experiment shows that our method has an error of no more than 0.9% in cross validation and can be used in independent datasets.

5 Conclusion

For the post-disaster building damage assessment task, this paper proposes the MVDA algorithm. The method consists of two stages: domain-specific distribution alignment and classifier alignment. By learning multiple domain-invariant representations and outputs from multiple source classifiers, and simultaneously aligning domain-specific distributions for each pair of source and target domains, the domain is reduced the difference between them improves the transfer performance. Extensive experiments on the hurricane datasets demonstrate the effectiveness of the proposed MVDA method, and the classification accuracy is superior to other current competing methods. After the disaster occurs, the remote sensing images captured by the satellite are simply processed, and the damaged buildings can be quickly and accurately identified through the trained MVDA model. This will help the government’s emergency management and disaster reduction and rescue departments to formulate more effective rescue plans and help rescuers find specific locations, which will significantly improve the speed of post-disaster rescue and reduce the loss of people’s lives and properties. In future work, we can try to use GAN to align each pair of source and target domains to improve feature alignment.

Footnotes

Acknowledgments

This research was supported by National Natural Science Foundation of China (Grant #62071006).

References

Frankenberg

, Sumantri

and Thomas

, Effects of a natural disaster on mortality risks over the longer term, Nat Sustain 3 (2020), 614–619.

Menderes

, Erener

and Sarp

, Automatic detection of damaged buildings after earthquake hazard by using remote sensingand information technologies, Procedia Earth Planet Sci 15 (2015), 257–262.

Y.D.

et al. Aligning discriminative and representative features: An unsupervised domain adaptation method for building damage assessment, IEEE Transactions on Image Processing 29 (2020), 6110–6122.

J.Z.

, Lu

, Li

, Khaitan

, Zaytseva

Building damage detection in satellite imagery using convolutional neural networks, arXiv2019, arXiv:1910.06444.

Jiang

et al. Building damage detection via superpixel-based belief fusion of space-borne SAR and optical images, IEEE Sensors Journal 20(4) (2020), 2008–2022.

Park

S.E.

and Jung

Y.T.

, Detection of earthquake-induced building damages using polarimetric SAR data, Remote Sens, 12(1) (2020), 137.

Vetrivel

et al. Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning, ISPRS Journal of Photogrammetry and Remote Sensing 140 (2018), 45–59.

Cao

Q.D.

and Choe

, Building damage annotation on post-hurricane satellite imagery based on convolutional neural networks, Nat Hazards 2020, 1–20.

, Liu

and Buchroithner

, Identifying collapsed buildings using post-earthquake satellite imagery and convolutional neuralnetworks: A case study of the Haiti Earthquake, Remote Sens 10 (2018), 1689.

10.

Duarte

, Nex

, Kerle

, Vosselman

Satellite image classification of building damages using airborne and satellite imagesamples in a deep learning approach, In Proceedings of the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Copernicus GmbH, Riva del Garda, Italy, Volume 4, 2018, pp. 89–96.

11.

Y.D.

, Ye

and Bartoli

, Semisupervised classification of hurricane damage from postevent aerial imagery using deep learning, J of Applied Remote Sensing 12(4) (2018), 045008.

12.

Sui

H.G.

et al. Application of remote sensing technology in building damage detection after earthquake, Journal of Wuhan University Information Science Edition 44(7) (2019), 1008–1019 (in Chinese).

13.

Andugula

, Durbha

S.S.

Post-Disaster image analysis using domain adaptation, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2015, pp. 4848–4851.

14.

Yang

, Zhang

and Luo

, Transferability of convolutional neural network models for identifying damaged buildings due to earthquake, Remote Sensing 13(3) (2021), 504.

15.

Sun

S.L.

, Shi

H.L.

and Wu

Y.B.

, A survey of multi-source domain adaptation[J], Information Fusion 24 (2015), 84–92.

16.

Riemer

et al. Learning to learn without forgetting by maximizing transfer and minimizing interference[C], International Conference on Learning Representations (ICLR), 2019.

17.

Zhu

, Zhuang

, Wang

Aligning domain-specific distribution and classifier for cross-domain classification from multiple sources[J], (2022).

18.

Wheeler

B.J.

and Karimi

H.A.

, Deep learning-enabled semantic inference of individual building damage magnitude from satellite images, Algorithms 13 (2020), 195.

19.

Ghaffarian

, Kerle

and Filatova

, Remote sensing-based proxies for urban disaster risk management andresilience: A review, Remote Sens 10 (2018), 1760.

20.

Y.D.

et al. Building damage detection from post-event aerial imagery using single shot multibox detector, Appl Sci 9(6) (2019), 1128.

21.

Abdollahi

et al. Building footprint extraction from high resolution aerial images using generative adversarial network (GAN) architecture, IEEE Access 8 (2020), 209517–209527.

22.

Kim

et al. CNN algorithm for roof detection and material classification in satellite images, Electronics 10(13) (2021), 1592.

23.

Zhuang

F.Z.

et al. Research progress on transfer learning, Journal of Software 26(1) (2015), 26–39. (in Chinese).

24.

Wang

and Deng

W.H.

, Deep visual domain adaptation: A survey, Neurocomputing 312 (2018), 135–153.

25.

Sun

and Saenko

, Deep CORAL: Correlation alignment for deep domain adaptation[C], ECCV Workshops (2016), 443–450.

26.

Zhu

, Park

, Isola

, Efros

A.A.

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks[C], 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2242–2251.

27.

Hoffman

et al. CyCADA: Cycle-Consistent Adversarial Domain Adaptation[J], Proceedings of the 35th International Conference on Machine Learning, 2018, pp. 1989–1998.

28.

Ganin

, Ustinova

, Ajakan

et al. Domain-adversarial training of neural networks[J], Journal of Machine Learning Research 17(1) (2017), 2096–2030.

29.

Tzeng

, Hoffman

, Saenko

, Darrell

Adversarial Discriminative Domain Adaptation[C], 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2962–2971.

30.

Ghifary

, Kleijn

, Zhang

et al. Deep reconstruction-classification networks for unsupervised domain adaptation[C], In European Conference on Computer Vision, 2016, pp. 597–613.

31.

Sun

S.L.

, Shi

H.L.

and Wu

Y.B.

, A survey of multi-source domain adaptation[J], Information Fusion 24 (2015), 84–92.

32.

Duan

, Xu

, Chang

Exploiting web images for event recognition in consumer videos: A multiple source domain adaptation approach[C], 2012 IEEE Conference on ComputerVision andPattern Recognition(CVPR), 2012, pp. 1338–1345.

33.

Schweikert

et al. An empirical analysis of domain adaptation algorithms for genomic sequence analysis[C], Neural Information Processing Systems(NIPS), 2009.

34.

Mansour

, Mohri

, Rostamizadeh

Domain adaptation with multiple sources[C], Proceedings of the 21st International Conference on Neural 864 Information Processing Systems(NIPS), 2009, pp. 1041–1048.

35.

Zhang

N.S.

, Mohri

and Hoffman

, Multiple-source adaptation theory and algorithms[J], Annals of Mathematics and Artificial Intelligence 2 (2020), 1–34.

36.

Zhao

et al. Adversarial multiple source domain adaptation, Advances in Neural Information Processing Systems(NeurIPS) (2018), 8568–8579.

37.

Y.T.

et al. Extracting relationships by multi-domain matching[C], Proceedings of the 32nd International Conference on Neural Information Processing Systems(NIPS), 2018, pp. 6799–6810.

38.

Blake

E.S.

et al. Tropical cyclone report hurricane Sandy[J], Chemical Information and Modeling 53 (2013), 1689–1699.

39.

Xian

S.Y.

et al. Rapid assessment of damaged homes in the florida keys after hurricane Irma[J], Natural Hazards and Earth System Sciences Discussions (2018).

40.

Zorrilla

C.D.

, The view from puerto rico - hurricane maria and its aftermath[J], New England Journal of Medicine 377(19) (2017), 1801–1803.

41.

Achanta

, Shaji

, Smith

, Lucchi

, Fua

and Süsstrunk

, SLIC superpixels compared to state-of-the-art superpixel methods, IEEE Transactions on Pattern Analysis and Machine Intelligence 34(11) (2012), 2274–2282.

42.

, Chen

, Zuo

, Yan

, Lin

Deep Cocktail Network: Multi-source Unsupervised Domain Adaptation with Category Shift, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 3964–3973

43.

Peng

, Bai

, Xia

, Huang

, Saenko

, Wang

Moment Matching for Multi-Source Domain Adaptation, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 1406–1415.

44.

Y.T.

et al. Extracting relationships by multi-domain matching[C]. Proceedings of the 32nd International Conference on Neural Information Processing Systems(NIPS), 2018, pp. 6799–6810.

45.

Zhao

et al. Adversarial multiple source domain adaptation. Advances in Neural Information Processing Systems(NeurIPS), 2018, pp. 8568–8579.

46.

Zhao

, Li

, Xu

et al. MADAN: Multi-source adversarial domain aggregation network for domain adaptation, Int J Comput Vis 129 (2021), 2399–2424.

47.

, Zhang

, Ren

, Sun

Deep residual learning for image recognition, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 5, 7, 8. 909

48.

Saberi-Movahed

, Mohammadifard

, Mehrpooya

, Rezaei-Ravari

, Berahmand

, Rostami

, Karami

, Tavassoly

Decoding clinical biomarker space of COVID-19: Exploring matrix factorization-based feature selection methods, Computers in Biology and Medicine 146 (2022), art. no. 105426.

Post-disaster building damage detection using multi-source variational domain adaptation

Abstract

Keywords

1 Introduction

2 Related work

3 Methodology

4.1 Datasets

4.4 Experimental results and analysis

Table 2 Comparison of the accuracy (%) of the MVDA algorithm and its variants Standard Method Task 1 Task 2 Average Multi-source MVDAcls 72.9 88.3 80.6 MVDAmmd 74.6 90.5 82.5 MVDA 76.5 90.2 83.3

Footnotes

Acknowledgments

References

Table 2
Comparison of the accuracy (%) of the MVDA algorithm and its variants

Standard Method Task 1 Task 2 Average

Multi-source MVDA_cls 72.9 88.3 80.6

MVDA_mmd 74.6 90.5 82.5

MVDA 76.5 90.2 83.3