Generative multiview inpainting for object removal in large indoor spaces

Abstract

As interest in image-based rendering increases, the need for multiview inpainting is emerging. Despite of rapid progresses in single-image inpainting based on deep learning approaches, they have no constraint in obtaining color consistency over multiple inpainted images. We target object removal in large-scale indoor spaces and propose a novel pipeline of multiview inpainting to achieve color consistency and boundary consistency in multiple images. The first step of the pipeline is to create color prior information on masks by coloring point clouds from multiple images and projecting the colored point clouds onto the image planes. Next, a generative inpainting network accepts a masked image, a color prior image, imperfect guideline, and two different masks as inputs and yields the refined guideline and inpainted image as outputs. The color prior and guideline input ensure color and boundary consistencies across multiple images. We validate our pipeline on real indoor data sets quantitatively using consistency distance and similarity distance, metrics we defined for comparing results of multiview inpainting and qualitatively.

Introduction

The rendering of real indoor spaces is typically achieved via an image-based rendering (IBR) method ^{1 –3} that allows a free-viewpoint exploration in a virtual world. Recent IBR applications ¹ have reduced geometric complexity for the real-time rendering of large-scale indoor spaces. The key idea of the method ¹ is to render only the architectural components without objects, which requires a consistent object removal over multiple images, as suggested in the literature. ^3,4

Multiview inpainting is a task to achieve image inpainting on multiple images while satisfying two conditions: color consistency and boundary consistency. For example, in Figure 1, images of C₂ and C₃ require image inpainting as ideal case. However, typical image inpainting methods might result in color or boundary inconsistency (cf. Figure 1(b)).

Figure 1.

Example of multiview inpainting: (a) Architectural component visible directly in images C₁ and C₄ but occluded by an object in images C₂ and C₃; (b) image inpainting is required in C₂ and C₃ as ideal image, avoiding color or boundary inconsistencies.

Philip and Drettakis ³ proposed a method for multiview inpainting. Their proposed method begins with estimating the plane, projecting visible quadrangle parts of images into a common rectified plane, and then filling in the occluded parts using a conventional patch-based method. ⁵ Although this method can ensure the color consistency up to the performance level achieved in traditional inpainting algorithm named PatchMatch, ⁵ the boundary consistency is not secured as the planar assumption is not applicable in nonplanar or complex environments.

Recent advances in image inpainting are based on deep learning approaches, ^{6 –8} which have shown their efficiency and effectiveness on applying any form of mask. Recent studies ^7,8 further enhance the performance by utilizing guidelines. Although the guidelines themselves ensure the boundary consistency in case the user sets consistent guidelines over the multiple images, the color consistency over multiple images cannot be secured as such approaches consider only a single image.

In this study, we target object removal in the images covering large-scale indoor spaces. We propose a novel pipeline that fills in occluded regions of multiple images in such a way that both the color and the boundary consistencies are well preserved. The proposed pipeline extends the state-of-the-art GConv ⁸ applied to a single-image inpainting problem.

First, for the color consistency, we create a color prior image, which is a pixel-wise color information observed from visible cameras (e.g. C₁ and C₄ in Figure 1(a)). This color prior is added along with other inputs of the inpainting network such as a masked image, a guideline input, and a mask similar to that of GConv, ⁸ allowing the inpainted colors to be consistent in multiple inpainted images.

Second, for the boundary consistency, we extend GConv ⁸ in such a way that imperfect guideline input can be refined within the network. Ideally, there should be no boundary inconsistency when the guidelines are perfectly set by users. However, because it is a laborious task to set the guidelines over multiple images, typical guidelines include various errors such as in the angle or location or missing information.

For an experiment validation, we utilized real indoor data sets ^1,9 enabling a real-time rendering of large-scale indoor spaces. For the color consistency, we developed new measures to evaluate intensity-free color consistency among multiple inpainted images, as well as the similarity between multiple inpainted images and images without occlusions. Given the metrics, the proposed method yields improvements of up to 46.4% and 30.6% over the previous single-image inpainting method EC ⁷ and GConv, ⁸ respectively. Regarding the boundary consistency, we conducted a qualitative comparison to determine whether the proposed pipeline predicts the enhanced guidelines.

This article is organized as follows: the second section briefly summarizes related works. The third section describes the proposed pipeline. The fourth section details the experiment results of multiview inpainting on real data sets. Finally, some concluding remarks are given in the fifth section. Our codes are available at https://github.com/kimjh069/generative-MVI. ¹⁰

Related work

Indoor modeling and IBR

Typical IBR methods require 3D map or model of the environment. Henry et al. ¹¹ use depth cameras integrating depth and color information for robust 3D mapping of indoor environments. The map, however, remains in point cloud rather than mesh, which makes hard for rendering novel viewpoints. Shao et al. ¹² introduce interactive approach for semantic modeling of indoor spaces. The approach retrieves 3D models from predefined database, which has limitations on reconstructing the real scene. More recent works focus on the architectural components for better representation of indoor spaces. Ikehata et al. ¹³ devise structural grammar that divides the elements of indoor environments such as rooms, walls, and objects. The approach builds the indoor model with Manhattan assumption in a single-story buildings. Recent works overcome Manhattan assumption for indoor modeling or integrate the 3D model with IBR. ^9,14

Hedman et al. ² proposed a real-time IBR method. However, this method has limitations in rendering larger than room-scale spaces. Turner et al. ¹⁵ reconstruct an architectural mesh but do not remove objects in the images, which results in flattened object textures on the architectural mesh. In addition, recent studies ^1,9 have focused on architectural component modeling and objects removal in images. However, the object removal in an image might cause a discordance of colors between multiple images.

Recently, there are attempts to use learning-based methods for IBR. Hedman et al. ¹⁶ use deep learning approach for blending novel views for IBR. It showed feasibility of using deep learning-based blending but needs further research to reduce blurriness and flicker. Thies et al. ¹⁷ use a novel deep learning-based method for image synthesis and real-time rendering. However, the method needs training for every new object requiring more research for the generalization.

Multiview inpainting

To achieve color consistency through multiview inpainting, other studies ^3,4 warp multiple images onto a common rectified target image plane and optimize the objective function or process traditional patch-based algorithms. Li et al. ¹⁸ apply RGB-D sensor, a device that augments the image with depth information, and employ exemplar-based image inpainting with sequential data. However, RGB-D sensors are inappropriate for large-scale indoor scanning or rendering owing to a short range of depth and an extremely large number of required images. Other approaches such as video inpainting ¹⁹ for recovering consistent color using dense sequential data, however, are not applicable on sparse data or wide-baseline data used for IBR. ^1,9,3

Single-image inpainting

Traditional inpainting algorithms ^5,20 use patch based or diffusion-based approaches with low-level features. Recent studies have focused on rapidly advanced deep learning-based approaches, such as a convolutional neural networks and generative adversarial networks (GANs), ²¹ which are capable of using high-level features.

Yu et al. ⁶ employed a GAN and proposed a coarse-to-fine network as well as a contextual attention module (CAM). CAM attends to known patches of features directly, and explicitly borrows patches to fill in unknown regions of features. It has inspired numerous recent studies ^8,22 making use of CAM for image inpainting. Yu et al. ⁸ proposed a gated convolution that enables a sketch channel input to propagate through a network along with semantic information.

An increasing number of recent studies ^7,23,8 have applied edge information (i.e. guideline) as an additional input for the purpose of enhancing the quality or achieving editable results. However, no studies have focused on combining the inputs of color and imperfect guidelines and a guideline refinement at the same time.

Method

In this section, we introduce the initial data and a multiview inpainting pipeline, which is shown in Figure 2, including a method for creating the color prior images and an image inpainting network. The color prior image is used as the additional input for the image inpainting network.

Figure 2.

Overview of our multiview inpainting pipeline. “Initial data” section describes the initial input data of the pipeline that was acquired by a mobile sensor robot system. The method of generating color prior images is described in “Color prior generation” section. Using the color prior images, “Image inpainting” section describes image inpainting that aims for the color consistency in multiview.

Initial data

Initial data consist of architectural mesh, images, and camera poses of each image of large-scale indoor spaces that were introduced in recent works. ^1,9 The data sets were acquired by scanning the indoor spaces employing a mobile robot system (cf. Figure 2). Global guidelines are obtained by automatic algorithm, ⁹ which are overlaid with user guidelines. Masks were made manually for the purpose of thorough multiview inpainting.

Color prior generation

A color prior image is a synthetic image created by projecting a colored point cloud data (PCD) onto the image plane. We introduce a method for coloring the PCD with only architectural component colors and projecting the colored PCD onto the images.

The PCD is initially sampled from the object-free architectural mesh. From the sampled PCD, we adopt the hidden point removal (HPR) algorithm, ²⁴ which is a simple and fast algorithm that approximates the visibility of the PCD for a given camera pose by flipping the point clouds and computing a convex hull onto it. To be more specific, for each given camera position, $c_j\in {ℝ}^3$ , the HPR algorithm yields a binary vector $v_j\in {\{0,1\}}^{m\times 1}$ indicating the visibility of the point clouds. We make the relation matrix $P^v\in {ℝ}^{m\times \left(3+n\right)}$ as

$P^v=\left[\begin{array}{cc}P& V\end{array}\right]$

1

where $P\in {ℝ}^{m\times 3}$ is the 3D position of the point clouds, $V=\left[\begin{array}{c}{v}_1,v_2,\dots ,v_n\end{array}\right]$ , m is the number of points, and n is the number of camera poses.

The next step is coloring the PCD with architectural colors and creating color prior images. For each point, we make candidates of the proper color by projecting the point onto each image plane where the point is visible. We can refer to a row of the relation matrix, $P_i^v$ , to find images in which the point is visible. If the point in a visible image plane corresponds to a nonmasked region (i.e. architectural component), the color becomes a candidate for the proper color, as shown in Figure 3(a). Among the accumulated candidates of color for a point, we simply take their median to determine the color of the point and achieve a robustness to the outliers and an efficient computation.

Figure 3.

Creating the color prior image: (a) The color for a given point is computed from a set of images that are not occluded by objects; (b) a colored point with the coverage space is projected onto the image plane to create the color prior image.

The colored point can be projected onto each visible image plane to produce a color prior image, where we assume a 360-camera model as a cubemap format

$p_{proj}=s_{img}\cdot \frac{{}^c{R}_w{}^{w}P{+}^c{T}_w}{\text{max}\left(\left|{}^c{R}_w{}^{w}P{+}^c{T}_w\right|\right)}=\left[\begin{array}{c}u\\ v\\ w\end{array}\right]$

2

where ${}^c{R}_w$ and ${}^c{T}_w$ are the rotation matrix and translation vector from the camera with respect to the world coordinates, respectively. In addition, ${}^{w}P$ is the point with respect to (w.r.t.) world coordinates, and $s_{img}$ is the image size.

The color prior image, however, depends on the sampling rate and the distance to a point from the camera pose. To be more precise, the sparse PCD originates from minimum sampling distance of r from a mesh. In addition, the points closer to an image plane appear even sparser in the image than those in farther (cf. Figure 4(a)).

Figure 4.

Example of created color prior: (a) The colored point closer to the camera appears sparser than that farther away; (b) we resolve the sparsity issue by defining the coverage area; (c) the original image from the given viewpoint.

To resolve the dependencies, we define point coverage area, $C_{proj}$ , in the image plane as shown in Figure 3(b). The distance between two points along the diagonal will be $\sqrt{3}r$ in 3D space. Hence, we set the coverage radius of a point to $\frac{\sqrt{3}}{2}r$ allowing some overlap. The coverage space, $C_{init}$ , of a point becomes a sphere

$C_{init}={\left(\frac{\sqrt{3}}{2}r\right)}^2{I}_3$

3

where $I_3$ refers to 3 $\times$ 3 identity matrix. The Jacobian of $p_{proj}$ w.r.t. each coordinate of a point is

$J=\frac{\partial p_{proj}}{\partial p}=\left[\begin{array}{ccc}\frac{\partial u}{\partial x}& \frac{\partial u}{\partial y}& \frac{\partial u}{\partial z}\\ \frac{\partial v}{\partial x}& \frac{\partial v}{\partial y}& \frac{\partial v}{\partial z}\\ \frac{\partial w}{\partial x}& \frac{\partial w}{\partial y}& \frac{\partial w}{\partial z}\end{array}\right]$

4

Combining these matrices, the coverage area $C_{proj}$ on the image plane can be defined by projecting the coverage space $C_{init}$ onto the image plane, which becomes an ellipse as

$C_{proj}=J{C}_{init}{J}^{{\rm T}}$

5

The coverage area, $C_{proj}$ , is approximated as a circle with a radius $r_{proj}$ equal to the minor axis of the ellipse

$r_{proj}=\text{median}\left(\sqrt{\text{eigenvalue}\left(C_{proj}\right)}\right)$

6

where $\sqrt{\text{eigenvalue}\left(C_{proj}\right)}$ gives three nonnegative values including the major axis, minor axis, and a zero value. Finally, we blur the color prior images using a Gaussian blur to alleviate the high-frequency property in the color priors, as the results in Figure 4(b) indicate.

Image inpainting

Typical image inpainting networks ^8,7 take a masked image and a guideline image as inputs and output an inpainting result. By contrast, to preserve color consistency over multiple images, we train a network that is capable of receiving the color prior information as an additional input. In addition, the network is able to perform two different tasks concurrently such as the image inpainting and guideline refinement. In the following sections, we introduce the input data we generated to train the network, architecture, and loss functions to achieve our goal: the color consistent inpainting and guideline refinement.

Input data generation for training

Given the ground truth image $I_{gt}$ and guideline $L_{gt}$ , we need to generate training input data including the color prior $I_{cp}$ , imperfect guideline $\stackrel{\sim }{L}$ , mask $M$ , and noncolor mask $M_{cp}$ as in Figure 5. The ground truth guideline $L_{gt}$ is generated by the pre-trained edge detector named as holistically nested edge detection, ²⁵ with a threshold of 0.6.

Figure 5.

From the left two ground truth images (image and guideline), we generate the other five images by automatic generation algorithms for training process of our model. The image is a sample validation image from Places2 data set.

For the generation of color prior input $I_{cp}$ and imperfect guideline input $\stackrel{\sim }{L}$ , we tried to reproduce the data similar to a real case. The pixels in the color prior input $I_{cp}$ are randomly sampled from two different randomly affine transformed and jittered ground truth images followed by Gaussian blur and random erase. The transformations are used to imitate various noises or errors that the color prior image of the real indoor data sets often contains such as sparsity, camera pose error, blurriness, occlusions by objects, and so on.

An imperfect guideline $\stackrel{\sim }{L}$ is generated through random affine transformed patches of the ground truth guideline $L_{gt}$ and is then randomly erased. These transformations are used to imitate angle, location, or missing errors that can be often made by users.

The mask $M$ is generated by a random free-form mask generation algorithm. ⁸ A noncolor mask $M_{cp}$ is another binary mask representing whether the color prior information exists in the mask

$M_{cp}=\left(\begin{array}{ll}0,\hfill & \text{where\hspace{0.17em}}{I}_{cp}+\left(1-M\right)\text{>0}\hfill \\ 1,\hfill & \text{otherwise}\hfill \end{array}\right.$

7

No-guideline zone is another binary mask used for a loss function. The usage of no-guideline zone is explained later in detail. The codes of generating the input data for the training are also available at the link. ¹⁰

Network architecture

We modify the coarse-to-fine network used in GConv ⁸ based on the key ideas from the recent works ^22,26 as shown in Figure 6. The coarse network receives an input tensor $X$ with a size of $9\times H\times W$ , where H and W are height and width of the input image, respectively. The nine input channels consist of a masked image $\stackrel{\sim }{I}$ , mask $M$ , imperfect guideline $\stackrel{\sim }{L}$ , color prior $I_{cp}$ , and noncolor mask $M_{cp}$ .

Figure 6.

The blue, orange, yellow, and green blocks in the generator represent a gated convolution, dilated convolution, self-attention module, and CAM, respectively. Each of upsampling layer consists of nearest-neighbor interpolation followed by a gated convolution. The refine network is simplified for visualization. Red blocks in the discriminator represent spectral normalized convolutions. CAM: contextual attention module.

The size of the output tensor of the coarse network is $4\times \frac{H}{2}\times \frac{W}{2}$ , where four channels consist of a coarse image output and coarse guideline correction. The refine network then receives the output of the coarse network and returns a tensor with a size of $4\times H\times W$ , in which the four channels consist of a refined image output $I_{pred}$ and refined guideline $L_{pred}$ .

The discriminator is similar to the SN-patchGAN structure in GConv ⁸ except that instead of having a stride 1 convolution at the very first layer, our discriminator starts with a stride 2 convolution directly and uses Leaky ReLU as the activation function. We also adopt a spectral normalization technique recently proposed by Miyato et al. ²⁷ The discriminator takes eight channels of input consisting of the image, guideline, mask, and color prior.

Loss functions

The coarse-to-fine network is trained in an end-to-end manner over several joint losses consisting of an adversarial loss, ${\ell }_1$ loss, perceptual loss, style loss, focal loss, and two more proposing losses including anti-specificity loss and ${\mathcal{L}}_{{\ell }_1,cp}$ loss.

Let G and D be the generator and discriminator, respectively. We use the hinge loss as the objective function

${\mathcal{L}}_D=\mathbb{E}\left(\text{ReLU}\left(1-D\left(X_{gt}\right)\right)\right)+\mathbb{E}\left(\text{ReLU}\left(1+D\left(Y_{gen}\right)\right)\right)$

8

${\mathcal{L}}_{GAN}=-\mathbb{E}\left(D\left(G\left(X\right),M,I_{cp}\right)\right)$

9

where $X_{gt}$ consists of the ground truth image $I_{gt}$ and guideline $L_{gt}$ , mask $M$ , and color prior $I_{cp}$ , whereas $Y_{gen}$ consists of the output image and guideline from generator, as well as the mask and color of input data. The ${\ell }_1$ loss is computed as follows

$\begin{array}{l}{\mathcal{L}}_{{\ell }_1}(I_{gt},I_{pred},M)=\frac{{\Vert \left(I_{gt}-I_{pred}\right)\odot M\Vert }_1}{n_M}\hfill \\ +\frac{{\Vert \left(I_{gt}-I_{pred}\right)\odot \left(1-M\right)\Vert }_1}{n_{1-M}}\hfill \end{array}$

10

where $n_M$ is the number of elements in the binary mask $M$ , $I_{gt}$ is the ground truth image, $I_{pred}$ is the output image of the generator, and $\odot$ is the Hadamard product. We normalize each term based on the mask size for proper scaling.

We also include the perceptual loss ${\mathcal{L}}_{perc}$ and style loss ${\mathcal{L}}_{style}$ proposed in works. ^28,29 We observed that the perceptual loss and the style loss help enhancing image details and reducing artifacts as addressed in the work. ³⁰ Here, ${\mathcal{L}}_{perc}$ penalizes the distance between two feature maps of a pre-trained network forcing the generator to make perceptually similar output to the ground truth image. This is computed as follows

${\mathcal{L}}_{perc}=\mathbb{E}\left[{\displaystyle \sum _i}\frac{1}{N_i}{\Vert {\varphi }_i\left(I_{gt}\right)-{\varphi }_i\left(I_{pred}\right)\Vert }_1\right]$

11

where ${\varphi }_i$ is the feature map of ith layer of the pre-trained VGG-16 with batch normalization applied on the ImageNet data set. ³¹ We use layers of pool1, pool3, pool4, and pool5. These layers are also used to compute style loss, which penalizes the difference between covariances of the feature maps. Here, ${\mathcal{L}}_{style}$ is computed as follows

${\mathcal{L}}_{style}={\displaystyle \sum _i}{\mathbb{E}}_i\left[\frac{{\Vert G_i^{\varphi }\left(I_{gt}\right)-G_i^{\varphi }\left(I_{pred}\right)\Vert }_1}{N_{H{W}_i}}\right]$

12

where $G_i^{\varphi }$ is a $C_i\times C_i$ gram matrix made from each feature map ${\varphi }_i$ of shape $C_i\times H_i\times W_i$ and $N_{H{W}_i}=H_i\times W_i$ .

The guideline refinement is trained using the focal loss ${\mathcal{L}}_{foc}$ proposed by Lin et al. ³² It regularizes the network not only to restore the ground truth guideline from randomly distorted and erased input guideline but also to sharpen the image edges in the inpainted image. It is computed as follows

${\mathcal{L}}_{foc}=-\mathbb{E}\left[{\left(L_{gt}-L_{pred}\right)}^2\odot \text{H}\left(L_{gt},L_{pred}\right)\right]$

13

where H is the binary cross entropy.

As can be seen in Figure 7, we found that the black region in the color prior affects on the prediction of the guideline and results in black holes. This is because the network regards a black region in the color prior as the proper color and predicts a borderline around the region. However, a black region is where no prior color information exists and should be filled in appropriately.

Figure 7.

(a) Masked image is overlaid with imperfect guideline. (b) Masked image is overlaid with color prior. When the model is trained without ${\mathcal{L}}_{{\ell }_1,cp}$ and ${\mathcal{L}}_{as}$ , a black hole problem arises, as shown in (c) and (d). Prediction of the guideline in (c) is affected by the black pixels where color prior does not exist and results in black holes in the inpainted images, as shown in (d).

We resolve this problem by proposing two new losses, anti-specificity loss ${\mathcal{L}}_{as}$ and another ${\ell }_1$ loss for the color prior ${\mathcal{L}}_{{\ell }_1,cp}$ . Here, ${\mathcal{L}}_{as}$ penalizes false-positive predicted guideline that belongs to the no-guideline zone, $N$ (as in Figure 5)

$\begin{array}{l}N=\left(1-L_{gt}\right)\odot Dilate\left(Canny\left(M_{cp}\right)\right)\hfill \end{array}$

14

It is where guideline prediction should be negative that belongs to the negative ground truth guideline $1-L_{gt}$ overlapping with thick edges surrounding the black region $M_{cp}$ , $Dilate\left(Canny\left(M_{cp}\right)\right)$ . The anti-specificity loss ${\mathcal{L}}_{as}$ is computed as follows

${\mathcal{L}}_{as}(L_{pred},N)=\frac{FP}{TN+FP}=\frac{\mathbb{E}[L_{pred}\odot N]}{{\displaystyle \mathbb{E}}[N]+\epsilon }$

15

where $FP=\mathbb{E}[L_{pred}\odot N]$ is the mean of the false-positive guideline prediction, $TN=\mathbb{E}[\left(1-L_{pred}\right)\odot N]$ is mean of true-negative guideline prediction, and $\epsilon$ is a small value to avoid dividing by zero. The code of ${\mathcal{L}}_{as}$ is also available at the link. ¹⁰ ${\mathcal{L}}_{{\ell }_1,cp}$ is the same as in equation (10) except for the mask $M_{cp}$ .

An adversarial loss, a perceptual loss, and a style loss are used to regularize the entire generator, whereas the others are additionally used to regularize both the coarse network and the entire generator. The overall loss function for our generative network ${\mathcal{L}}_G$ is defined as follows

$\begin{array}{l}{\mathcal{L}}_G={\mathcal{L}}_{color}+{\mathcal{L}}_{guideline}\hfill \end{array}$

16

$\begin{array}{l}{\mathcal{L}}_{color}={\lambda }_g{\mathcal{L}}_{GAN}+{\lambda }_s{\mathcal{L}}_{style}+{\lambda }_p{\mathcal{L}}_{perc}+{\lambda }_{{\ell }_1}{\mathcal{L}}_{{\ell }_1}\hfill \end{array}$

17

$\begin{array}{l}{\mathcal{L}}_{guideline}={\lambda }_{{\ell }_1,cp}{\mathcal{L}}_{{\ell }_1,cp}+{\lambda }_f{\mathcal{L}}_{foc}+{\lambda }_{as}{\mathcal{L}}_{as}\hfill \end{array}$

18

In short, ${\mathcal{L}}_{color}$ is a typical loss similar to ones used in other works. ^8,7 By contrast, ${\mathcal{L}}_{guideline}$ is introduced to resolve the black hole problem and to perform two different tasks in a single model concurrently which are the inpainting task, ${\mathcal{L}}_{color}$ , and the guideline refinement task, ${\mathcal{L}}_{foc}$ . We choose ${\lambda }_g=0.1$ , ${\lambda }_s=200$ , ${\lambda }_p=0.08$ , ${\lambda }_{{\ell }_1}=0.5$ , ${\lambda }_{{\ell }_1,cp}=0.7$ , ${\lambda }_f=1$ , and ${\lambda }_{as}=0.1$ .

Experiments and results

We evaluate our multiview inpainting pipeline on real indoor spaces introduced in recent works. ^1,9 The properties of each space are summarized in Table 1. The degree of light difference, occlusions, or environmental complexity of a space is indicated qualitatively by “+” sign, where more “+” signs represent harder conditions for multiview inpainting (Experiencing real-time rendering using our multiview inpainting results is available at www.teevr.net:2019/generative-mvi. ³³ )

Table 1.

Properties, statistics, and results on real data sets.

Data set	Space 1	Space 2	Space 3
Size (m2)	80	1930	2804
Camera poses	5	29	59
Num. images	30	174	354
Light difference	+	++	+++
Occlusions	+++	+	++
Env. complexity	+	+++	++
r ( $cm$ )	0.75	1.25	2.75
Evaluation points	253,798	844,441	501,873
avg(ni)	2.65	3.25	4.67
avg(mi)	2.16	5.92	8.52
EC − avg(dc)	0.0140	0.0238	0.0236
GConv − avg(dc)	0.0108	0.0222	0.0200
OURs w/o cp − avg(dc)	0.0099	0.0209	0.0171
OURs w/ cp − avg(dc)	0.0075	0.0184	0.0166
EC − avg(ds) 7	0.0170	0.0262	0.0269
GConv − avg(ds) 8	0.0152	0.0266	0.0233
OURs w/o cp − avg(ds)	0.0145	0.0249	0.0210
OURs w/ cp − avg(ds)	0.0110	0.0226	0.0200

Among the different methods, the best performance of avg(d_c) and avg(d_s) are indicated in boldface.

For comparison, we selected recent two state-of-the-art papers that utilize the edge information, EC ⁷ and GConv. ⁸ EC consists of two separate generative networks, one predicting the image edge and the other completing the inpainting with the use of the edge prediction. By contrast, GConv uses a user-sketch input for the image inpainting but does not predict the edge. Our model uses imperfect guideline as well as the color prior as input and refines the guideline and completes the image inpainting concurrently. We used 512 $\times$ 512 images of the Places2 ³⁴ data set for training our model, which is the same resolution as the indoor data sets. EC was pre-trained over the Places2 data set using 256 $\times$ 256 images.

Color evaluation

Before beginning color evaluation for multiview inpainting, pre-processing is necessary to alleviate luminance dependency due to a lot of light sources affecting differently on color intensity w.r.t. camera poses. To resolve the problem, all images are normalized for conversion into the intensity-free rg chromaticity space ³⁵ as

$\begin{array}{l}r=\frac{R}{R+G+B},g=\frac{G}{R+G+B}\hfill \end{array}$

19

where r and g are bounded between 0 and 1.

Following data arrangement is also processed for pixel-wise evaluation. For a given 3D point p_i from the sampled PCD, two sets of images are defined, ${\mathbb{A}}_i$ and ${\mathbb{B}}_i$ . ${\mathbb{A}}_i$ is a set of images that p_i is occluded by object while ${\mathbb{B}}_i$ is a set of images that the point is directly visible (i.e. sets of inpainted images vs. architectural component visible images). Then pixel-wise rg chromaticity matrices, $A_i$ and $B_i$ , can be constructed by utilizing $P^v$ and $p_{proj}$ in equation (1) and (2) as

$\begin{array}{l}{A}_i=\left[\begin{array}{cc}{r}_1& g_1\\ ⋮& ⋮\\ r_{n_i}& g_{n_i}\end{array}\right],B_i=\left[\begin{array}{cc}{r^{\prime }}_1& {g^{\prime }}_1\\ ⋮& ⋮\\ {r^{\prime }}_{m_i}& {g^{\prime }}_{m_i}\end{array}\right]\hfill \end{array}$

20

where r_j and g_j are the rg chromaticity of the pixel corresponding to the point p_i in jth inpainted image belonging to the set ${\mathbb{A}}_i$ , and similarly ${r^{\prime }}_k$ and ${g^{\prime }}_k$ are the rg chromaticity of that in kth visible image belonging to the set ${\mathbb{B}}_i$ . In addition, n_i and m_i refer to the number of elements in the set ${\mathbb{A}}_i$ and ${\mathbb{B}}_i$ , respectively, where n_i is larger than one and m_i is larger than zero. The number of point clouds to evaluate that satisfy the conditions are subset of the overall sampled PCD, which is shown in 7th row of Table 1. The average of the number of elements n_i and m_i over the points set is presented in the following rows of the Table 1.

For the evaluation, we compare two factors, consistency over n_i images (d_c) and similarity between the sets ${\mathbb{A}}_i$ and ${\mathbb{B}}_i$ (d_s) as visualized in Figure 8. First, consistency distance d_c is computed as follows

$\begin{array}{l}{\stackrel{\sim }{A}}_i=A_i-{\overline{A}}_i\hfill \end{array}$

21

$\begin{array}{l}{d}_c=\sqrt{\text{max}\left(\sigma \left(\frac{1}{n_i-1}\left({\stackrel{\sim }{A}}_i^{{\rm T}}{\stackrel{\sim }{A}}_i\right)\right)\right)}\hfill \end{array}$

22

where ${\overline{A}}_i$ is $A_i$ mean over n_i-elements and $\sigma$ refers to singular values. We decide consistency distance as the square root of the largest singular value of sample covariance of $A_i$ .

Figure 8.

Let a given 3D point p_i be occluded by objects in five images ( ${\mathbb{A}}_i$ ) and visible directly in four images ( ${\mathbb{B}}_i$ ). A red point corresponds to rg chromaticity of the pixel from an inpainted image. A blue point is from an image that shows architectural color directly. d_c is consistency distance and d_s is similarity distance.

Second, similarity distance d_s is the Euclidean distance between the means of the matrices $A_i$ and $B_i$ , which is computed as

$\begin{array}{l}{d}_s={\Vert {\overline{A}}_i-{\overline{B}}_i\Vert }_2\hfill \end{array}$

23

Finally, since d_c and d_s are point-wise measures, they are averaged over all points. Evaluations were conducted on EC, GConv, OURs in the absence of the color prior to verify the effects of the color prior, and OURs using the color prior. The results are presented in the Table 1. Improvements of up to 46.4% and 30.6% in the consistency distance and also up to 35.6% and 27.3% in the similarity distance were achieved compared to EC and GConv, respectively. The qualitative comparisons are shown in Figure 9.

Figure 9.

Sample test results of Space 3. Other methods result in color inconsistency while our method using color prior preserves color consistency in the results of multiview inpainting.

Boundary evaluation

We evaluate boundary consistency qualitatively. As can be seen in Figure 10, We observed that EC is good at finding finer edges because it was trained using a Canny edge detector followed by nonmaximum suppression. However, it often fails to generate proper edge on the area where the architectural boundary should exist in a real data set (cf. Figure 10(b) and (c)). By contrast, both GConv and our approach achieve better results in terms of the architectural boundary because of the explicit guideline inputs. Furthermore, with the approach of guideline refinement in the network, Figure 11 shows that our model is able to find the missing guidelines in the images.

Figure 10.

(a) The original image. (b) Boundary comparison on the left side pillar. (c) Boundary comparison on the right-side pillar.

Figure 11.

A sample result of missing guidelines. Line patterns on the left side pillar in the input guideline image are not drawn but are visible in the refined guideline output. For visualization, the output guideline corresponding to the mask is colored in blue and emphasized.

Ablation studies

For the quantitative evaluation of multiview inpainting on the real indoor data sets, ^1,9 we defined the point-wise consistency metrics, consistency distance and similarity distance, which are averaged over all points. However, averaging over the number of all points might blur the point of multiview inpainting, which is color-consistent inpainting regardless of the number of inpainted images.

This time, we further analyze the impact of the number of inpainted images on the color consistency. For this, each point, p_i, is classified according to the number of times, n_i, that it is used for inpainting. Then, the consistency distance and similarity distance of each classified group of points are averaged over the number of the group’s points. In specific, n_i is the number of rows in the matrix $A_i$ in equation 20. Finally, we have the average values of each metric according to the number of inpainted images and make plots of them as Figures 12 and 13.

Figure 12.

Quantitative comparisons of consistency distance for different models: (a), (b), and (c) are the results of the Space 1, 2, and 3, respectively. The dashed vertical line represents the average value of n_i’s of the given space. Our approach shows lower value on every n compared to the other methods. It means our approach outperforms the others on the perspective of consistency distance regardless of the number of inpainted images of the multiview inpainting task.

Figure 13.

Quantitative comparisons of similarity distance for different models: (a), (b), and (c) are the results of the Space 1, 2, and 3, respectively. The dashed vertical line represents the average value of n_i’s of the given space. Our approach shows lower value on every n compared to the other methods. It means our approach outperforms the others on the perspective of similarity distance regardless of the number of inpainted images of the multiview inpainting task.

Figure 12 shows the impact of n_i on consistency distance. Consistency distance tends to increase as the number n_i increases. Our approach consistently shows lower value on every n_i compared to other single-image inpainting methods. Likewise, Figure 13 also shows similar results and that our approach outperforms other methods on every n_i in similarity distance. Both results suggest that our pipeline using the color prior is more effective on the multiview inpainting task regardless of the number of inpainted images compared to the single-image inpainting approaches. Figure 14 shows more qualitative comparisons on the real data sets.

Figure 14.

Qualitative comparisons on real data sets. Each pair of rows is Space 1, 2, and 3 of the real data set, respectively.

Conclusions

We proposed a multiview inpainting pipeline, that is, creating color prior images and an image inpainting network. Our pipeline aims at achieving both color and boundary consistency for multiview inpainting by introducing a color prior and guideline input, respectively. While training the network, we introduced new losses to resolve the black hole problem and achieve two goals, inpainting and guideline refinement, concurrently. We evaluated the results of our pipeline using a consistency distance and a similarity distance, which indicate that the method outperforms other methods applied to multiview inpainting.

We plan to extend the proposed multiview image inpainting with a better pipeline to further stabilize the light effects during the color prior generation step. There are also other promising applications and future works using our pipeline. Since the inpainting is affected by the color prior images, user-interactive inpainting can be possible. Users may remodel their virtual spaces as the inpainting can be modified by user input. Similarly, our method can be used for image synthesis in indoor or outdoor of target space if the point clouds and poses of images are given. Also, our pipeline may be used as a preprocessing and extended to recent deep learning-based rendering approaches.