Sage Journals: Discover world-class research

Abstract

Image segmentation and registration are two of the most challenging tasks in medical imaging. They are closely related because both tasks are often required simultaneously. In this article, we present an improved variational model for a joint segmentation and registration based on active contour without edges and the linear curvature model. The proposed model allows large deformation to occur by solving in this way the difficulties other jointly performed segmentation and registration models have in case of encountering multiple objects into an image or their highly dependence on the initialisation or the need for a pre-registration step, which has an impact on the segmentation results. Through different numerical results, we show that the proposed model gives correct registration results when there are different features inside the object to be segmented or features that have clear boundaries but without fine details in which the old model would not be able to cope.

Keywords

Image registration non-parametric image registration interactive segmentation variational models

Introduction

Image segmentation aims to separate objects or features in the image that have similar characteristics into different classes or sub-regions, via detection and visualisation of the contours of the objects in the images. Meanwhile, image registration is the process of finding a geometric transformation between images such that the template (target) images are aligned with the reference (source) images. In a wide range of fields, such as medical image processing, pattern recognition, geophysics, comparison of data to a common reference frame, comparison of images taken at different times, shape tracking or similar problems are challenging issues that are encountered. In those cases, image registration and segmentation depend on each other and should be treated simultaneously in a joint framework. One important applications of such a combination can be found in Gooya et al.¹ and similar article where atlases are constructed from magnetic resonance (MR) scans to analyse and understand brain tumour development. The task of construction of the atlases requires alignment of the brain tumour MR scans to a common coordinate system and the automatic segmentation of the scans. According to Erdt et al.,² 25% of published works in medical imaging literature are joint segmentation and registration methods. Many of the methods developed in this context used shape prior models in an energy minimisation framework. The first work on variational model for joint region based segmentation and registration was proposed for rigid registration by Yezzi et al.³ Later, other publications extended the work on segmentation and rigid registration, see literature.^4–8 Those approach however involves a pre-segmentation step, using different criteria for segmentation and rigid registration in a sequence of images, hence is not a joint segmentation and registration approach and will fail for shapes, which non-rigidly deforms in different images. On the other hand, it is worth mentioning approaches developed for the purpose of non-rigid registration.^9–11 These techniques globally register images and estimate the deformation field over the whole image and work for non-rigid deformations, such as registration for CT and MR images. These models have difficulties with multiple objects or they do highly depend on the initialisation, which has an impact on the segmentation results. In difference with literature,^3,11 Wang and Vemuri¹² propose a registration and segmentation model for multi-modality images using cross cumulative residual entropy as a distance measure for registration. To model the deformation, Wang and Vemuri¹² used the parametric model based on cubic B-spline and for segmentation the piecewise constant Chan–Vese (CV) model.¹³ However, the model requires segmentation of the reference image and the work can be considered as registration driven by segmentation.

On the other hand, it is worth mentioning the work of Le Guyader and Vese (GV-JSR),¹⁴ which presents a non-rigid coupled segmentation and registration using the non-linear elastic model to register the segmented template and reference images. The model manages to produce topology-preserving segmentation where the initial contour from the template image is deformed to the contour of the reference image without merging or breaking and allows large deformations to occur. However, the model is limited to the well-defined objects or features that have clear boundaries but without fine details.

The contribution of this article is twofold. First, to improve the GV-JSR model, for cases where the objects are with fine details, by adding a weighted Heaviside sum of the squared difference (SSD) term in the GV-JSR model. Second for a better registration, invariant to the affine registration and which allows large deformation, we use the linear curvature model^15,16 to replace the nonlinear elastic term in the GV-JSR model. In this way, there is no need for a pre-registration step to cater for affine linear transformation.¹⁵ Beside the ability to recover affine linear transformation, the linear curvature model for registration also produces more smooth transformation than a nonlinear elastic model. It is well known that low-order regularisation terms, such as nonlinear elasticity are less effective than high-order ones such as linear curvature in producing smooth transformations.^17,18 To the best of our knowledge, only diffusion, linear and nonlinear elastic model for non-parametric image registration have been used in the task of joining segmentation and registration.

The outline of this article is as follows: In “Relation to previous work: The GV-JSR model” section, we review the task of joining segmentation and registration. In “The proposed NJSR model” section, we introduce our proposed new joint segmentation and registration (NJSR) model, which improves the original GV-JSR model. We show in “Numerical results” section, some numerical tests including comparisons. Finally, we present our conclusion and future work in the “Conclusion” section.

Relation to previous work: The GV-JSR model

The idea of joining the tasks of segmentation and registration utilised by Le Guyader and Vese¹⁴ using level set representation, which aligns the contour of the template image and simultaneously segment the reference image demonstrate a state of art work with a potential of large deformation of displacement field guided by a segmentation process. The method relates both problems using an active contour based segmentation idea,¹³ which is solved in terms of the displacement field. In this section, we provide a brief review of the variational formulation of GV-JSR model for joint segmentation and registration. Before we proceed, we introduce some notation.

Let T denote the template image and R the reference image, $R, T : Ω \subset ℝ 2 \to ℝ$ , given as compactly support functions and denote by $ϕ = ϕ (x) : Ω \to ℝ 2$ , the unknown transformation aiming for $T (ϕ (x)) \approx R (x)$ with $x = (x 1, x 2)$ . In the non-parametric (variational approach) image registration, the transformation is written as $ϕ (x) = x + u (x)$ , with $u (x)$ the displacement vector field defined as $u (x) : Ω \to ℝ n, (n = 2 or 3)$ . This transformation enables us to focus on the unknown displacement vector $u (x) = (u 1 (x), u 2 (x))$ . Here, $u (x)$ is searched over admissible functions in the set $U$ , a linear subspace of a Hilbert space with Euclidean scalar product.

The GV-JSR model uses the initial given segmentation of the template image to find the geometric transformation of the template image and the segmentation of the reference image. The segmentation of the template image is represented by the zero level line $φ 0 : Ω \to ℝ$ to represent target contour Γ given as

{Γ = \partial Ω 1 = {(x 1, x 2) \in Ω | φ 0 (x 1, x 2) = 0}, inside (Γ) = Ω 1 = {(x 1, x 2) \in Ω | φ 0 (x 1, x 2) > 0}, outside (Γ) = Ω 2 = {(x 1, x 2) \in Ω | φ 0 (x 1, x 2) < 0} .

(1)

The joint functional for segmentation and registration¹⁹ is given by

min c 1, c 2, u (x) J (c 1, c 2, u (x)) = λ 1 \int Ω | R (x) - c 1 | 2 H ε (φ 0 (x + u (x))) d x + λ 2 \int Ω | R (x) - c 2 | 2 (1 - H ε (φ 0 (x + u (x)))) d x + α S NLE (p) + α β | | p - \nabla u (x) | | 2

(2) where c₁ and c₂ are the average intensities inside and outside the curve Γ in the reference image, which is represented by the zero level line as in equation (1) and

H ε

is a regularised Heaviside function

H ε (z) = \frac{1}{2} (1 + \frac{2}{π} \arctan \frac{z}{ε})

with its corresponding Delta function

δ ε (z) = \frac{d H ε (z)}{d z} = \frac{ε}{π (ε 2 + z 2)} .

The variable p shown in equation (2) is a matrix auxiliary variable, which approximates the Jacobian matrix of $\nabla u (x)$ helping in reducing the nonlinearity in the regularisation term. It is given by

p = (p 11 p 12 p 21 p 22) \approx \nabla u (x) = (\frac{\partial u 1}{\partial x 1} \frac{\partial u 1}{\partial x 2} \frac{\partial u 2}{\partial x 1} \frac{\partial u 2}{\partial x 2}) .

(3)

The regularisation term in (2), denoted by $S NLE$ , is the nonlinear elastic regularisation term for image registration based on Yanovsky et al.,^20–22 which is given by

S NLE (p) = \int Ω \frac{λ}{8} (2 (p 11 + p 22) + p_{11}^{2} + p_{12}^{2} + p_{21}^{2} + p_{22}^{2}) 2 + \frac{μ}{4} ((2 p 11 + p_{11}^{2} + p_{21}^{2}) 2 + (2 p 22 + p_{12}^{2} + p_{22}^{2}) 2 + 2 (p 12 + p 21 + p 11 p 22 + p 21 p 22) 2) d x

(4) where λ and μ are the Lame constants. The model uses the Dirichlet boundary condition.

The GV-JSR model^14,19 is incorporated with the regridding step, thus it manages to recover large deformation. The idea of regridding is proposed by Christensen et al.²³ to model large deformation. The regridding step is as follows. The determinant of the Jacobian matrix of the transformation is calculated during the registration process to make sure there is no folding or cracking in the deformation field. If the minimum value of the determinant falls below a certain threshold, the last displacement field is stored and the template image is initialised using the last displacement field. Then, the displacement field is set to zero and the process is continues until convergence. In Cahill et al.,²⁴ the authors extend the regridding concept and show how the method can be applied in the case of other regularisation terms such as diffusion, linear curvature and linear elastic with several types of boundary conditions. For example, to solve the famous large deformation problem, where we want to align a letter C with a dot (refer to Modersitzki¹⁸ for more details), the model requires two regridding step. So, it is natural to any regularisation based models to recover large deformation as long as the regridding step is incorporated in the model.

One of the main advantages of the GV-JSR model is the ability to produce topology-preserving segmentation where the initial contour from the template image is deformed to the contour of the reference image without merging or breaking. The contour of the reference image is the deformed version of the contour of the template image using the found smooth transformation. It is deformed without separation of the initial contour from the template image, which is difficult to achieve with the standard level set implementation of the active contour.¹⁴ Topology preservation is important for several applications in medical imaging such as in computational brain anatomy. The GV-JSR model manages to preserve the topology of the initial contour without corporation of soft or hard constraint in the model. Based on our experiments, however, we found that the model is only suitable to single object in a well-defined image with relatively large structures. Registration process is only drive by the forces on the boundary of the outer structures of the objects and does produce an incomplete deformation field for the inner structures of the objects.

The proposed NJSR model

Since in GV-JSR model, the registration process is only drive by the forces on the boundary of the outer structures of the objects, it produces an incomplete deformation field for the inner structures of the objects. To deal with the two cases where the GV-JSR model fails to register, we propose to include two new terms in the functional (2). The first term is a SSD term of the form

D SSD (T, R, u (x)) = \frac{1}{2} \int Ω (T (x + u (x)) - R (x)) 2 d x

(5) which is weighted by the parameter λ₃ and the term

H ε (φ 0 (x + u (x)))

and the second term is the linear curvature term to regularise the deformation field in the NJSR model. Thus, our new NJSR model is the following

min c 1, c 2, u (x) J (c 1, c 2, u (x)) = λ 1 \int Ω | R (x) - c 1 | 2 H ε (φ 0 (x + u (x))) d x + λ 2 \int Ω | R (x) - c 2 | 2 (1 - H ε (φ 0 (x + u (x)))) d x + D SSDH (T, R, φ 0 (x), u (x)) + α S LC (u)

(6) where

D SSDH (T, R, φ 0 (x), u (x)) = λ 3 \int Ω (T ((x + u (x)) - R (x))) 2 H ε (φ 0 (x + u (x))) d x

(7) and

S LC (u) = \int Ω (Δ u 1) 2 + (Δ u 2) 2 d x .

(8)

$D SSDH$ is a weighted $L 2$ norm of the difference in the intensity value between the reference and template images. When the intensity values of the region in the transformed template image is not equal to the intensity of the corresponding region in the reference image, this term will be turn on (active). The strength of this term is controlled by the regularised Heaviside function $(H ε)$ of the deformed level set from the transformed template image. Schumacher et al.²⁵ also present similar work in registration where the fitting term in their model is the SSD term weighted by the segmentation of the template and reference images. For the term $D SSDH$ to be successful, the intensities of the reference and template images must be comparable. Thus, it is only applicable to mono-modal applications where images generated from the same imaging machine. However, the term can be adjusted to multi-modal images using normalised gradient field or cross correlation distance measures. The other term $S LC$ is a smoothing term based on linear curvature registration as introduced in Fischer and Modersitzki.²⁶ As stated in Fischer and Modersitzki,²⁶ the integral can be viewed as an approximation to the mean curvature of the first and second component of the displacement field $u (x)$ . Thus, the term penalises oscillations. It has a non-trivial kernel containing affine linear transformation where

S LC (C x + B) = 0

for all

C \in ℝ 2 \times 2

and

B \in ℝ 2

. Based on this observation, the linear curvature registration does not require an addition of pre-registration step with affine linear registration in contrast with the conventional registration schemes such as diffusion or linear elastic image registration models. In addition, the proposed model can be extended to the mean and Gaussian curvature registration models.^27–29

As c₁ and c₂ in equation (6) are the average intensity values inside and outside the boundary $φ 0 (x)$ in the reference image, by adopting a level set formulation, $φ 0 (x)$ as in Chan–Vese, we minimise over equation (6) to obtain:

c 1 = \frac{\int R (x) H ε (φ 0 (x + u (x))) d x}{\int H ε (φ 0 (x + u (x))) d x}, c 2 = \frac{\int R (x) (1 - H ε (φ 0 (x + u (x)))) d x}{\int 1 - H ε (φ 0 (x + u (x))) d x} .

(9)

For any given parameter set $λ 1, λ 2, λ 3$ and α, we can compute a numerical solution $u (x)$ of the minimisation problem (6) using two main types of numerical schemes. First, the so-called optimise-then-discretise approach where the resulting Euler–Lagrange equations in the continuous domain is discretise using finite difference method. Second, the so-called discretise-then-optimise approach where the discrete version of the minimisation problem (6) is solved using standard optimisation problem such as steepest descent method. From either of these two approaches, we would obtain a nonlinear system of equations to be solved iteratively to obtain the final solution. We adopt the second approach to solve for $u (x)$ in problem (6) using LBFGS method as our optimisation scheme. Since we are dealing with a large system of unknown, we use multilevel representation of the reference and template images for fast and efficient implementation. The problem in (6) is solve on the coarser level first, before interpolating the solution to next finer level.

The grid points are located at the centre of the cell

Ω h = {x i, j = (x 1, i, x 2, j) = ((i - 0.5) h, (j - 0.5) h) | 1 \leq i, j \leq N}

(10) where the domain

Ω h

is split into N × N cells of size h × h. We shall re-use the notation T, R for discrete images of size N × N. We re-define the solution vector

U = [u 1 u 2] 2 N 2 \times 1, x = [x 1 x 2] 2 N 2 \times 1,

(11) where

u 1 = [u 1, 1, 1 u 1, 2, 1 \dots u 1, N, 1 u 1, 1, 1 \dots u 1, N, 1 u 1, 1, 2 \dots u 1, N, N] T, u 2 = [u 2, 1, 1 u 2, 2, 1 \dots u 2, N, 1 u 2, 1, 1 \dots u 2, N, 1 u 2, 1, 2 \dots u 2, N, N] T

and $x 1, x 2$ are similarly defined.

The discretised form of the functional in (6), by a finite difference method is

min c 1, c 2, U J h (c 1, c 2, U) = λ 1 \sum_{i, j = 1}^{N} | R (x i, j) - c 1 | 2 H ε (φ 0 (x i, j + u (x i, j))) + λ 2 \sum_{i, j = 1}^{N} | R (x i, j) - c 2 | 2 (1 - H ε (φ 0 (x i, j + u (x i, j)))) + λ 3 \sum_{i, j = 1}^{N} (T ((x i, j + u (x i, j)) - R (x i, j))) 2 H ε (φ 0 (x i, j + u (x i, j))) + α \sum_{l = 1}^{2} \sum_{i, j = 1}^{N} (- 4 u l (x i, j) + u l (x i + 1, j) + u l (x i - 1, j) + u l (x i, j + 1) + u l (x i, j - 1)) 2 .

(12)

Here, we are using homogeneous Neumann boundary conditions where

u l (x i, 1) = u l (x i, 2), u l (x 1, j) = u l (x 2, j), u l (x i, N - 1) = u l (x i, N), u l (x N - 1, j) = u l (x N, j), l = 1, 2 .

(13)

Starting with zero initial guess,

U = 0,

(14) we solve

H δ U = - G

(15) for

δ U

and update

U \to U + τ δ U

with τ as the Armijo line search parameter.³⁰ H and G are the Hessian and gradient matrix for the functional

J h

in equation (12) with respect to the displacement vector U . The algorithm for the proposed model is given in Algorithm 1

Algorithm 1

The NJSR model for joint segmentation and registration.

Initialisation:

R, T, α, λ 1, λ 2, λ 3, U = 0, φ 0 (x) .

For $level = Minlevel, \dots, Maxlevel$

Solve registration problem on this level using Quasi-Newton method (Algorithm 2),

U level \to Register (T level, R level, φ_{0}^{level}, U level, 0) .

(16)

If $level < Maxlevel$ , interpolate $U level$ to the next finer level.

End for.

where the multilevel of images of the reference and template images denoted by

T level, R level

using standard coarsening in the implementation. The multilevel representation of the surface

φ 0 (x)

represents the contour Γ of the template image. The coarsest and finest levels of images are denoted by

Minlevel

and Maxlevel, respectively. We start with zero initial guess for the displacement field on the

Minlevel

. After registration on each

level

using Algorithm 2, the deformation field

U level

is interpolated to the next finer level

(level = level + 1)

using bilinear interpolation. These recursive procedures are perform iteratively until we reach

level = Maxlevel

Algorithm 2

The NJSR model on one fixed level.

U level \to Register (T level, R level, φ_{0}^{level}, U level, 0) .

1. For $k = 1, \dots, MAXIT$ .

Update c₁ and c₂ using equation (9).

Solve equation (15) for $δ U level$ and update $U level$ with $U level, 0$ as initial values.

Check convergence criterion, if satisfied exit, else continue.

End for.

Numerical results

We use four sets of images for testing the GV-JSR model and the NJSR model (Algorithm 1) on a variety of images and deformation. To judge the quality of the registration, we calculate the relative reduction of the similarity measure

ɛ = \frac{D SSD (T, R, u (x))}{D SSD (T, R)} .

(17)

In all of the experiments, we do not use the regridding step for fair comparison and the value of the regularisation parameters are chosen such that the minimum value of the determinant of the Jacobian matrix J of the transformation, denoted as $F$

J = [1 + \frac{\partial u 1}{\partial x 1} \frac{\partial u 1}{\partial x 2} \frac{\partial u 2}{\partial x 1} 1 + \frac{\partial u 1}{\partial x 2}], F = min (det (J)),

(18) is greater than zero. This indicates that the deformed grid obtained from the displacement field is free from folding and cracking. Details of the experiments are:

Experiment 1 (Comparison between GV-JSR and NJSR Models for One Feature Object) Experiment 1 consists of two X-ray images of a human hand from Modersitzki¹⁷ to illustrate the type of images where the GV-JSR and NJSR models are able to segment and register. The images in Experiment 1 consist of one object with relatively large structure.

Experiment 2 (Brain MRI with GV-JSR and NJSR Models)

Experiment 2 is used to illustrate that the GV-JSR and NJSR models are capable to solve registration problem using real medical images. We use brain MRI from IBSR⁹ (https://www.nitrc.org/project/ibsr) database to test the models. We choose a pair of brain images from different individuals to test the models.

Experiment 3 (Global Deformation with GV-JSR and NJSR Models using Synthetic Images) The images for the Experiment 3 come from Hömke³¹ where the GV-JSR and NJSR models manage to deliver good results because the features inside the objects in the template image pose the same deformation with the boundary of the object to be segmented.

Experiment 4 (Local Deformation with GV-JSR and NJSR Models)

Experiment 4 is used to illustrate images where the GV-JSR model fails to provide the deformation field between the reference and template images where the data set is from Henn.³² In this experiment, the features inside the contour pose different kinds of deformation with the contour. Since the GV-JSR model is based on the boundary mapping, we obtain no alignment for the features inside the contour Γ. Note that the outer structure is nicely registered whereas the inner structure is poorly registered. We show that our proposed model, NJSR, is able to solve the existing problem Experiment 3, which involves different kinds of deformation for the boundary (contour) of the object and the features inside the contour.

In all experiments, we use $λ 1 = λ 2 = 250, λ = 0.5, μ = 0.005$ for the GV-JSR model in a single level implementation. The parameters values chosen above are based on GV-JSR model¹⁹ for an optimum performance of this model. We solve the GV-JSR model based on the numerical solver provides in Le Guyader and Vese¹⁹ without the regridding step.

Experiment 1: One feature with GV-JSR and NJSR models

Images for Experiment 1 are the same as Modersitzki¹⁷ where X-ray images of two hands of different individuals need to be aligned. The size of the images is 128 × 128 and the recovered transformation is expected to be smooth. For this experiment, we take $α = β = 25$ . We show the results of Experiment 1 obtained by GV-JSR model in Figure 1. The template image and the zero level set of Γ are shown in red in Figure 1(a). The resulting deformation field is shown in Figure 1(d) with the value of $F = 0.4790$ . The zero level of $φ 0 (x + u)$ is shown in red with the reference image in Figure 1(e). The model uses Dirichlet boundary conditions, which explains why the lower part of the hand is not aligned as shown in Figure 1(f) with the value of $ɛ = 0.2343$ . In this experiment, the object inside Γ exhibits the same deformation as Γ, thus the GV-JSR model manages to deliver an acceptable level of results. Our new model, NJSR with $λ 1 = λ 2 = λ 3 = 1$ and $α = 0.5$ also manages to solve Experiment 1 with similar results as in Figure 2.

Figure 1.

Experiment 1: GV-JSR model. Illustration of the type of images where the GV-JSR model delivers good results where the object to be segmented in the template image is relatively large. The results obtained in this experiment are for $α = β = 25$ . (a) T and φ₀ (x); (b) R; (c) T−R; (d) x + u(x), $F$ = 0.4790; (e) R and φ₀ (x + u); (f) T(x + u(x)), ɛ = 0.2343.

Figure 2.

Experiment 1: NJSR model with $λ 1 = λ 2 = λ 3 = 1$ and $α = 0.5$ . (a) x + u(x), $F$ = 0.5200; (b) R and φ₀ (x + u); (c) T(x + u(x)), ɛ = 0.0783.

Experiment 2: Brain MRI with GV-JSR and NJSR models

In Experiment 2, we use the images in Figure 3 to illustrate that the proposed model NJSR is capable to solve real medical images. Here, the size of images are 128 × 128. However, the model is applicable for larger size of images using parallel computing (Figure 4).

Figure 3.

Experiment 2: GV-JSR model. The results obtained in this experiment are for $α = β = 25$ . (a) T and φ₀ (x); (b) R; (c) T − R; (d) x + u(x), $F$ = 0.6839; (e) R and φ₀ (x + u); (f) T(x + u(x)), ɛ = 0.2605.

Figure 4.

Experiment 2: NJSR model. We have better results using the NJSR model for Experiment 2. Here, we are using $λ 1 = λ 2 = λ 3 = 1$ and α = 1. We also have smaller value of $ɛ = 0.1187$ for the NJSR model than ɛ = 0.2605, which is obtained from the GV-JSR model. (a) x + u(x), $F$ = 0.5389; (b) R and φ₀ (x + u); T(x + u(x)), ɛ = 0.1187.

Experiment 3: Global deformation with GV-JSR and NJSR models

Synthetic images for Experiment 2 from Hömke³¹ are used to illustrate cases where the features inside the object have the same deformation as the boundary of the object. The results of Experiment 3 using the GV-JSR model with $α = β = 25$ are shown in Figure 5. The template image and the zero level set of Γ in red are shown in Figure 5(a). The resulting deformation field is shown in Figure 5(d) with $F = 0.7424$ . The zero level set of $φ 0 (x + u)$ is shown in red with the reference image in Figure 5(e). The resulting transformed template image using the deformation in (d) is shown in Figure 5(f) with $ɛ = 0.0518$ . In this problem, the object inside Γ exhibits the same deformation as Γ, thus the GV-JSR model manages to deliver an acceptable level of results.

Figure 5.

Experiment 3: GV-JSR model. Illustration of the second class of problem where the GV-JSR model manages to provide good results where the deformation of the features inside the object to be segmented pose the same deformation with the object itself. (a) T and φ₀ (x); (b) (x + u); (c) T(x + u(x)); (d) x + u(x), $F$ = 0.7424; (e) R and φ₀ (x + u); (f) T(x + u(x)), ɛ = 0.0518.

Our new model, NJSR with $λ 1 = λ 2 = λ 3 = 1$ and $α = 0.5$ also manages to solve this particular experiment with similar results as shown in Figure 6.

Figure 6.

Experiment 3: NJSR model with $λ 1 = λ 2 = λ 3 = 1$ and $α = 0.5$ . (a) x + u(x), $F$ = 0.8372; (b) R and φ₀ (x + u); (c) T(x + u(x)), ɛ = 0.0019.

Experiment 4: Local deformation with GV-JSR and NJSR models

In Experiment 4, we use the images in Figure 7 to illustrate where the GV-JSR model with α = 5 and β = 25 fails to deliver good results. In the figure, we can observe that the deformation inside Γ is different from the deformation of Γ. We can see in Figure 7(f), the resulting transform template image contains a huge difference with the reference image in (b) for the inner squares. However, the model manages to align the outermost square. In the figure, we have $F = 0.3319$ and $ɛ = 0.0509$ .

Figure 7.

Experiment 4: GV-JSR model. Illustration of the type of image, which has different deformation for the boundary Γ and the features inside Γ. The GV-JSR model fails to align the features inside Γ but manages to align the outer most square in the template image. In this experiment, we are using α = 5 and β = 25. (a) T and φ₀ (x); (b) R; (c) T − R; (d) x + u(x), $F$ = 0.3319; (e) R and φ₀ (x + u); (f) T(x + u(x)), ɛ = 0.0509.

We resolve the issues in Experiment 3 by using the NJSR model, and the resulting images are depicted in Figure 8. In this figure, we obtain the segmentation of the reference image as shown in Figure 8(b). Since the NJSR model uses the linear curvature model for registration which contains affine linear transformation, it manages to recover the rotation part of the deformation without affine pre-registration step as shown in Figure 8(a) with $F = 0.3004$ . The resulting transformed template image, shown in Figure 8(c), has better alignment with the reference image in Figure 7(b) compared with the one obtained by the GV-JSR model in Figure 7(f). In this experiment, we have ɛ = 0.0062, which is lower than the one obtain from the GV-JSR model in Figure 7(f).

Figure 8.

Experiment 4: NJSR model. We have better results using the NJSR model for Experiment 3 where the circles in T are deformed to squares as in R. Here, we are using $λ 1 = λ 2 = λ 3 = 1$ and $α = 0.25$ . We also have smaller value of $ɛ = 0.0062$ for the NJSR model than $ɛ = 0.0509$ , which is obtained from the GV-JSR model. (a) x + u(x), $F$ = 0.3004; (b) R and φ₀ (x + u); (c) T(x + u(x)), ɛ = 0.0062.

Conclusion

We have present an improved model for joint segmentation and registration in a variational formulation. The proposed model consists of two new terms, which extend the original Le Guyader and Vese (GV-JSR) model’s applicability. The first term is a weighted SSD with a regularised Heaviside of the zero level set function to quantify the different deformations exhibited by the features inside of the contour of the template image. The second term is the linear curvature term to control the smoothness of the deformation field, which is superior than the non-linear elastic term in the old GV-JSR model.

Future work involves developing an efficient multigrid method to solve the model, analytical justification for the model, and automatic selection of regularisation parameters. While there has been work in parameter selection in registration,³³ further work is required to develop a method for the selection of optimal parameters for regularisation term in the joint segmentation and registration model. In addition, we can further extend the work for selective segmentation method or shape prior segmentation models.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The first author acknowledges the support from the Ministry of Higher Education of Malaysia (KPT(BS)860520465478).

References

Gooya A, Pohl KM, Bilello M, et al. Joint segmentation and deformable registration of brain scans guided by a tumor growth model. In: Medical image computing and computer-assisted intervention–MICCAI 2011, 2011, pp.532–540. Berlin Heidelberg: Springer.

Erdt

Steger

Sakas

. Regmentation: A new view of image segmentation and registration. J Radiat Oncol Inform 2012; 4: 1–23.

Yezzi A, Zollei L and Kapur T. A variational framework for joint segmentation and registration. In: IEEE workshop on mathematical methods in biomedical image analysis, Los Alamitos, California, 2001, pp.44–51.

Chen

Tagare

Thiruvenkadam

. Using prior shapes in geometric active contours in a variational framework. Int J Comput Vis 2002; 50: 315–328.

Paragios N, Rousson M and Ramesh M. Knowledge based registration and segmentation of the left ventricle: A level set approach. In: IEEE workshop on applications of computer vision, Los Alamitos, California, 2002, pp.37–42.

Vemuri B and Chen Y. Joint image registration and segmentation. In: Geometric level set methods in imaging, vision and graphics, 2003, pp.251–269. New York, USA: Springer Verlag.

Wyatt P and Noble J. Map MRF joint segmentation & registration. In: MICCAI, Berlin Heidelberg, 2002, pp.580–587.

Dydenko I, Friboulet D and Magnin I. A variational framework for affine registration and segmentation with shape prior: application in echocardiographic imaging. In: VLSM Workshop-ICCV, Los Alamitos, California, 2003, pp.201–208.

Alvarez

Weickert

Sanchez

. Reliable estimation of dense optical flow fields with large displacements. Int J Comput Vis 2000; 39: 41–56.

10.

Amit

. A nonlinear variational problem for image matching. SIAM J Sci Comput 1994; 15: 207–224.

11.

Unal G and Slabaugh G. Coupled PDEs for non-rigid registration and segmentation. In: IEEE computer society conference on computer vision and pattern recognition, Los Alamitos, California, 2005, pp.168–175.

12.

Wang F and Vemuri BC. Simultaneous registration and segmentation of anatomical structures from brain MRI. Berlin Heidelberg: Springer, 2005, pp.17–25.

13.

Chan T and Vese L. An active contour model without edges. Berlin Heidelberg: Springer, 1999, pp.141–151.

14.

Le Guyader

Vese

. A combined segmentation and registration framework with a nonlinear elasticity smoother. Comput Vis Image Understand 2011; 115: 1689–1709.

15.

Fischer

Modersitzki

. Curvature based image registration. J Math Imag Vis 2003; 18: 81–85.

16.

Fischer B and Modersitzki J. Curvature based registration with applications to MR-mammography. In: Computational science ICCS 2002, 2002, pp.202–206. Berlin Heidelberg: Springer.

17.

Modersitzki

. Flexible algorithms for image registration, Philadelphia, USA: SIAM Publications, 2009.

18.

Modersitzki

. Numerical methods for image registration, Oxford, UK: Oxford University Press, 2004.

19.

Le Guyader C and Vese L. A combined segmentation and registration framework with a nonlinear elasticity smoother, 2009, pp.600–611. Berlin, Heidelberg: Springer.

20.

Yanovsky I, Thompson PM, Osher S, et al. Large deformation unbiased diffeomorphic nonlinear image registration: Theory and implementation. UCLA CAM Report 06-71, Los Angeles, California, 2006.

21.

Yanovsky I, Le Guyader C, Leow A, et al. Unbiased volumetric registration via nonlinear elastic regularization. In: 2nd MICCAI workshop on mathematical foundations of computational anatomy, New York, USA, 2008, pp.1–12.

22.

Lin

Le Guyader

Dinov

. Gene expression data to mouse atlas registration using a nonlinear elasticity smoother and landmark points constraints. J Sci Comput 2012; 50: 586–609.

23.

Christensen

Rabbitt

Miller

. Deformable templates using large deformation kinematics. IEEE T Med Imag 1996; 5: 1435–1447.

24.

Cahill ND, Noble JA and Hawkes DJ. Fourier methods for nonparametric image registration. In: IEEE conference on computer vision and pattern recognition, 2007, pp.1–8.

25.

Schumacher H, Franz A and Fischer B. Weighted medical image registration with automatic mask generation. In: Medical imaging, 2006, pp.61442B–61442B. Bellingham, Washington: International Society for Optics and Photonics.

26.

Fischer

Modersitzki

. A unified approach to fast image registration and a new curvature based registration technique. Linear Algebra Appl 2004; 380: 107–124.

27.

Noppadol Chumchob

. Improved variational image registration model and a fast algorithm for its numerical approximation. Numer Meth Part D E 2011; 28: 1966–1995.

28.

Ibrahim

Chen

Brito-Loeza

. A novel variational model for image registration using Gaussian curvature. J Geom Imag Comput 2014; 1: 417–446.

29.

Chumchob

Chen

Brito-Loeza

. A fourth-order variational image registration model and its fast multigrid algorithm. Multiscale Model Simul 2011; 9: 89–128.

30.

Wright

Nocedal

. Numerical optimization, volume 2, New York: Springer, 1999.

31.

Hömke

. A multigrid method for anisotropic PDEs in elastic image registration. Numer Linear Algebra Appl 2006; 13: 215–229.

32.

Henn

. A translation and rotation invariant Gauss Newton like scheme for image registration. BIT 2006; 46: 325–344.

33.

Larrey-Ruiz J and Morales-Sánchez J. Optimal parameters selection for non-parametric image registration methods. In: Advanced concepts for intelligent vision systems, 2006, pp.564–575. Berlin Heidelberg: Springer.

An improved model for joint segmentation and registration based on linear curvature smoother

Abstract

Keywords

Introduction

Relation to previous work: The GV-JSR model

The proposed NJSR model

Algorithm 1

Algorithm 2

Numerical results

Experiment 1: One feature with GV-JSR and NJSR models

Experiment 2: Brain MRI with GV-JSR and NJSR models

Experiment 3: Global deformation with GV-JSR and NJSR models

Experiment 4: Local deformation with GV-JSR and NJSR models

Conclusion

Footnotes

Declaration of conflicting interests

Funding

References