Sage Journals: Discover world-class research

Abstract

The purpose of this study is to prove the convergence of the simultaneous estimation of the optical flow and object state (SEOS) method. The SEOS method utilizes dynamic object parameter information when calculating optical flow in tracking a moving object within a video stream. Optical flow estimation for the SEOS method requires the minimization of an error function containing the object's physical parameter data. When this function is discretized, the Euler-Lagrange equations form a system of linear equations. The system is arranged such that its property matrix is positive definite symmetric, proving the convergence of the Gauss-Seidel iterative methods. The system of linear equations produced by SEOS can alternatively be resolved by Jacobi iterative schemes. The positive definite symmetric property is not sufficient for Jacobi convergence. The convergence of SEOS for a block diagonal Jacobi is proved by analysing the Euclidean norm of the Jacobi matrix. In this paper, we also investigate the use of SEOS for tracking individual objects within a video sequence. The illustrations provided show the effectiveness of SEOS for localizing objects within a video sequence and generating optical flow results.

Keywords

Optical Flow Simultaneous Estimation Tracking

1. Introduction

Computer vision-based motion analysis has been the subject for numerous research activities [1 –15] for several decades. Optical flow- and feature-based techniques can be identified as the two most popular approaches employed in estimating motion parameters in a video stream. Feature-based approaches rely on the feature extraction of the moving object and the tracing of correspondence points to capture the motion, whereas in the optical flow approach, object motion is represented as sampled velocity fields in the image plane. In this study, we consider an optical flow-based approach to determine the object's motion relative to the camera frame of reference.

Both optical flow- and feature-based techniques have been used with the dynamic modelling of the objects involved for the purpose of parameter estimation [16]. Broida et al. [17] modelled the object's motion by retaining an arbitrary number of terms in the appropriate Taylor series, while the neglected terms of the series were modelled as process noise. Recursive estimation can be performed with an iterative extended Kalman filter (IEKF). Roach and Aggarwal [18] introduced a system of nonlinear equations that relate the relative position of the object with respect to the camera position, with solutions being generated by numerical techniques. This also draws attention (also in [19]) to the fact that most existing techniques perform poorly when the images (coordinates of matched points) are noisy. More recently, Blostein et al. [20] demonstrated the feasibility and significance of an optical flow-based approach over a feature-based approach while applying an extended Kalman filter to an object motion model with a constant velocity.

The process of determining optical flow is generally carried out by applying a brightness constancy constraint equation (BCCE), which makes use of spatio-temporal derivatives of the image intensity [1, 2]. Determining optical flow using the BCCE is an ill-posed problem. This arises from the fact that when a straight moving edge is viewed through a narrow aperture, the only motion component that can be determined is perpendicular to that edge [21]. There are several methods to overcome the ill-posedness of differential techniques, two of which are the Lucas-Kanade [1] and Horn-Schunck [2] techniques. Methods utilizing the BCCE are differential techniques, which can be classified as either local or global. Local techniques involve the optimization of a local energy functional, as in the Lucas-Kanade method [1] or the frequency-based minimization methods [22] and [23]. The global category refers to methods that determine optical flow through the minimization of a global energy functional [2], and numerous other discontinuity-preserving approaches [24 –30]. Differential techniques are widely used due to their high level of performance [31]. Local methods offer robustness to noise, but lack the ability to produce dense optical flow fields. Global techniques provide 100 percent dense flow fields, but exhibit much greater sensitivity to noise [31, 32].

Motion parameter estimation and optical flow calculation are generally considered to be separate problems. In this paper, we focus on a unified approach to image-object localization coupled with 3D parameter estimation of a moving target. This unified approach is known as the simultaneous estimation of optical flow and object state (SEOS) [9, 16]. In SEOS, a more refined and robust approach will be employed for image-object localization coupled with the parameter estimation of a moving target. The nonlinear relationship between the motion parameters of the object and its projected image on the video sequence is used to simultaneously estimate the optical flow as well as the motion parameters of the object in question. Uncertainty exists in vision-based systems with respect to the initial conditions and also in image position localization [19]. To deal with this issue, a robust extended Kalman filtering (REKF) technique is implemented [33, 34]. This robust version of the extended Kalman filter takes into account large uncertainties and errors in the measurements and the system model. The performance that is attainable using SEOS is investigated for real image sequences. The use of SEOS for target localization and tracking applications is investigated. SEOS is used to track specific objects within image sequences that are generated from a stationary mounted camera. An optical flow equation-based segmentation technique is used to locate the object of interest within the video stream. These optical flow results are compared and contrasted with the well-established Horn-Schunck method [2].

This paper is organized as follows. Firstly, in Section 2, motion models and measurement equations for the simultaneous estimation problem are formulated. In Section 3, we outline a procedure for the set-value state estimation of a nonlinear signal model. In Section 4, the SEOS cost function is stated and iterative expressions to the solution (the Gauss-Seidel and Jacobi methods) are analysed for the convergence. Section 5 contains optical flow and state estimation results for both the SEOS and the ‘Horn and Schunck’ techniques. SEOS performance improvements (over ‘Horn and Schunck') on real-image data are shown. Section 6 contains the conclusion.

2. Dynamic Model and Monocular Vision Measurement

2.1 Dynamic Model

Consider a fixed camera and a mobile target (object) within the image plane of the camera, having unknown state dynamics (x ∊ 葷⁶). Here, $x = [x, \dot{x}]^{T}$ , in which x ∊ 葷³ represents the displacement in 3D space and $\dot{x} \in 葷^{3}$ represents the velocity in 3D space. Accordingly, the continuous time dynamic model of the object can be defined as:

\dot{x} = L x + B w_{a},

(1)

where:

L = [\begin{matrix} 0_{3} & I_{3} \\ 0_{3} & 0_{3} \end{matrix}], and B = [\begin{matrix} 0_{3} \\ I_{3} \end{matrix}] .

In (1), w_a(t) ∊ 葷³ is the unknown deterministic object manoeuvre at time t. Using (1), state estimates can be determined through the use of refined estimates of optical flow. This standard form of dynamical system modelling can be used in the implementation of REKF [35] for state estimation. The details associated with 3D physical parameter estimation are outlined in Section 3, with the implementation of an extended Kalman filter [36, 37].

2.2 Perspective Projection

Let $y^{i} = [\begin{matrix} y_{1}^{i} & y_{2}^{i} \end{matrix}]^{T}$ denote the measurement coordinates of the landmark (object) i on the image plane of the camera (see Figure 1). According to the principles of perspective projection [38], yⁱ can be written as:

y^{i} = g (x^{i}) + æ

(2)

where $g (x^{i}) = {[\begin{matrix} \frac{\hat{f} x_{1}^{i}}{x_{3}^{i}} & \frac{\hat{f} x_{2}^{i}}{x_{3}^{i}} \end{matrix}]}^{T}$ , in which $\hat{f}$ represents the camera focal length and æ = [ρ₁ ρ₂]^T represents the measurement noise. From the above (2), the flow field velocity can be derived as:

f_{1} u + f_{2} v = 0,

(3)

where $f_{1} = ({\dot{x}}_{3}^{i} x_{2}^{i} - {\dot{x}}_{2}^{i} x_{3}^{i})$ , $f_{2} = ({\dot{x}}_{1}^{i} x_{3}^{i} - {\dot{x}}_{3}^{i} x_{1}^{i})$ , $u = \frac{d y_{1}^{i}}{d t}$ and $v = \frac{d y_{2}^{i}}{d t}$ .

Figure 1.

Object i $[x_{1}^{i}, x_{3}^{i}, x_{3}^{i}]$ is projected into the image plane as $[y_{1}^{i}, y_{2}^{i}]$ in the image coordinate system [y₁, y₂] with the camera centred around the origin of the camera coordinate system [x₁, x₂, x₃]

Distances are measured in relation to the camera reference frame. The pixel coordinates of an image point are linked to the camera reference frame coordinates by the camera's intrinsic properties [39]. These properties represent the optical and geometric characteristics of the camera. We assume that the image coordinates have their origin at the point where the optical axis intersects the image plane. This intersection point is commonly referred to as the principle point.

3. Set-Value State Estimation with a Nonlinear Signal Model

Consider a nonlinear uncertain system of the form:

\begin{array}{l} \dot{z} = F (z, h) + B_{2} w \\ p = K (z, h) \\ \hat{D} = D (z) + μ, \end{array}

(4)

defined on the finite time interval [0, s], as a general form of the system given by Equation 1. Here, z(t) ∊ 葷ⁿ denotes the ‘state’ of the system, $\hat{D} (t) \in 葷^{l}$ is the ‘measured output’ and p(t) ∊ 葷^q is the ‘uncertainty output'. The uncertainty inputs are w(t) ∊ 葷^p and μ(t) ∊ 葷^l. Furthermore, h(t) ∊ 葷^m is the known ‘control input'. It is assumed that all of the functions appearing in (4) possess continuous and bounded partial derivatives. Additionally, assume that K(z,h) is bounded. This assumption simplifies the mathematical derivations, but can be removed in practice [34]. The matrix B₂ is assumed to be independent of z, and is of full rank.

The uncertainty in the system is defined by the following nonlinear integral constraint [33,34,40,41]:

Φ (z (0)) + \int_{0}^{s} L_{1} (w (t), μ (t)) d t \leq d + \int_{0}^{s} L_{2} (p (t)) d t,

(5)

where d ≥ 0 is a positive real number. Here, Φ, L₁ and L₂ are bounded non-negative functions with continuous partial derivatives satisfying growth conditions of the type:

‖ ϕ (z) - ϕ (z^{T}) ‖ \leq β (1 + ‖ z ‖ + ‖ z^{T} ‖) ‖ z - z^{T} ‖

(6)

where || · || is the Euclidean norm with β > 0 and ϕ = Φ, L₁, L₂. The uncertainty inputs w(·), μ(·) satisfying this condition are called ‘admissible uncertainties'. We consider the problem of characterizing the set of all possible states $Z_{s}$ of the system (4) at time s ≥ 0 which are consistent with an uncertain initial control input h⁰ of a given output path ${\hat{D}}^{0} (\cdot)$ , i.e., $z \in Z_{s}$ if, and only if, there exists admissible uncertainties such that if h⁰ (t) is the control input and z(·) and $\hat{D} (\cdot)$ are resulting trajectories, then z(s) = z and $\hat{D} (t) = {\hat{D}}^{0} (t)$ , for all 0 ≤ t ≤ s.

3.1 A Robust Extended Kalman Filter

Petersen and Savkin in [34] presented a characterization of the set $Z_{s}$ as an extended Kalman filter version of the solution to the set-value state estimation problem for a linear plant with the uncertainty described by an integral quadratic constraint (IQC). This IQC is also presented as a special case of Equation 5. We consider the uncertain system described by (4) and an integral quadratic constraint of the form:

\begin{array}{l} {(z (0) - z_{0})}^{T} Y_{0} (z (0) - z_{0}) \\ + \frac{1}{2} \int_{0}^{s} (w {(t)}^{T} Q (t) w (t)) + μ {(t)}^{T} R (t) μ (t) d t \\ \leq d + \frac{1}{2} \int_{0}^{s} p {(t)}^{T} p (t) d t . \end{array}

(7)

where N > 0, Q > 0 and R > 0. For the system (4), (7), the REKF generalization presented in [34] can be written as:

\begin{array}{l} \dot{\tilde{z}} (t) = F (\tilde{z} (t), h^{0}) + Y^{- 1} [\nabla_{z} D {(\tilde{z} (t))}^{T} R ({\hat{D}}^{0} - D (\tilde{z} (t))) \\ + \nabla_{z} K {(\tilde{z} (t), h^{0})}^{T} K (\tilde{z} (t), h^{0})], \end{array}

(8)

where $\tilde{z} (t) = z_{0}$ . Y(t) is defined as the solution to the Riccati differential equation (RDE):

\begin{array}{l} \dot{Y} + \nabla_{z} F {(\tilde{z}, h^{0})}^{T} Y + Y \nabla_{z} F (\tilde{z}, h^{0}) \\ + Y B_{2} Q^{- 1} B_{2}^{T} Y - \nabla_{z} D {(\tilde{z})}^{T} R \nabla_{z} D (\tilde{z}) \\ + \nabla_{z} K {(\tilde{z}, h^{0})}^{T} \nabla_{z} K (\tilde{z}, h^{0}) = 0, Y (0) = N . \end{array}

(9)

An approximate formula for the set $Z_{s}$ is given by:

{\tilde{Z}}_{s} = {z \in 葷^{n} : \frac{1}{2} {(z - \tilde{z} (s))}^{T} Y (s) (z - \tilde{z} (s)) \leq d - ϕ (s)},

where:

\begin{array}{c} ϕ (t) ≜ \frac{1}{2} \int_{0}^{t} [{({\hat{D}}^{0} - D (\tilde{z}))}^{T} R ({\hat{D}}^{0} - D (\tilde{z})) \\ - K {(\tilde{z}, h^{0})}^{T} K (\tilde{z}, h^{0})] d τ . \end{array}

(10)

In the application of the robust extended Kalman filter (REKF) in video-based tracking, the object system is represented during the time interval by the nonlinear uncertain system in (1) and the IQC given in (7), where Q > 0,R > 0 and N > 0 are the weighting matrices (with appropriate dimensions) for the system in each case. The initial state (z₀) is the estimated state of the respective systems at the initial time. With an uncertainty relationship in the form of (7), the inherent measurement noise (2), unknown target object acceleration/driving command and the uncertainty in the initial condition are considered as bounded deterministic uncertain inputs. In particular, the measurement equation (Equation 2) with the standard norm-bounded uncertainty can be written as:

\hat{D} = D (z) + δ D (z) + μ_{0}

where |δ| ≤ ξ with ξ is a constant indicating the upper bound of the norm-bounded portion of the noise. By choosing p = ξD(z) and ν = δD(x):

\int_{0}^{s} | ν | d t \leq \int_{0}^{s} p^{T} p d t .

Considering ν₀ and the corresponding uncertainty in w as w₀ satisfying the bound in the form of:

Φ (z (0)) + \int_{0}^{s} [w_{0}^{T} Q w_{0} + μ_{0}^{T} R μ_{0}] d t \leq d,

it is clear that this uncertain system leads to the satisfaction of the inequality in (5) and, hence, the constraint in (7) is satisfied (see [34]). This more realistic approach removes any noise model assumptions in the algorithm development and guarantees the robustness of the solution.

3.2 Robust versus Optimal State Estimation

The REKF seeks to increase the robustness of the state estimation process and reduce the chance that a small deviation from the Gaussian process in the system noise causes a significant negative impact on the solution. However, we will lose optimality and our solution will be only sub-optimal. To explain the connection between REKF and the standard extended Kalman filter, consider the system (4) with:

K (z, h) = ν K_{0} (z, h)

(11)

where K₀ (z, h) is some bounded function and ν > 0 is a parameter. Thus, the REKF estimate $\tilde{z} (t)$ for the system (4), (11) and (7) defined by (8) and (9) converges to ${\tilde{z}}^{0} (t)$ as ν tends to 0. Here, ${\tilde{z}}^{0} (t)$ is the extended Kalman state estimate for the system (4) with the Gaussian noise [w(t)^T μ(t)^T] satisfying

E {[\begin{array}{l} w (t) \\ μ (t) \end{array}] [\begin{array}{l} w {(t)}^{T} & μ {(t)}^{T} \end{array}]} = [\begin{array}{l} Q (t) & 0 \\ 0 & R (t) \end{array}];

see, e.g., [42]. The parameter ν in (11) describes the size of uncertainty in the system and measurement noise. For small ν, our robust state estimate becomes close to the Kalman state estimate with Gaussian noise; for larger ν we achieve more robustness but less optimality. Hence, there is always some trade-off between robustness and optimality.

4. Numerical Methods in Solving Optical Flow

Optical flow is the apparent motion of the brightness/intensity patterns observed when there is relative motion between a camera (observer) and the objects being imaged; [38, 43]. Let I(y₁, y₂, t) denote the image intensity function at time t at the image point (y₁, y₂). Assuming that the overall intensity of the image is time-independent, the BCCE equation can be written as:

I_{1} \tilde{u} + I_{2} \tilde{v} + I_{t} = 0,

(12)

where $I_{1} = \frac{\partial I}{\partial y_{1}}$ , $I_{2} = \frac{\partial I}{\partial y_{2}}$ , $I_{t} = \frac{\partial I}{\partial t}$ , $\tilde{u} = \frac{d y_{1}}{d t}$ and $\tilde{v} = \frac{d y_{2}}{d t}$ . Consider the functional for minimization:

\begin{array}{l} F (\tilde{u}, \tilde{v}) = \frac{1}{2} \iint {(I_{1} \tilde{u} + I_{2} \tilde{v} + I_{t})}^{2} \\ + P E (- α) {(f_{1} \tilde{u} + f_{2} \tilde{v})}^{2} \frac{M}{2} (E (α) + 1) \\ ({‖ \nabla \tilde{u} ‖}^{2} + {‖ \nabla \tilde{v} ‖}^{2}) d y_{1} d y_{2}, \end{array}

(13)

where $E (α) = e^{α ((y_{1} - {\bar{y}}_{1})^{2} + {(y_{2} - {\bar{y}}_{2})}^{2})}$ , P is a weighting coefficient for object state parameters, M sets the weighting for overall image smoothing, and α adjusts the smoothness for regions within the object-specified area. In (13), the optical flow is assumed to be equal to the object motion field [2]; hence, $u = \tilde{u}$ and $v = \tilde{v}$ . Outside the image-focused region, (u, ν) = 0. The image plane coordinates ${\bar{y}}_{1}$ and ${\bar{y}}_{2}$ represent the coordinates for the centroid of the tracked object.

The following two conditions are obtained from the function (13) using the calculus of variations [2]:

\begin{array}{l} [I_{1}^{2} + P f_{1}^{2} E (- α)] \tilde{u} + [I_{1} I_{2} + P f_{1} f_{2} E (- α)] \tilde{v} \\ = \frac{M}{2} (E (α) + 1) \nabla^{2} \tilde{u} + M α U E (α) - I_{1} I_{t} \end{array}

(14)

\begin{array}{l} [I_{1} I_{2} + P f_{1} f_{2} E (- α)] \tilde{u} + [I_{2}^{2} + P f_{2}^{2} E (- α)] \tilde{v} \\ = \frac{M}{2} (E (α) + 1) \nabla^{2} \tilde{v} + M α V E (α) - I_{2} I_{t} \end{array}

(15)

where $U = [(y_{1}^{i} - {\bar{y}}_{1}) {\tilde{u}}_{1} + (y_{1}^{i} - {\bar{y}}_{1}) {\tilde{u}}_{2}]$ and $V = [(y_{1}^{i} - {\bar{y}}_{1}) {\tilde{v}}_{1} + (y_{2}^{i} - {\bar{y}}_{2}) {\tilde{v}}_{2}]$ , in which ${\tilde{u}}_{1} = \frac{\partial \tilde{u}}{\partial y_{1}}$ , ${\tilde{v}}_{1} = \frac{\partial \tilde{v}}{\partial y_{1}}$ , ${\tilde{u}}_{2} = \frac{\partial \tilde{u}}{\partial y_{2}}$ and ${\tilde{v}}_{2} = \frac{\partial \tilde{v}}{\partial y_{2}}$ . Approximating the Laplacians using five-point central differences gives:

\begin{array}{l} \nabla^{2} \tilde{u} \approx \sum_{j \in N_{i}} ({\tilde{u}}_{j} - {\tilde{u}}_{i}) \\ \nabla^{2} \tilde{v} \approx \sum_{j \in N_{i}} ({\tilde{v}}_{j} - {\tilde{v}}_{i}) \end{array}

for all grid point indices i ∊ (1,.., N_t). The procedure behind numerically approximating the Laplacian is outlined in Appendix A. From (14) and (15), we have:

\begin{array}{l} [I_{1_{i}}^{2} + P f_{1}^{2} E (- α) + \frac{1}{2} M c_{i} (E (α) + 1)] {\tilde{u}}_{i} \\ + [I_{1_{i}} I_{1_{i}} + P f_{1} f_{2} E (- α)] {\tilde{v}}_{i} \\ = \frac{1}{2} M (E (α) + 1) \sum_{j \in N_{i}} {\tilde{u}}_{j} + M α U E (α) - I_{1_{i}} I_{t i} \end{array}

(16)

and:

\begin{array}{l} [I_{1 i} I_{1_{i}} + P f_{1} f_{2} E (- α)] {\tilde{u}}_{i} \\ + [I_{1_{i}}^{2} + P f_{2}^{2} E (- α) + \frac{1}{2} M c_{i} (E (α) + 1)] {\tilde{v}}_{i} \\ = \frac{1}{2} M (E (α) + 1) \sum_{j \in N_{i}} {\tilde{v}}_{j} + M α V E (α) - I_{1_{i}} I_{t i}, \end{array}

(17)

where N_i is a set of neighbourhood indices of i, and c_i is a proportionality constant related to the neighbours of i.

4.1 SEOS Convergence with the Gauss-Seidel Iterative Scheme

Various numerical methods have been suggested [44] to solve the large-scale system of equations represented by (16) and (17). One such iterative technique is the Gauss-Seidel method [45]. The Gauss-Seidel iterative scheme exhibits a fast rate of convergence due to its ability to use refined estimates within a particular iteration as they become available. A Gauss-Seidel iterative solution scheme for u and v with an iteration number n is:

\begin{array}{l} {\tilde{u}}_{i}^{n + 1} = \frac{A}{Ω} (\sum_{j \in N_{i}; j < i} {\tilde{u}}_{j}^{n + 1} + \sum_{j \in N_{i}; j > i} {\tilde{u}}_{j}^{n}) \\ + \frac{B}{Ω} (\sum_{j \in N_{i}; j < i} {\tilde{v}}_{j}^{n + 1} + \sum_{j \in N_{i}; j > i} {\tilde{v}}_{j}^{n}) + \frac{C}{Ω} \\ {\tilde{v}}_{i}^{n + 1} = \frac{B}{Ω} (\sum_{j \in N_{i}; j < i} {\tilde{u}}_{j}^{n + 1} + \sum_{j \in N_{i}; j > i} {\tilde{u}}_{j}^{n}) \\ + \frac{Ε}{Ω} (\sum_{j \in N_{i}; j < i} {\tilde{v}}_{j}^{n + 1} + \sum_{j \in N_{i}; j > i} {\tilde{v}}_{j}^{n}) + \frac{F}{Ω}, \end{array}

(18)

where:

\begin{array}{c} A = M^{2} c_{i} {[E (α) + 1]}^{2} + 2 M [E (- α) + 1] [I_{2_{i}}^{2} E (α) + P f_{2}^{2}] \\ B = - 2 M I_{1_{i}} I_{2_{i}} [E (α) + 1] - 2 P M f_{1} f_{2} [E (- α) + 1] \\ \begin{array}{l} C = 2 M^{2} α c_{i} U [E (α) + E (2 α)] + 4 M α E (α) [U I_{2_{i}}^{2} - V I_{1_{i}} I_{2_{i}}] \\ - 2 M c_{i} I_{1_{i}} I_{t i} [E (α) + 1] + 4 P E (- α) f_{2} I_{t i} [f_{1} I_{2_{i}} - f_{2} I_{1_{i}}] \\ - 4 M P α f_{2} [f_{1} V - f_{2} U] \end{array} \\ E = M^{2} c_{i} {[E (α) + 1]}^{2} + 2 M [E (- α) + 1] [I_{1_{i}}^{2} E (α) + P f_{1}^{2}] \\ \begin{array}{l} F = 2 M^{2} α c_{i} V [E (α) + E (2 α)] + 4 M α E (α) [V I_{1_{i}}^{2} - U I_{1_{i}} I_{2_{i}}] \\ - 2 M c_{i} I_{2_{i}} I_{t i} [E α + 1] + 4 P E (- α) f_{1} I_{t i} [f_{2} I_{1_{i}} - f_{1} I_{2_{i}}] \\ - 4 M P α f_{2} [f_{1} U - f_{2} V] \end{array} \\ \begin{array}{l} Ω = M^{2} c_{i}^{2} {[E (α) + 1]}^{2} + 2 M c_{i} {(I_{1_{i}}^{2} + I_{2_{i}}^{2}) [E (α) + 1] \\ + 2 c_{i} P (f_{1}^{2} + f_{2}^{2}) [E (- α) + 1]} \\ + 4 P E (- α) {[I_{1_{i}} f_{2} - I_{2_{i}} f_{1}]}^{2} . \end{array} \end{array}

In order to examine the convergence of the Simultaneous Estimation of Optical Flow and Object State (SEOS) method, the cost functions (14) and (15) are represented as a large scale system (S) of linear equations by rearranging equations (16) and (17), followed by the application of sufficient convergence criteria. A sufficient condition of convergence will be shown, identifying that the SEOS solutions for a Gauss-Seidel iterative scheme will converge for any arbitrary choice of initial approximation. The following system (S) of linear equations represents discrete approximations of the SEOS cost function for all grid point indices i ∊ {1,…, N_t}.

S = {\begin{array}{l} [I_{1_{i}}^{2} + P f_{1}^{2} E (- α) + \frac{1}{2} M c_{i} (E (α) + 1)] {\tilde{u}}_{i} \\ + [I_{1_{i}} I_{2_{i}} + P f_{1} f_{2} E (- α)] {\tilde{v}}_{i} \\ - M α U E (α) - \frac{1}{2} M (E (α) + 1) \sum_{j \in N_{i}} ({\tilde{u}}_{j}) \\ = - I_{\tilde{x} i} I_{t i} \\ [I_{1_{i}} I_{2_{i}} + P f_{1} f_{2} E (- α)] {\tilde{u}}_{i} \\ + [I_{2_{i}}^{2} + P f_{2}^{2} E (- α) + \frac{1}{2} M c_{i} (E (α) + 1)] {\tilde{v}}_{i} \\ - \frac{1}{2} M (E (α) + 1) \sum_{\begin{matrix} j \in N_{i} \end{matrix}} ({\tilde{v}}_{j}) - M α V E (α) \\ = - I_{2_{i}} I_{t i} \end{array}

Large scale systems are most effectively solved by iterative techniques [46]. We express the system (S) of linear equations in matrix form as

A_{r} r_{n} = b_{r},

(19)

where $r_{n} = [{\tilde{u}}_{i} {\tilde{v}}_{i}]^{T}$ . Assuming that matrix A_r is non-singular, the goal is to solve (19) with an efficient and sufficient method of convergence. A sufficient condition for the convergence of Gauss-Seidel iterative scheme for any initial approximation is shown by illustrating that the coefficient matrix A_r is both symmetric and positive definite. Hence, we verify that $r_{n}^{T} A_{r} r_{n} > 0$ for all r_n ∊ 葷^2N; r_n ≠ 0. Simplified format of $r_{n}^{T} A_{r} r_{n}$ can be written as;

\begin{array}{l} r_{n}^{T} A_{r} r_{n} = \sum_{i = 1}^{N_{t}} {(I_{1_{i}} {\tilde{u}}_{i} + I_{2_{i}} {\tilde{v}}_{i})}^{2} \\ + {(\sqrt{P E (- α)} f_{1} {\tilde{u}}_{i} + \sqrt{P E (- α)} f_{2} {\tilde{v}}_{i})}^{2} \\ + W \sum_{i = 1}^{N_{t}} \sum_{j \in N_{i}; j > i} {{({\tilde{u}}_{i} - {\tilde{u}}_{j})}^{2} + {({\tilde{v}}_{i} - {\tilde{v}}_{j})}^{2}} \\ + [M α E (α)] {({\tilde{u}}_{1} {\tilde{u}}_{i} + {\tilde{v}}_{1} {\tilde{v}}_{i}) ({\bar{y}}_{1} - y_{1}) \\ + ({\tilde{u}}_{2} {\tilde{u}}_{i} + {\tilde{v}}_{2} {\tilde{v}}_{i}) ({\bar{y}}_{2} - y_{2})}, \end{array}

(20)

where $W = \frac{M}{2} (E (α) + 1)$ .

In order to verify that matrix A_r is positive definite, $r_{n}^{T} A_{r} r_{n} > 0$ for all non-zero vectors r. Looking at the first sum in (20), the terms are only zero if the optical flow vector is orthogonal to the spatial intensity gradient at all points in the image domain. In this particular case, the optical flow vector is in the direction of the image intensity curve at every point. The second sum will only be zero when the object movement is of the exponential form e^kt/k in relation to all axes. This exponential movement e^kt/k is derived by setting the second sum equal to zero and solving the differential equation. This simultaneous movement is highly unlikely ever to occur along the real-world object coordinates x₁, x₂ and x₃. Provided there is movement of the object in relation to the camera axes, {f₁, f₂} > 0, the second term will be greater than zero for r > 0. The terms in the third sum will only be zero when the optical flow vectors are constant across the entire domain. The last sum will be zero when the tracker reaches the location of the object of interest within an image frame. The value of the smoothing parameter α is selected such that $r_{n}^{T} A_{r} r_{n} > 0$ . The weighting variable M can be set small to limit the effect of object smoothness adjustment. Smoothing is decreased within the object region, increasing the sharpness of optical flow vectors. In areas outside the object, smoothing is increased, resulting in uniform flow outside the object area.

Provided the object is not moving in the form of e^kt/k, and providing the optical flow is not constant over the entire image domain, matrix A_r is positive definite symmetric. Hence, the pointwise and blockwise Gauss-Seidel and relaxation iterative methods for solving the linear system of equations converge [44].

4.2 SEOS Convergence with the Jacobi Iterative Scheme

It was previously shown in section 4.1 that matrix A_r is positive definite symmetric. This is not a sufficient condition for the convergence of the Jacobi iterative solution [47]. It will be shown that the solution to SEOS with a Jacobi iterative scheme converges for an arbitrary choice of initial approximation. A block Jacobi iterative solution scheme for SEOS is given by:

{\tilde{u}}_{i}^{n + 1} = \frac{A}{Ω} \sum_{j \in N_{i}} {\tilde{u}}_{j}^{n} + \frac{B}{Ω} \sum_{j \in N_{i}} {\tilde{v}}_{j}^{n} + \frac{C}{Ω}

(21)

{\tilde{v}}_{i}^{n + 1} = \frac{B}{Ω} \sum_{j \in N_{i}} {\tilde{u}}_{j}^{n} + \frac{E}{Ω} \sum_{j \in N_{i}} {\tilde{v}}_{j}^{n} + \frac{F}{Ω}

(22)

Equations (21) and (22) can be written in matrix form as:

z_{m}^{n + 1} = B_{m} z_{m}^{n} + c_{m}

(23)

where z_m ∊ 葷^2N with elements ${\tilde{u}}_{i} = z_{m : 2 i - 1}$ and ${\tilde{v}}_{i} = z_{m : 2 i}$ ; i ∊ {1,..,N_t}. Matrix B_m contains the following elements1: $B_{m : 2 i - 1,2 j - 1} = \frac{A}{Ω}$ , $B_{m : 2 i - 1,2 j} = B_{m : 2 i,2 j - 1} = \frac{B}{Ω}$ and $B_{m : 2 i,2 j} = \frac{E}{Ω}$ , with all other matrix entries equal to 0. Vector c_m has the elements $c_{m : 2 i - 1} = \frac{C}{Ω}$ and $c_{m : 2 i} = \frac{F}{Ω}$ .

Consider the following vector and matrix norms:

\begin{array}{c} z_{m} \mapsto ‖ z_{m} ‖ = \max_{1 \leq i \leq N_{t}} {(z_{m : 2 i - 1}^{2} + z_{m : 2 i}^{2})}^{\frac{1}{2}}, \\ ‖ B_{m} ‖ = \max_{1 \leq i \leq N_{t}} {({(\sum_{j = 1}^{2 N_{t}} B_{m : (2 i - 1) j})}^{2} + {(\sum_{j = 1}^{2 N_{t}} B_{m : (2 i) j})}^{2})}^{\frac{1}{2}} . \end{array}

The following inequality satisfies the Euclidean norm in: 葷^2N.

‖ B_{m} z_{m} ‖ \leq ‖ B_{m} ‖ ‖ z_{m} ‖

(24)

Hence:

\begin{array}{c} ‖ B_{m} z_{m} ‖ = \\ \max_{1 \leq i \leq N_{t}} {({(\sum_{j = 1}^{2 N_{t}} B_{m : (2 i - 1) j} z_{m : j})}^{2} + {(\sum_{j = 1}^{2 N_{t}} B_{m : (2 i) j} z_{m : j})}^{2})}^{\frac{1}{2}} \\ \leq \max_{1 \leq i \leq N_{t}} {(\sum_{j = 1}^{2 N_{t}} B_{m : (2 i - 1) j}^{2} + \sum_{j = 1}^{2 N_{t}} B_{m : (2 i) j}^{2})}^{1 / 2} {(\sum_{j = 1}^{2 N_{t}} z_{m : j}^{2})}^{\frac{1}{2}} \end{array}

Introducing the neighbourhood weighting factor c_i, we have:

\begin{array}{r} ‖ B_{m} z_{m} ‖ \leq \max_{1 \leq i \leq N_{t}} {(\sum_{j = 1}^{2 N_{t}} B_{m : (2 i - 1) j}^{2} + \sum_{j = 1}^{2 N_{t}} B_{m : (2 i) j}^{2})}^{\frac{1}{2}} \\ \times {(c_{i} \max_{1 \leq j \leq N_{t}} (z_{m : 2 j - 1}^{2} + z_{m : 2 j}^{2}))}^{\frac{1}{2}} . \end{array}

Therefore:

\begin{array}{r} ‖ B_{m} z_{m} ‖ \leq \max_{1 \leq i \leq N_{t}} \sqrt{c_{i}} (\sum_{j = 1}^{2 N_{t}} B_{m : (2 i - 1) j}^{2} \\ {+ \sum_{j = 1}^{2 N_{t}} B_{m : (2 i) j}^{2})}^{\frac{1}{2}} ‖ z_{m} ‖ . \end{array}

(25)

We have:

\begin{array}{l} \sum_{j = 1}^{2 N_{t}} B_{m : (2 i - 1) j}^{2} + \sum_{j = 1}^{2 N_{t}} B_{m : (2 i) j}^{2} \\ = ({[A_{v} c_{i} + D_{v} (I_{2_{i}}^{2} E (α) + P f_{2}^{2})]}^{2} \\ + 2 {[D_{v} E (α) I_{1_{i}} I_{2_{i}} + D_{v} P f_{1} f_{2}]}^{2} \\ + {[A_{v} c_{i} + D_{v} (I_{1_{i}}^{2} E (α) + P f_{1}^{2})]}^{2}) / \\ (A_{v} c_{i}^{2} + D_{v} E (α) c_{i} {[I_{1_{i}}^{2} + I_{2_{i}}^{2}]}^{2} \\ {+ D_{v} c_{i} P [f_{1}^{2} + f_{2}^{2}])}^{2}, \end{array}

(26)

where A_ν = M²[E(α) + 1]² and D_ν = 2M[E(–α) + 1]. Scaling out the constants in (26) and applying a five-point Laplacian approximation (c_i = 4), we have:

\begin{array}{l} F_{m} = \sum_{j = 1}^{2 N_{t}} B_{m : (2 i - 1) j}^{2} + \sum_{j = 1}^{2 N_{t}} B_{m : (2 i) j}^{2} \\ = \frac{{(4 + I_{2}^{2} + f_{2}^{2})}^{2} + 2 {(I_{1} I_{2} + f_{1} f_{2})}^{2} + {(4 + I_{1}^{2} + f_{1}^{2})}^{2}}{{(16 + 4 I_{1}^{2} + 4 I_{2}^{2} + 4 f_{1}^{2} + 4 f_{2}^{2})}^{2}} \\ < \frac{1}{16} \frac{{(4 + I_{2}^{2} + f_{2}^{2})}^{2} + 2 (G_{m}) + {(4 + I_{1}^{2} + f_{1}^{2})}^{2}}{{(4 + I_{2}^{2} + f_{2}^{2})}^{2} + 2 (G_{m}) + (I_{1}^{2} + I_{2}^{2}) (I_{1}^{2} + 8 + f_{1}^{2})} \\ < \frac{1}{8} \frac{{(4 + I_{2}^{2} + f_{2}^{2})}^{2} + 2 {(G_{m})}^{2} + {(4 + I_{1}^{2} + f_{1}^{2})}^{2}}{{(4 + I_{2}^{2} + f_{2}^{2})}^{2} + 2 {(G_{m})}^{2} + {(4 + I_{1}^{2} + f_{1}^{2})}^{2}} < \frac{1}{8}, \end{array}

(27)

where $G_{m} = I_{1}^{2} I_{2}^{2} + f_{1}^{2} f_{2}^{2}$ .

For sequences ${\tilde{u}}_{i}^{n + 1}$ and ${\tilde{v}}_{i}^{n + 1}$ , a sufficient convergence condition is that $∥ B_{m} ∥ < 1$ ; [44]. Therefore, from (25) and (27), we have: $∥ B_{m} z_{m} ∥ \leq \frac{∥ z_{m} ∥}{\sqrt{2}} \leq ∥ B_{m} ∥ ∥ z_{m} ∥$ where $0 < ∥ B_{m} ∥ < 1$ , showing that the Jacobi iterations (23) converge for any arbitrary choice of initial approximation.

5. Simulations

5.1 Optical Flow Estimation

Tests were conducted on the well-known ‘Hamburg taxi sequence'; see Figure 2. The optical flow results generated by the ‘Horn and Schunck’ and SEOS algorithms can be seen in Figures 3(a) and 3(b), respectively. The main goal of the SEOS simulation was to focus on an object and track it in the image sequence while smoothing out all remaining objects. As seen in Figure 3(b), SEOS produced a dense and uniform flow for the central vehicle in the image sequence.

Figure 2.

‘Hamburg taxi sequence’ showing the central vehicle as the object to be tracked

An important feature of the SEOS method is the fact that any object within the image sequence can be individually tracked and its corresponding flow vectors plotted.

The object being tracked within the image is changed by adjusting the real-world coordinates of the object, and through projection, finding the corresponding image plane coordinates. Figures 4 and 5 illustrate that, with a change in image coordinates, the object being tracked is changed. The SEOS results (Figures 3(b) and 5) illustrate that smoothing is decreased within the object region, increasing the sharpness of the optical flow vectors. In areas outside the object, the smoothing is increased, resulting in minimal flow outside the object. With the ‘Horn and Schunck’ method (the data of Figure 3(a)), there is no uniformity in the vector directions associated with each vehicle, and there is randomness in the vector magnitudes.

SEOS produces an improvement in optical flow estimation over the ‘Horn and Schunck’ method by allowing the compensation of flow to the objects of interest, thereby aligning vector directionality. This is achieved due to the input of the object state parameters into the SEOS function and the assignment of suitable weighting parameters. When comparing the two methods, it is quite evident that there are noticeable performance gains with SEOS.

Figure 3.

(a) Optical flow field of the ‘Hamburg taxi sequence’ using the Horn and Schunck method. (b) ‘Hamburg taxi sequence’ using SEOS to focus in on the central vehicle.

Figure 4.

‘Hamburg taxi sequence’ showing the right moving vehicle as the object of interest

To further establish the benefits of employing SEOS for tracking, another test sequence was evaluated (see Figure 6). This video sequence consists of a toy train moving clockwise around a rectangular track. Consider the train sequence with the optical flow estimated with the use of the Horn and Schunck algorithm in Figure 7. It is quite obvious in Figure 7 that the Horn and Schunck method has quite a lot of difficulty identifying the motion of the train, with optical flow vectors scattering in an array of different directions and with varying magnitudes. The SEOS method applied to the train sequence can be seen in Figure 8. The SEOS method is used to focus in on the moving train. Results show that SEOS generates a dense and uniform flow field, and represents a good approximation of the true 2D motion of the train. The environmental scene around the train is not moving and, in an ideal scenario, should not experience optical flow. The Horn and Schunck method generates erroneous flow results in the area surrounding the train. On the other hand, the SEOS method only calculates the optical flow in areas pertaining to the object of interest. This area of interest is a user-specified input in the form of dynamic state parameters.

Figure 5.

‘Hamburg taxi sequence’ using SEOS to focus in on the rightmost moving vehicle

Figure 6.

Image of the toy train sequence

Figure 7.

Optical flow of the toy train sequence estimated with the Horn and Schunck algorithm

5.2 State Estimation

State parameter estimation is achieved through the implementation of REKF. Robustness is crucial in vision-based systems due to the inherent and relatively significant initial state error. The performance characteristics of the SEOS state estimation were evaluated for the toy train sequence (Figure 6). The noise in the images is assumed to be a bounded function of time and space, and hence the estimated target locations are subject to bounded functions in time. These robust assumptions are in line with the REKF assumptions presented in Section 3. As we are using monocular vision, we only use the linear motion to demonstrate the underlying concept of simultaneous estimation. The nonlinear motion for trajectory tracking is considered (without simultaneous localization) in our stereo-vision papers ([7, 8]).

Figure 8.

Optical flow of the toy train sequence estimated with the SEOS algorithm

Figure 9.

Experimental schematic for state estimation of the toy train sequence

The simulation parameters used are given in Table 1. The train is assumed to have a constant velocity of 170mms⁻¹. The portion of the video sequence that was investigated involved the train moving along the X axis at a distance of 250mm. The stationary observer (camera) is orientated at known distances from the moving train (refer to Figure 9). The optical flow and state estimation are fused together in the SEOS approach to improve the overall estimator performance for the purpose of object-tracking. In Figures 10 and 11, the performance of the REKF state estimation is illustrated for the SEOS and ‘Horn and Schunck’ optical flow techniques. Figure 10 shows the 3D motion of the train. Figure 11 shows the error in magnitude estimation. Estimating the 3D position of the train using the SEOS technique shows an improvement over the Horn and Schunck method, with estimates converging faster towards the true state of the train.

Table 1.

Simulation parameters

Parameter	Value	Comments
Q	10¹⁰	Weighting on uncertain input
N	10²	Weighting on the uncertain initial condition
R	10⁹	Weighings on the measurement noise
$\hat{f}$	50mm	Focal length of the camera
s	1.48sec	Simulation time
x(0)	[−125, 500, 1000, 170, 0, 0]^T	Object Initial state
ξ	0.1	Upper bound of the norm bound portion of noise

Figure 10.

Motion of the object/train in 3D

Figure 11.

Error in magnitude estimation of [x₁, x₂, x₃] train coordinates from camera origin

6. Conclusion

In this paper, we examined the convergence of the SEOS optical flow method. The system of linear equations associated with the SEOS technique was ordered such that its property matrix was positive definite symmetric. Satisfying this property is a sufficient condition for proving that the Gauss-Seidel pointwise and blockwise solutions converge. A block diagonal Jacobi iterative scheme was devised for SEOS and its convergence was proved. We also examined the effect of applying SEOS to a real-image sequence and tracking various objects within the ‘Hamburg taxi sequence'. Our results show a uniform directional flow field for the tracked vehicle. All objects outside the vehicle of interest were able to be smoothed out. This object isolation feature of the SEOS technique could prove invaluable in numerous tracking applications. The effectiveness of using SEOS for 3D state parameter estimation was evaluated. The state parameters were estimated with the use of a REKF. The clear performance benefits of SEOS (over the Horn and Schunck method) were exhibited for the toy train sequence, with reductions in state estimation error. The Horn and Schunck method is not capable of deciphering one object from another, and as such lacks accuracy in the estimation of state parameters.

7. Acknowledgements

This article is a revised and expanded version of paper entitled “Object focused simultaneous estimation of optical flow and state dynamics” by N.J. Bauer and P.N. Pathirana, presented at the International Conference of Intelligent Sensors, Sensor Networks and Information Processing, Melbourne, Australia in 2008.

Footnotes

1

Recall from [] that for a Jacobi iterative scheme B m = I − Q m − 1 A r and c m = Q m − 1 b r , where I is the identity matrix and Q_m is a block diagonal matrix containing 2×2 diagonal block sub-matrices of A_r.

References

Lucas

B. D.

and Kanade

. An iterative image restoration technique with an application in stereo vision. In Proceedings of the 5th International Joint Conference on Artificial Intelligence, pages 674–679, 1981.

Horn

B. K. P.

and Schunk

B. G.

. Determining optical flow. Artificial Intelligence, 17:185–203, 1981.

Jianbo

Yugeng

Hanebeck

U.D.

, and Schmidt

. Nonlinear visual mapping model for 3-d visual tracking with uncalibrated eye-in-hand robotic system. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 34(1):652–659, feb. 2004.

Schmudderich

Willert

Eggert

Rebhan

Goerick

Sagerer

, and Korner

. Estimating object proper motion using optical flow, kinematics, and depth information. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 38(4):1139–1151, aug. 2008.

Chaudhury

Mehrotra

, and Srinivasan

. Detecting 3-d motion field from range image sequences. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 29(2):308–314, apr 1999.

Pathirana

P.N.

Lim

A.E.K.

Savkin

A.V.

, and Hodgson

P.D.

. Robust video/ultrasonic fusion-based estimation for automotive applications. Vehicular Technology, IEEE Transactions on, 56(4):1631–1639, 2007.

Pathirana

Pubudu N.

Bishop

Adrian N.

Savkin

Andrey V.

Ekanayake

Samitha W.

, and Black

Timothy J.

. A method for stereo-vision-based tracking for robotic applications. Robotica, 18:517–524, 2010.

Pathirana

P.N.

Savkin

A.V.

Ekanayake

S.W.

, and Bauer

N.J.

. A robust solution to the stereo-vision based localization and mapping problem with steady and moving landmarks. Advanced Robotics, 25(6–7):765–788, 2011.

Bauer

N.J.

and Pathirana

P.N.

. Object focused simultaneous estimation of optical flow and state dynamics. In Intelligent Sensors, Sensor Networks and Information Processing, 2008. ISSNIP 2008. International Conference on, pages 61–66, dec. 2008.

10.

Souhila

Kahlouche

and Karim

Achour

. Optical flow based robot obstacle avoidance. International Journal of Advanced Robotic Systems, 4(1):13–16, 2007.

11.

Serres

Julien

Ruffier

Franck

Viollet

Stephane

, and Franceschini

Nicolas

. Toward optic flow regulation for wall-following and centring behaviours. International Journal of Advanced Robotic Systems, 3(2):147–154, 2006.

12.

Maier

Josef

and Humenberger

Martin

. Movement detection based on dense optical flow for unmanned aerial vehicles. International Journal of Advanced Robotic Systems, 10:1–11, 2013.

13.

Ahad

Atiqur Rahman

Md Tan

Joo Kooi

Kim

Hyoungseop

, and Ishikawa

Seiji

. Motion history image: its variants and applications. Machine Vision and Applications, 23(2):255–281, 2012.

14.

Fernández-Caballero

Antonio

Castillo

José Carlos

Martínez-Cantos

Javier

, and Martínez-Tomás

Rafael

. Optical flow or image subtraction in human detection from infrared camera on mobile robot. Robotics and Autonomous Systems, 58(12):1273–1281, 2010.

15.

Delgado

Ana E

López

María T

, and Fernández-Caballero

Antonio

. Real-time motion detection by lateral inhibition in accumulative computation. Engineering Applications of Artificial Intelligence, 23(1):129–139, 2010.

16.

Pathirana

P. N.

Lim

A. E.

Carminati

, and Premaratne

. Simultaneous estimation of optical flow and object state: A modified approach to optical flow calculation. Networking, Sensing and Control, 2007 IEEE International Conference on, pages 634–638, April 2007.

17.

Broida

T. J.

Chandrashekhar

, and Chellappa

. Recursive 3-d motion estimation from a monocular image sequence. IEEE transactions on Aerospace and Electronic Systems, 26(4):639–655, July 1990.

18.

Roach

J. W.

and Aggarwal

J. K.

. Determining the movement of objects from a sequence of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-2:554–562, November 1980.

19.

Fang

J. Q.

and Huang

T. S.

. Some experiments on estimating the 3-d motion parameters of a rigid body from two consecutive image frames. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-6:545–554, September 1984.

20.

Blostein

S. D.

Zhao

, and Chann

R. M.

. Three-dimensional trajectory estimation from image position and velocity. IEEE Transactions on Aerospace and Electronic Systems, 36(4):1075–1088, October 2000.

21.

Bertero

Poggio

, and Torre

. Ill-posed problems in early vision. In Proceedings of the IEEE, volume 76, pages 869–889, Aug 1988.

22.

Bigün

and Granlund

G. H.

. Optical flow based on the inertia matrix of the frequency domain. In Proceedings from SSAB Symposium on Picture Processing, Mar 1988.

23.

Bigün

Granlund

G. H.

, and Wiklund

. Multidimensional orientation: texture analysis and optical flow. In Proceedings of the SSAB Symposium on Image Analysis, Mar 1991.

24.

Alvarez

Esclarin

Lefebure

, and Sanchez

. A pde model for computing the optical flow. In Proceedings of the 11th Congreso de Ecuaciones Diferenciales y Aplicaciones, pages 1349–1356, 1999.

25.

Aubert

Deriche

, and Kornprobst

. Computing optical flow via variational techniques. SIAM Journal on Applied Mathematics, 60(1):156–182, 1999.

26.

Black

M. J.

and Anandan

. Robust dynamic motion estimation over time. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 292–302, Jun 1991.

27.

Cohen

. Nonlinear variational method for optical flow computation. In Proceedings of the 8th Scandinavian Conference on Image Analysis, volume 1, pages 523–530, May 1993.

28.

Heitz

and Bouthemy

. Multimodal estimation of discontinuous optical flow using markov random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(12):1217–1232, dec 1993.

29.

Proesmans

Van Gool

Pauwels

, and Oosterlinck

. Determination of optical flow and its discontinuities using non-linear diffusion. In Proceedings of the European Conference on Computer Vision, volume 801, pages 295–304, 1994.

30.

Schnörr

. Segmentation of visual motion by minimizing convex nonquadratic functionals. In Proceedings. Twelfth International Conference on Pattern Recognition, volume A, pages 661–663, oct 1994.

31.

Barron

J. L.

Fleet

D. J.

, and Beauchemin . Performance of optical flow techniques. International Journal of Computer Vision, 12:43–77, 1994.

32.

Galvin

McCane

Novins

Mason

, and Mills

. Recovering motion fields: an analysis of eight optical flow algorithms. In Proceedings of the British Machine Vision Conference, 1998.

33.

Savkin

A. V.

and Petersen

I. R.

. Recursive state estimation for uncertain systems with an integral quadratic contraint. IEEE Transactions on Automatic Control, 40(6):1080–1083, Jun 1995.

34.

Petersen

I. R.

and Savkin

A. V.

. Robust kalman filtering for signals and systems with large uncertainties. Boston, MA:Birkhauser, 1999.

35.

Pathirana

P. N.

Savkin

A. V.

, and Jha

. Location estimation and trajectory prediction for cellular networks with mobile base stations. IEEE Transactions on Vehicle Technology, 53(6):1903–1913, 2004.

36.

Sorenson

H. W.

. Kalman Filtering: Theory and Application. IEEE Press, 1985.

37.

Poor

H. V.

. An Introduction to Signal Detection and Estimation. IEEE Press, 1994.

38.

Marat

T. A.

. Digital Video Processing. Prentice Hall PTR, Upper Saddle River, NJ, 1995.

39.

Forsyth

D. A.

and Ponce

. Computer Vision: A Modern Approach. Prentice Hall, 2003.

40.

Savkin

A. V.

and Petersen

I. R.

. Nonlinear versus linear control in the absolute stabilization of uncertain systems with structured uncertainty. IEEE Transactions on Automatic Control, 40(1):122–127, 1995.

41.

Savkin

A. V.

and Evans

R. J.

. Hybrid Dynamical Systems: Controller and Sensor Switching Problems. Birkhäuser, Boston, 2002.

42.

Anderson

B. D. O.

and Moore

J. B.

. Optimal Filtering. Prentice Hall, New Jersey, 1979.

43.

Horn

B. K. P.

. Robot vision. MIT Press, Massachusetts, Cambridge, 1986.

44.

Datta

B. N.

. Numerical Linear Algebra and Applications. Brooks/Cole, 1995.

45.

Cheney

and Kincaid

. Numerical Mathematics and Computing. Brooks/Cole, 5th edition, 2004.

46.

Stoer

and Bulirsch

. Introduction to Numerical Analysis. Springer-Verlag, Berlin, Germany, 2nd edition, 1993.

47.

Mitiche

and Mansouri

A.R.

. On convergence of the horn and schunck optical-flow estimation method. IEEE Transactions on Image Processing, 13(6):848–852, June 2004.

Convergence of Object Focused Simultaneous Estimation of Optical Flow and State Dynamics

Abstract

Keywords

1. Introduction

2. Dynamic Model and Monocular Vision Measurement

2.1 Dynamic Model

5.1 Optical Flow Estimation

7. Acknowledgements

Footnotes

1

References