Article Contents

2017, Volume 11, Issue 4: 761-781. Doi: 10.3934/ipi.2017036

This issue Previous Article Increasing stability for the inverse source scattering problem with multi-frequencies Next Article

On a spatial-temporal decomposition of optical flow

Aniello Raffaele Patrone^1, , and
Otmar Scherzer^1,2,

1.
Computational Science Center, University of Vienna, Oskar-Morgenstern Platz 1,1090 Vienna, Austria
2.
Johann Radon Institute for Computational and Applied Mathematics, (RICAM), Altenbergerstraẞe 69,4040 Linz, Austria

* Corresponding author: Aniello Raffaele Patrone
* Corresponding author: Aniello Raffaele Patrone

Received: July 2015

Revised: April 2017

Published: October 2017

The first author is supported by WWTF.

Abstract / Introduction Full Text(HTML) Figure(8) / Table(3) Related Papers Cited by

Abstract

In this paper we present a decomposition algorithm for computation of the spatial-temporal optical flow of a dynamic image sequence. We consider several applications, such as the extraction of temporal motion features and motion detection in dynamic sequences under varying illumination conditions, such as they appear for instance in psychological flickering experiments. For the numerical implementation we are solving an integro-differential equation by a fixed point iteration. For comparison purposes we use a standard time dependent optical flow algorithm, which in contrast to our method, constitutes in solving a spatial-temporal differential equation.

Keywords:

Mathematics Subject Classification: Primary: 65F22, 35A15, 68U10.

Citation:

Full Text(HTML)

Figure 1. $f(x,t)=x(1-x)(1-t)$ from (7). Level lines of $f$ are parametrized by $(\Psi(x,t),t)$

Download: Full-size image PowerPoint slide

Figure 2. $g(t)=\exp \left\{-\frac{1}{\beta}(1-t)^\beta\right\}$

Download: Full-size image PowerPoint slide

Figure 3. Color Wheel

Download: Full-size image PowerPoint slide

Figure 4. ${\vec u^{\left( 2 \right)}}$ at different frequencies of rotations: $2$, $4$ and $8 \times$ faster than the original motion frequency. $\alpha^{(1)}=1$, $\alpha^{(2)}=\frac{1}{4}$. The intensity of ${\vec u^{\left( 2 \right)}}$ increases when the frequency of rotation is increased

Download: Full-size image PowerPoint slide

Figure 6. ${\vec u^{\left( 1 \right)}}$: Movement of a Ferris wheel and people walking in the foreground (top left). ${\vec u^{\left( 2 \right)}}$ consists of blinking lights and the reflections of the wheel (top right). The third image (bottom) is a reference frame

Download: Full-size image PowerPoint slide

Figure 5. The dynamic sequence consists of the smooth (translation like) motion of a cube and an oscillating background. The oscillation has a periodicity of four frames and takes place along the diagonal direction from the bottom left to the top right, moving at a rate of 5% of the frame size in each frame. The proposed model decomposes the motion, obtaining the global movement of the cube in ${\vec u^{\left( 1 \right)}}$ (left) and the background movement exclusively in ${\vec u^{\left( 2 \right)}}$ (right).

Download: Full-size image PowerPoint slide

Figure 8. The two frames of the flickering sequence containing information (top), the difference between these two frames (down left), and the ${\vec u^{\left( 2 \right)}}$ flow field resulting from the proposed approach (down right). As predicted in Section 3 and Appendix A the ${\vec u^{\left( 1 \right)}}$ component is negligible, instead ${\vec u^{\left( 2 \right)}}$ detects the change of intensity across the blank sheet.

Download: Full-size image PowerPoint slide

Figure 7. Result with Horn-Schunck

Download: Full-size image PowerPoint slide

Table 1. Continuous notation

$\vec x = (x_1,x_2)$	vector in two-dimensional Euclidean space
$\partial_k = \frac{\partial}{\partial x_k}$	differentiation with respect to spatial variable $x_k$
$\partial_t = \frac{\partial}{\partial t}$	differentiation with respect to time
$\nabla = (\partial_1, \partial_2)^T$	gradient in space
$\nabla_3 = (\partial_1, \partial_2, \partial_t)^T$	gradient in space and time
$\nabla \cdot = \partial_1 + \partial_2$	divergence in space
$\nabla_3 \cdot = \partial_1 + \partial_2 + \partial_t$	divergence in space and time
$\vec{n}$	outward pointing normal vector to $\Omega$
$f$	input sequence
$f(\cdot,t)$	movie frame
${\vec u^{\left( i \right)}}$	optical flow module, $i=1,2$
$\vec u = {\vec u^{\left( 1 \right)}} + {\vec u^{\left( 2 \right)}}$	optical flow
$u_j^{\left( i \right)}$	$j$-th optical flow component of the $i$-th module
$\widehat{u}(\cdot,t) = \int_0^t u(\cdot,\tau)\,{\rm{d}} \tau$	primitive of $u$
$\widehat{\widehat{u}}(\cdot,t) = -\int_t^1 \widehat{u}(\cdot,\tau)\,{\rm{d}} \tau$	2nd primitive of $u$ -note that $\partial_t \widehat{\widehat{u}}(\cdot,t)=\widehat{u}(\cdot,t)$

| Show Table

DownLoad: CSV

Table 2. Discrete Notation

$f= f(r,s,t) \in \mathbb{R}^{M \times N \times T}$	input sequence
${\vec u^{\left( i \right)}} = {\vec u^{\left( i \right)}}(r,s,t;k) \in \mathbb{R}^{M \times N \times T \times K \times 2}$	discrete optical flow approximating the continuous flow ${\vec u^{\left( i \right)}}$ at $(\frac{r}{M-1},\frac{s}{N-1},\frac{t}{T-1})$
$\partial_k^h$	finite difference approximation in direction $x_k$
$\partial_t^h$	finite difference approximation in direction $t$
$\Delta_x=\frac{1}{M-1}$, $\Delta_y=\frac{1}{N-1}$ and $\Delta_t=\frac{1}{T-1}$	Discretization
$ \hat u_j^{\left( 2 \right)}(r,s,t;k) = \Delta_t \sum_{\tau=1}^t u_j^{\left( 2 \right)}(r,s,\tau;k)$, $j=1,2$	finite difference approximation of $\widehat{u}(\cdot,t)$
$ \hat {\hat {u}}_j^{\left( 2 \right)}(r,s,t;k)= - \Delta_t \sum_{\tau = t}^T \hat u_j^{\left( 2 \right)}(r,s,\tau;k)$	finite difference approximation of $\widehat{\widehat{u}}(\cdot,t)$

| Show Table

DownLoad: CSV

Table 3. Comparison of squared residuals over space and time $\mathcal{E}$ between Weickert-Schnörr and the proposed method

	Weickert-Schnörr	Proposed model
Hamburg Taxi	1374.9	1021
RubberWhale	4459.7	3046.8
Hydrangea	8533.3	7647.2
DogDance	9995.4	8217.6
Walking	8077.5	5944.3

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	J. Abhau, Z. Belhachmi and O. Scherzer, On a decomposition model for optical flow, in Energy Minimization Methods in Computer Vision and Pattern Recognition, Lecture Notes in Computer Science, 5681, Springer-Verlag, Berlin, Heidelberg, 2009,126-139. doi: 10.1007/978-3-642-03641-5_10.
[2]	R. Andreev, O. Scherzer and W. Zulehner, Simultaneous optical flow and source estimation: Space-time discretization and preconditioning, Applied Numerical Mathematics, 96 (2015), 72-81. doi: 10.1016/j.apnum.2015.04.007.
[3]	G. Aubert and J. F. Aujol, Modeling very oscillating signals. Application to image processing, Appl. Math. Optim., 51 (2005), 163-182. doi: 10.1007/s00245-004-0812-z.
[4]	J. F. Aujol and A. Chambolle, Dual norms and image decomposition models, Int. J. Comput. Vision, 63 (2005), 85-104.
[5]	J. F. Aujol and S. Kang, Color image decomposition and restoration, J. Vis. Commun. Image Represent, 17 (2006), 916-928. doi: 10.1016/j.jvcir.2005.02.001.
[6]	J. F. Aujol, G. Aubert, L. Blanc-Féraud and A. Chambolle, Image decomposition into a bounded variation component and an oscillating component, J. Math. Imaging Vision, 22 (2005), 71-88. doi: 10.1007/s10851-005-4783-8.
[7]	J. F. Aujol, G. Gilboa, T. Chan and S. Osher, Structure-texture image decomposition-modeling, algorithms, and parameter selection, Int. J. Comput. Vision, 67 (2006), 111-136.
[8]	S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black and R. Szeliski, A database and evaluation methodology for optical flow, Int. J. Comput. Vision, 92 (2011), 1-31. doi: 10.1007/s11263-010-0390-2.
[9]	M. Bauer, M. Bruveris and W. P. Michor, Overview of the geometries of shape spaces and diffeomorphism groups, J. Math. Imaging Vision, 50 (2014), 60-97. doi: 10.1007/s10851-013-0490-z.
[10]	B. Berkels, A. Effland and M. Rumpf, Time discrete geodesic paths in the space of images, SIAM J. Imaging Sci., 8(2015),1457-1488, arXiv: 1503.02001. doi: 10.1137/140970719.
[11]	A. Borzi, K. Ito and K. Kunisch, Optimal control formulation for determining optical flow, SIAM J. Sci. Comput., 24 (2002), 818-847. doi: 10.1137/S1064827501386481.
[12]	A. Bruhn, Variational Optic Flow Computation: Accurate Modeling and Efficient Numerics, PhD thesis 2006, Saarland University, Germany.
[13]	V. Duval, J. F. Aujol and L. Vese, Mathematical modeling of textures: Application to color image decomposition with a projected gradient algorithm, J. Math. Imaging Vision, 37 (2010), 232-248. doi: 10.1007/s10851-010-0203-9.
[14]	M. Hanke and O. Scherzer, Inverse problems light: Numerical differentiation, Amer. Math. Monthly, 108 (2001), 512-521. doi: 10.2307/2695705.
[15]	B. K. P. Horn and B. G. Schunck, Determining optical flow, Artificial Intelligence, 17 (1981), 185-203.
[16]	A. Jain and L. Younes, A kernel class allowing for fast computations in shape spaces induced by diffeomorphisms, J. Comput. and Applied Math., 245 (2013), 162-181. doi: 10.1016/j.cam.2012.10.019.
[17]	C. Kirisits, L. Lang and O. Scherzer, Decomposition of optical flow on the sphere, GEM-Int. J. on Geomathematics, 5 (2014), 117-141. doi: 10.1007/s13137-013-0055-8.
[18]	T. Kohlberger, E. Memin and C. Schnörr, Variational dense motion estimation using the helmholtz decomposition, in Scale Space Methods in Computer Vision (eds. L. D. Griffin and M. Lillholm), Lecture Notes in Computer Science, 2695, Springer, Berlin, 2003,432-448. doi: 10.1007/3-540-44935-3_30.
[19]	J. Lee and J. C. Wright, A versatile parallel block-tridiagonal solver for spectral codes, MIT Plasma Science & Fusion Center, 2010.
[20]	B. McCane, K. Novins, D. Crannitch and B. Galvin, On benchmarking optical flow, Comput. Vision Image Understanding, 84 (2001), 126-143.
[21]	Y. Meyer, Oscillating Patterns in Image Processing and Nonlinear Evolution Equations, University Lecture Series, vol 22, American Mathematical Society, Providence, RI, 2001. doi: 10.1090/ulect/022.
[22]	M. I. Miller and L. Younes, Group actions, homeomorphisms, and matching: A general framework, Int. J. Comput. Vision, 41 (2001), 61-84.
[23]	J. K. O'Regan, Change blindness, E Cognitive Science, 2007.
[24]	O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier and F. Lenzen, Variational Methods in Imaging, Applied Mathematical Sciences, vol 167, Springer, New York, 2009. doi: 10.1007/978-0-387-69277-7.
[25]	L. Vese and S. Osher, Modeling textures with total variation minimization and oscillating patterns in image processing, J. Sci. Comput., 19 (2003), 553-572. doi: 10.1023/A:1025384832106.
[26]	L. Vese and S. Osher, Image denoising and decomposition with total variation minimization and oscillatory functions, J. Math. Imaging Vision, 20 (2004), 7-18. doi: 10.1023/B:JMIV.0000011316.54027.6a.
[27]	C. M. Wang, K. C. Fan and C. T. Wang, Estimating optical flow by integrating multi-frame information, Journal of Information Science and Engineering, 24 (2008), 1719-1731.
[28]	J. Weickert, A. Bruhn, T. Brox and N. Papenberg, A survey on variational optic flow methods for small displacements, Mathematical Models for Registration and Applications to Medical Imaging, Springer Berlin Heidelberg, 10 (2006), 103-136. doi: 10.1007/978-3-540-34767-5_5.
[29]	J. Weickert and C. Schnörr, A theoretical framework for convex regularizers in PDE-based computation of image motion, Int. J. Comput. Vision, 45 (2001), 245-264.
[30]	J. Weickert and C. Schnörr, Variational optic flow computation with a spatio-temporal smoothness constraint, J. Math. Imaging Vision, 14 (2001), 245-255.
[31]	J. Yuan, C. Schörr and G. Steidl, Simultaneous higher-order optical flow estimation and decomposition, SIAM J. Sci. and Stat. Comput., 29 (2007), 2283-2304 (electronic). doi: 10.1137/060660709.
[32]	J. Yuan, G. Steidl and C. Schnörr, Convex Hodge decomposition of image flows, in Pattern Recognition, Lecture Notes in Comput. Sci., 5096, Springer, Berlin, 2008,416-425. doi: 10.1007/978-3-540-69321-5_42.
[33]	J. Yuan, C. Schnörr and G. Steidl, Convex Hodge decomposition and regularization of image flows, J. Math. Imaging Vision, 33 (2009), 169-177. doi: 10.1007/s10851-008-0122-1.