Nonlocal variational models for inpainting and interpolation. (English) Zbl 1267.68278
Image inpainting is a very active research area in image analysis and processing. The main task of inpainting is to obtain an image interpolation in a region in which data are missing. Image inpainting has numerous applications to image and video editing, and it is of theoretical interest. In this paper the authors establish a precise mathematical formulation for image inpainting models. The paper is divided into three parts. In the first part the authors review a general nonlocal variational model for image inpainting. In the second part, they study the problem of simultaneous inpainting of stereo images and the inpainting of disparity maps. In the third part, the authors consider an example of video inpainting based on the propagation of inpainted frame. Finally the authors discuss some basic inpainting algorithms and display some numerical experiments illustrating the properties of proposed models. The paper is hard to read for a non-mathematician reader.
Reviewer: Oscar Bustos (Córdoba)
MSC:
| 68U10 | Computing methodologies for image processing |
| 35A15 | Variational methods applied to PDEs |
| 65C50 | Other computational problems in probability (MSC2010) |
Software:
PRMLTReferences:
| [1] | Alvarez, L.Weickert, J.Sánchez, J., Int. J. Comput. Vis.39, 41 (2000). · Zbl 1060.68635 |
| [2] | Ambrosio, L.; Fusco, N.; Pallara, D., Functions of Bounded Variation and Free Discontinuity Problems, 2000, Oxford Univ. Press · Zbl 0957.49001 |
| [3] | Andreu, F., Nonlocal Diffusion Problems, 2010, Amer. Math. Soc · Zbl 1214.45002 |
| [4] | P. Arias, V. Caselles, G. Facciolo, V. Lazcano and R. Sadek, On the manifold of patches of an image, preprint (2012). |
| [5] | Arias, P.Caselles, V.Sapiro, G., A variational framework for non-local image inpainting, EMMCVPR, (Springer, 2009) pp. 245-258. |
| [6] | P. Arias, G. Facciolo and V. Caselles, Analysis of a variational framework for exemplar-based image inpainting, preprint, University Pompeu Fabra (2011). |
| [7] | Arias, P.et al., Int. J. Comput. Vis.93, 319 (2011). |
| [8] | Aujol, J. F.Ladjal, S.Masnou, S., SIAM J. Math. Anal.42, 1246 (2010). · Zbl 1210.49002 |
| [9] | Awate, S. P.Whitaker, R. T., IEEE Trans. Pattern Anal. Machine Intell.28, 364 (2006). |
| [10] | Ballester, C.et al., IEEE Trans. Image Process.10, 1200 (2001). · Zbl 1037.68771 |
| [11] | Barnes, C.et al., ACM Trans. Graph.28, 1 (2009). |
| [12] | Barnes, C.et al., The generalized PatchMatch correspondence algorithm, European Conf. Computer Vision (Springer, 2010) pp. 29-43. |
| [13] | Bertalmío, M.et al., Image inpainting, Proc. 27th Annual Conf. Computer Graphics and Interactive Techniques (ACM Press, 2000) pp. 417-424. |
| [14] | Bertalmío, M.et al., IEEE Trans. Image Process.12, 882 (2003). |
| [15] | Bertsekas, D. P., Nonlinear Programming, 1999, Athena Scientific · Zbl 1015.90077 |
| [16] | Bishop, C. M., Pattern Recognition and Machine Learning, 2006, Springer · Zbl 1107.68072 |
| [17] | Bornard, R.et al., Missing data correction in still images and image sequences, Proc. ACM Int. Conf. Multimedia (ACM Press, 2002) pp. 355-361. |
| [18] | Bornemann, F.März, T., J. Math. Imaging Vis.28, 259 (2007). · Zbl 1451.94007 |
| [19] | Bourgain, J.Brezis, H.Mironescu, P., Optimal Control and Partial Differential Equation: In Honor to Professor Alain Bensoussan’s 60th Birthday (IOS Press, 2001) pp. 439-455. · Zbl 1103.46310 |
| [20] | Brox, T.et al., High accuracy optical flow estimation based on a theory for warping, European Conf. Computer Vision — ECCV (Springer, 2004) pp. 25-36. |
| [21] | Brox, T.Kleinschmidt, O.Cremers, D., IEEE Trans. Image Process.17, 1083 (2008). |
| [22] | Buades, A.Coll, B.Morel, J. M., Numer. Math.105, 1 (2006). · Zbl 1151.94310 |
| [23] | Buades, A.Coll, B.Morel, J. M., Multiscale Model. Simulat.4, 490 (2006). |
| [24] | Buades, A.et al., IEEE Trans. Image Process.18, 1192 (2009). |
| [25] | Cannarsa, P.; Sinestrari, C., Semiconcave Functions, Hamilton-Jacobi Equations, and Optimal Control, 58, 2004, Birkhäuser · Zbl 1095.49003 |
| [26] | Cao, F.et al., SIAM J. Imaging Sci.4, 1143 (2011). |
| [27] | Chan, T.Kang, S. H.Shen, J. H., SIAM J. Appl. Math.63, 564 (2002). · Zbl 1028.68185 |
| [28] | Chan, T.Shen, J. H., SIAM J. Appl. Math.62, 1019 (2001). |
| [29] | Comaniciu, D.Meer, P., IEEE Trans. Pattern Anal. Machine Intell.24, 603 (2002). |
| [30] | Criminisi, A.Perez, P.Toyama, K., IEEE Trans. Image Process.13, 1200 (2004). |
| [31] | Demanet, L.Song, B.Chan, T., Appl. Comput. Math.1100, 217 (2003). |
| [32] | Desolneux, A.Moisan, L.Morel, J. M., J. Math. Imaging Vis.14, 271 (2001). · Zbl 0988.68819 |
| [33] | Digne, J., Handbook of Mathematical Methods in Imaging, 2010, Springer |
| [34] | Drori, I.Cohen-Or, D.Yeshurun, H., ACM Trans. Graph.22, 303 (2003). |
| [35] | Efros, A. A.Leung, T. K., Texture synthesis by non-parametric sampling, Proc. IEEE Int. Conf. Computer Vision (IEEE Press, 1999) pp. 1033-1038. |
| [36] | Ellis, R. S., Entropy, Large Deviations, and Statistical Mechanics, 2005, Springer |
| [37] | Esedoglu, S.Ruuth, S.Tsai, R., Interfaces Free Boundaries10, 263 (2008). |
| [38] | Esedoglu, S.Shen, J. H., European J. Appl. Math.13, 353 (2002). · Zbl 1017.94505 |
| [39] | Facciolo, G.et al., Energy Minimization Methods in Computer Vision and Pattern Recognition, (Springer, 2009) pp. 331-344. |
| [40] | Facciolo, G.et al., Energy Minimization Methods in Computer Vision and Pattern Recognition, (Springer, 2011) pp. 59-73. |
| [41] | Gilboa, G.Osher, S. J., Multiscale Model. Simulat.6, 595 (2007). · Zbl 1140.68517 |
| [42] | Gilboa, G.Osher, S., Multiscale Model. Simulat.7, 1005 (2008). · Zbl 1181.35006 |
| [43] | P. Harrison, Texture tools, Ph.D. thesis, Monash University (2005). |
| [44] | Hartley, R.; Zisserman, A., Multiple View Geometry in Computer Vision, 2000, Cambridge Univ. Press · Zbl 0956.68149 |
| [45] | Hervieu, A.et al., Stereoscopic image inpainting: Distinct depth maps and images inpainting, 2010 Int. Conf. Pattern Recognition (IEEE Press, 2010) pp. 4101-4104. |
| [46] | Hervieu, A.et al., Stereoscopic image inpainting using scene geometry, Int. Conf. Multimedia and Expo (ICME) (IEEE Press, 2011) pp. 1-6. |
| [47] | Huhle, B.et al., Computer Vis. Image and Understanding114, 1336 (2010). |
| [48] | Jaynes, E. T., Phys. Rev.106, 620 (1957). |
| [49] | Jia, J.et al., IEEE Trans. Pattern Anal. Machine Intell.28, 832 (2006). |
| [50] | Jia, J.et al., Video repairing: Inference of foreground and background under severe occlusion, Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (IEEE Press, 2004) pp. 364-371. |
| [51] | Jia, J.Tang, C.-K., IEEE Trans. Pattern Anal. Machine Intell.26, 771 (2004). |
| [52] | Kawai, N.Sato, T.Yokoya, N., Adv. in Image and Video Tech (Springer, 2009) pp. 271-282. |
| [53] | Kindermann, S.Osher, S.Jones, P. W., Multiscale Model. Simulat.4, 1091 (2006). |
| [54] | Kolmogorov, V.Zabih, R., Multi-camera scene reconstruction via graph cuts, European Conf. Computer Vision (Springer, 2002) pp. 8-40. |
| [55] | Komodakis, N.Tziritas, G., IEEE Trans. Image Process.16, 2649 (2007). |
| [56] | Kwatra, V.et al., ACM Trans. Graph.24, 795 (2005). |
| [57] | V. Lazcano, P. Arias and V. Caselles, A gradient based neighborhood filter for depth interpolation, preprint (2011). |
| [58] | Levin, A.Zomet, A.Weiss, Y., Learning how to inpaint from global image statistics, Proc. IEEE Int. Conf. Computer Vision (IEEE Press, 2003) pp. 305-312. |
| [59] | Y. Liu and V. Caselles, A model for stereoscopic image inpainting, preprint (2012). |
| [60] | Masnou, S., IEEE Trans. Image Process.11, 68 (2002). |
| [61] | Masnou, S.Morel, J.-M., Level lines based disocclusion, Proc. IEEE Int. Conf. Image Processing3 (IEEE Press, 1998) pp. 259-263. |
| [62] | Morel, J. M.; Solimini, S., Variational Methods in Image Segmentation: With Seven Image Processing Experiments, 1995, Birkhäuser |
| [63] | Nitzberg, M.; Mumford, D.; Shiota, T., Filtering, Segmentation, and Depth, 1993, Springer · Zbl 0801.68171 |
| [64] | Patwardhan, K. A.Sapiro, G.Bertalmio, M., Video inpainting of occluding and occluded objects, Proc. IEEE Int. Conf. Image Processing2 (IEEE Press, 2005) pp. 69-72. |
| [65] | Perez, P.Gangnet, M.Blake, A., ACM Trans. Graph.22, 313 (2003). |
| [66] | P. Perez, M. Gangnet and A. Blake, PatchWorks: Example-based region tiling for image editing, Technical Report, Microsoft Research (2004). |
| [67] | Perona, P.Malik, J., IEEE Trans. Pattern Anal. Machine Intell.12, 629 (1990). |
| [68] | Peyré, G., Computer Vis. Image and Understanding113, 249 (2009). |
| [69] | Peyré, G.Bougleux, S.Cohen, L., Non-local regularization of inverse problems, European Conf. Computer Vision (Springer, 2008) pp. 57-68. |
| [70] | Pizarro, L.et al., Int. J. Comput. Vis.90, 62 (2010). |
| [71] | Ponce, A. C., Calc. Var. Partial Differential Equations19, 229 (2004). |
| [72] | Protter, M.et al., IEEE Trans. Image Process.18, 36 (2009). · Zbl 1371.94300 |
| [73] | Rose, K., Proc. IEEE86, 2210 (1998). |
| [74] | R. Sadek, P. Arias, G. Facciolo and V. Caselles, Gradient based models for video editing, preprint (2012). |
| [75] | Smith, S. M.Brady, J. M., Int. J. Comput. Vis.23, 45 (1997). |
| [76] | Srivastava, S. M., A Course on Borel Sets, 1998, Springer · Zbl 0903.28001 |
| [77] | Sun, J.et al., ACM Trans. Graph.24, 861 (2005). |
| [78] | Tomasi, C.Manduchi, R., Bilateral filtering for gray and color images, Sixth Int. Conf. Computer Vision (IEEE Press, 1998) pp. 839-846. |
| [79] | Tschumperlé, D., Int. J. Comput. Vis.68, 65 (2006). · Zbl 1481.94031 |
| [80] | Wang, H.et al., Graph. Models69, 55 (2007). |
| [81] | Wang, L.et al., Stereoscopic inpainting: Joint color and depth completion from stereo images, IEEE Conf. Computer Vision and Pattern Recognition (IEEE Press, 2008) pp. 1-8. |
| [82] | Wei, L. Y.Levoy, M., Fast texture synthesis using tree-structured vector quantization, Proc. 27th Annual Conf. Computer Graphics and Interactive Techniques (ACM Press, 2000) pp. 479-488. |
| [83] | Wexler, Y.Shechtman, E.Irani, M., IEEE Trans. Pattern Anal. Machine Intell.29, 463 (2007). |
| [84] | Wu, C. F. J., Ann. Statist.11, 95 (1983). |
| [85] | Yang, Q.et al., Spatial-depth super-resolution for range images, IEEE Conf. Computer Vision and Pattern Recognition (IEEE Press, 2007) pp. 1-8. |
| [86] | Yaroslavsky, L. P., Digital Picture Processing. An Introduction, 1985, Springer · Zbl 0585.94001 |
| [87] | Yoon, K. J.Kweon, I. S., IEEE Trans. Pattern Anal. Machine Intell.28, 650 (2006). |
| [88] | Zach, C.Pock, T.Bischof, H., Pattern Recognition, (Springer, 2007) pp. 214-223. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
