An optimal fluid optical flow registration for super-resolution with Lamé parameters learning. (English) Zbl 07702981
Summary: The main idea of multi-frame super-resolution (SR) algorithms is to recover a single high-resolution image through a series of low-resolution ones of a captured scene. The success of the SR approaches is often related to well registration and restoration steps. In this work, we propose a new approach based on fluid optical flow image registration and a second-order regularization term to treat both the registration and restoration steps. The fluid registration is introduced to avoid misregistration errors, while the second-order regularization resolved by the Bregman iteration is employed to reduce the image artifacts. Moreover, we propose a bilevel supervised learning framework to compute the Lamé coefficients \(\lambda\) and \(\mu\), which perform the nonparametric registration of the super-resolution result. The numerical part demonstrated that the proposed method copes with some competitive SR methods.
MSC:
| 68U10 | Computing methodologies for image processing |
| 90C25 | Convex programming |
| 94A08 | Image processing (compression, reconstruction, etc.) in information and communication theory |
Software:
nlpdegmReferences:
| [1] | Ardila, JP; Tolpekin, VA; Bijker, W.; Stein, A., Markov-random-field-based super-resolution mapping for identification of urban trees in vhr images, ISPRS J. Photogramm. Remote Sens., 66, 6, 762-775 (2011) |
| [2] | Baker, S.; Kanade, T., Super-resolution optical flow (1999), Pittsburgh: Carnegie Mellon University, The Robotics Institute, Pittsburgh |
| [3] | Borzi, A.; Ito, K.; Kunisch, K., Optimal control formulation for determining optical flow, SIAM J. Sci. Comput., 24, 3, 818-847 (2003) · Zbl 1041.68103 |
| [4] | Buades, A.; Coll, B.; Morel, J-M, The staircasing effect in neighborhood filters and its solution, IEEE Trans. Image Process., 15, 6, 1499-1505 (2006) |
| [5] | Castro, P.; Reyes, JCDL, A bilevel learning approach for optimal observation placement in variational data assimulation, Inverse Probl., 36, 3, 1-31 (2020) · Zbl 1473.35608 |
| [6] | Chambolle, A.; Pock, T., A first-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vis., 40, 1, 120-145 (2011) · Zbl 1255.68217 |
| [7] | Chen, J.; Nunez-Yanez, J.; Achim, A., Video super-resolution using generalized gaussian Markov random fields, IEEE Signal Process. Lett., 19, 2, 63-66 (2012) |
| [8] | Cheong, JY; Park, IK, Deep CNN-based super-resolution using external and internal examples, IEEE Signal Process. Lett., 24, 8, 1252-1256 (2017) |
| [9] | Clason, C.; Valkonen, T., Primal-dual extragradient methods for nonlinear nonsmooth PDE-constrained optimization, SIAM J. Optim., 27, 3, 1314-1339 (2017) · Zbl 1369.49040 |
| [10] | Dempe, S.; Harder, F.; Mehlitz, P.; Wachsmuth, G., Solving inverse optimal control problems via value functions to global optimality, J. Global Optim., 74, 2, 297-325 (2019) · Zbl 1417.49047 |
| [11] | El Mourabit, I.; El Rhabi, M.; Hakim, A.; Laghrib, A.; Moreau, E., A new denoising model for multi-frame super-resolution image reconstruction, Signal Process., 132, 51-65 (2017) |
| [12] | Engl, HW; Hanke, M.; Neubauer, A., Regularization of inverse problems (mathematics and its applications) (2000), Netherlands: Springer, Netherlands |
| [13] | Farsiu, S.; Robinson, MD; Elad, M.; Milanfar, P., Fast and robust multiframe super resolution, IEEE Trans. Image Process., 13, 10, 1327-1344 (2004) |
| [14] | Fransens, R.; Strecha, C.; Van Gool, L., Optical flow based super-resolution: a probabilistic approach, Computer Vis. Underst., 106, 1, 106-115 (2007) |
| [15] | Freeman, WT; Jones, TR; Pasztor, EC, Example-based super-resolution, IEEE Comput. Graph. Appl., 22, 2, 56-65 (2002) |
| [16] | Gao, X.; Zhang, K.; Tao, D.; Li, X., Joint learning for single-image super-resolution via a coupled constraint, IEEE Trans. Image Process., 21, 2, 469-480 (2012) · Zbl 1372.94088 |
| [17] | Gernot Holler, KK; Barnard, RC, A bilevel approach for parameter learning in inverse problems, Inverse Probl., 34, 11 (2018) · Zbl 1400.49046 |
| [18] | Ghosh, D.; Kaabouch, N.; Hu, W-C, A robust iterative super-resolution mosaicking algorithm using an adaptive and directional Huber-Markov regularization, J. Vis. Commun. Image Represent., 40, 98-110 (2016) |
| [19] | Goldstein, T.; Osher, S., The split Bregman method for l1-regularized problems, SIAM J. Imaging Sci., 2, 2, 323-343 (2009) · Zbl 1177.65088 |
| [20] | Golub, GH; Heath, M.; Wahba, G., Generalized cross-validation as a method for choosing a good ridge parameter, Technometrics, 21, 2, 215-223 (1979) · Zbl 0461.62059 |
| [21] | Hansen, PC, Analysis of discrete ill-posed problems by means of the l-curve, SIAM Rev., 34, 4, 561-580 (1992) · Zbl 0770.65026 |
| [22] | Hansen, PC; O’Leary, DP, The use of the l-curve in the regularization of discrete ill-posed problems, SIAM J. Sci. Comput., 14, 6, 1487-1503 (1993) · Zbl 0789.65030 |
| [23] | He, Y.; Yap, K-H; Chen, L.; Chau, L-P, A nonlinear least square technique for simultaneous image registration and super-resolution, IEEE Trans. Image Process., 16, 11, 2830-2841 (2007) |
| [24] | Jia, K.; Gong, S., Hallucinating multiple occluded face images of different resolutions, Pattern Recognit. Lett., 27, 15, 1768-1775 (2006) |
| [25] | Kasetkasem, T.; Arora, MK; Varshney, PK, Super-resolution land cover mapping using a Markov random field based approach, Remote Sens. Environ., 96, 3-4, 302-314 (2005) |
| [26] | Khattab, MM; Zeki, AM; Alwan, AA; Badawy, AS, Regularization-based multi-frame super-resolution: a systematic review, J. King Saud Univ. Comput. Inf. Sci., 32, 7, 755-762 (2018) |
| [27] | Kim, KI; Kwon, Y., Single-image super-resolution using sparse regression and natural image prior, IEEE Trans. Pattern Anal. Mach. Intell., 32, 6, 1127-1133 (2010) |
| [28] | Konstantin, P.; Mattias, G., Necessary conditions for a class of bilevel optimal control problems exploiting the value function, Pure Appl. Funct. Anal., 1, 4, 505-524 (2016) · Zbl 1355.49019 |
| [29] | Kumar, N.; Verma, R.; Sethi, A., Convolutional neural networks for wavelet domain super resolution, Pattern Recognit. Lett., 90, 65-71 (2017) |
| [30] | Laghrib, A.; Chakib, A.; Hadri, A.; Hakim, A., A nonlinear fourth-order PDE for multi-frame image super-resolution enhancement, Discret. Contin. Dyn. Syst. B, 25, 1, 415 (2020) · Zbl 1468.94020 |
| [31] | Laghrib, A.; Ezzaki, M.; El Rhabi, M.; Hakim, A.; Monasse, P.; Raghay, S., Simultaneous deconvolution and denoising using a second order variational approach applied to image super resolution, Comput. Vis. Image Underst., 168, 50-63 (2018) |
| [32] | Laghrib, A.; Ghazdali, A.; Hakim, A.; Raghay, S., A multi-frame super-resolution using diffusion registration and a nonlocal variational image restoration, Comput. Math. Appl., 72, 9, 2535-2548 (2016) · Zbl 1408.94329 |
| [33] | Laghrib, A.; Hadri, A.; Hakim, A., An edge preserving high-order PDE for multiframe image super-resolution, J. Franklin Inst., 356, 11, 5834-5857 (2019) · Zbl 1455.94026 |
| [34] | Laghrib, A.; Hadri, A.; Hakim, A.; Raghay, S., A new multiframe super-resolution based on nonlinear registration and a spatially weighted regularization, Inf. Sci., 493, 34-56 (2019) · Zbl 1451.94010 |
| [35] | Laghrib, A.; Hakim, A.; Raghay, S., A combined total variation and bilateral filter approach for image robust super resolution, EURASIP J. Image Video Process., 2015, 1, 1-10 (2015) |
| [36] | Laghrib, A.; Hakim, A.; Raghay, S., An iterative image super-resolution approach based on Bregman distance, Signal Process. Image Commun., 58, 24-34 (2017) |
| [37] | Lang, LF; Neumayer, S.; Öktem, O.; Schönlieb, C-B, Template-based image reconstruction from sparse tomographic data, Appl. Math. Optim., 82, 1081-1109 (2020) · Zbl 1461.49048 |
| [38] | Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4681-4690 |
| [39] | Lucas, A.; Lopez-Tapia, S.; Molina, R.; Katsaggelos, AK, Generative adversarial networks and perceptual losses for video super-resolution, IEEE Trans. Image Process., 28, 7, 3312-3327 (2019) |
| [40] | Maiseli, BJ; Ally, N.; Gao, H., A noise-suppressing and edge-preserving multiframe super-resolution image reconstruction method, Signal Process. Image Communi., 34, 1-13 (2015) |
| [41] | Papafitsoros, K.; Schönlieb, C-B, A combined first and second order variational approach for image reconstruction, J. Math. Imaging Vis., 48, 2, 308-338 (2014) · Zbl 1362.94009 |
| [42] | Peyré, G.; Bougleux, S.; Cohen, LD, Non-local regularization of inverse problems, Inverse Probl. Imaging, 5, 2, 511-530 (2011) · Zbl 1223.68116 |
| [43] | Rasti, P., Demirel, H., and Anbarjafari, G. (2014). Improved iterative back projection for video super-resolution. In 2014 22nd Signal Processing and Communications Applications Conference (SIU), pages 552-555. IEEE |
| [44] | Robinson, M. D., Chiu, S. J., Toth, C. A., Izatt, J. A., Lo, J. Y., and Farsiu, S. (2017). New applications of super-resolution in medical imaging. In Super-Resolution Imaging, pages 401-430. CRC Press |
| [45] | Su, H.; Jiang, N.; Wu, Y.; Zhou, J., Single image super-resolution based on space structure learning, Pattern Recognit. Lett., 34, 16, 2094-2101 (2013) |
| [46] | Tsai, R., Multiframe image restoration and registration, Adv. Comput. Vis. Image Process., 1, 317-339 (1984) |
| [47] | Wang, Z.; Bovik, AC; Sheikh, HR; Simoncelli, EP, Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process., 13, 4, 600-612 (2004) |
| [48] | Wheeler, F. W., Hoctor, R. T., and Barrett, E. B. (2005). Super-resolution image synthesis using projections onto convex sets in the frequency domain. In Computational Imaging III, volume 5674, pages 479-490. International Society for Optics and Photonics |
| [49] | Yang, S.; Wang, M.; Sun, Y.; Sun, F.; Jiao, L., Compressive sampling based single-image super-resolution reconstruction by dual-sparsity and non-local similarity regularizer, Pattern Recognit. Lett., 33, 9, 1049-1059 (2012) |
| [50] | Yue, L.; Shen, H.; Li, J.; Yuan, Q.; Zhang, H.; Zhang, L., Image super-resolution: the techniques, applications, and future, Signal Process., 128, 389-408 (2016) |
| [51] | Zeng, X.; Yang, L., A robust multiframe super-resolution algorithm based on half-quadratic estimation with modified BTV regularization, Digital Signal Process., 23, 1, 98-109 (2013) |
| [52] | Zhang, K.; Gao, X.; Tao, D.; Li, X., Single image super-resolution with non-local means and steering kernel regression, IEEE Trans. Image Process., 21, 11, 4544-4556 (2012) · Zbl 1373.94482 |
| [53] | Zhang, Y.; Liu, J.; Yang, W.; Guo, Z., Image super-resolution based on structure-modulated sparse representation, IEEE Trans. Image Process., 24, 9, 2797-2810 (2015) · Zbl 1408.94825 |
| [54] | Zhao, L.; Bai, H.; Liang, J.; Zeng, B.; Wang, A.; Zhao, Y., Simultaneous color-depth super-resolution with conditional generative adversarial networks, Pattern Recognit., 88, 356-369 (2019) |
| [55] | Zhao, W., Sawhney, H., Hansen, M., and Samarasekera, S. (2002). Super-fusion: a super-resolution method based on fusion. In Object Recognition Supported by User Interaction for Service Robots, volume 2, pages 269-272. IEEE |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
