×

Constrained and unconstrained deep image prior optimization models with automatic regularization. (English) Zbl 1507.65104

Summary: Deep Image Prior (DIP) is currently among the most efficient unsupervised deep learning based methods for ill-posed inverse problems in imaging. This novel framework relies on the implicit regularization provided by representing images as the output of generative Convolutional Neural Network (CNN) architectures. So far, DIP has been shown to be an effective approach when combined with classical and novel regularizers. Unfortunately, to obtain appropriate solutions, all the models proposed up to now require an accurate estimate of the regularization parameter. To overcome this difficulty, we consider a locally adapted regularized unconstrained model whose local regularization parameters are automatically estimated for additively separable regularizers. Moreover, we propose a novel constrained formulation in analogy to Morozov’s discrepancy principle which enables the application of a broader range of regularizers. Both the unconstrained and the constrained models are solved via the proximal gradient descent-ascent method. Numerical results demonstrate the robustness with respect to image content, noise levels and hyperparameters of the proposed models on both denoising and deblurring of simulated as well as real natural and medical images.

MSC:

65K10 Numerical optimization and variational techniques
68T07 Artificial neural networks and deep learning
68U10 Computing methodologies for image processing
90C06 Large-scale problems in mathematical programming
90C25 Convex programming
92C55 Biomedical imaging and signal processing
94A08 Image processing (compression, reconstruction, etc.) in information and communication theory
Full Text: DOI

References:

[1] Arridge, S.; Maass, P.; Öktem, O.; Schönlieb, CB, Solving inverse problems using data-driven models, Acta Numer., 28, 1-174 (2019) · Zbl 1429.65116 · doi:10.1017/S0962492919000059
[2] Baguer, DO; Leuschner, J.; Schmidt, M., Computed tomography reconstruction using Deep Image Prior and learned reconstruction methods, Inverse Prob., 36, 9 (2020) · Zbl 1460.68093 · doi:10.1088/1361-6420/aba415
[3] Bertero, M., Boccacci, P.: Introduction to inverse problems in imaging. CRC press (1998) · Zbl 0914.65060
[4] Bevilacqua, M., Roumy, A., Guillemot, C., line Alberi Morel, M.: Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. In: Proceedings of the British Machine Vision Conference, pp. 135.1-135.10. BMVA Press (2012). doi:10.5244/C.26.135
[5] Bortolotti, V.; Brown, R.; Fantazzini, P.; Landi, G.; Zama, F., Uniform Penalty inversion of two-dimensional NMR relaxation data, Inverse Prob., 33, 1, 015003 (2016) · Zbl 1361.65098 · doi:10.1088/1361-6420/33/1/015003
[6] Boţ, R.I., Böhm, A.: Alternating proximal-gradient steps for (stochastic) nonconvex-concave minimax problems. arXiv preprint arXiv:2007.13605 (2020)
[7] Cascarano, P., Comes, M.C., Mencattini, A., Parrini, M.C., Piccolomini, E.L., Martinelli, E.: Recursive deep prior video: a super resolution algorithm for time-lapse microscopy of organ-on-chip experiments. Medical Image Analysis p. 102124 (2021)
[8] Cascarano, P., Sebastiani, A., Comes, M.C., Franchini, G., Porta, F.: Combining weighted total variation and deep image prior for natural and medical image restoration via admm. In: 2021 21st International Conference on Computational Science and Its Applications (ICCSA), pp. 39-46 (2021). doi:10.1109/ICCSA54496.2021.00016
[9] Chen, Z., Zhou, Y., Xu, T., Liang, Y.: Proximal gradient descent-ascent: variable convergence under Kł Geometry. In: International Conference on Learning Representations (2021)
[10] Cheng, Z., Gadelha, M., Maji, S., Sheldon, D.: A Bayesian perspective on the Deep Image Prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5443-5451 (2019)
[11] Dittmer, S.; Kluth, T.; Maass, P.; Baguer, DO, Regularization by architecture: a deep prior approach for inverse problems, J. Math. Imag. Vis., 62, 3, 456-470 (2020) · Zbl 1434.68505 · doi:10.1007/s10851-019-00923-x
[12] Gan, W., Eldeniz, C., Liu, J., Chen, S., An, H., Kamilov, U.S.: Image Reconstruction for MRI using Deep CNN Priors Trained without Groundtruth. In: 2020 54th Asilomar Conference on Signals, Systems, and Computers, pp. 475-479. IEEE (2020)
[13] Gandelsman, Y., Shocher, A., Irani, M.: “Double-DIP”: unsupervised image decomposition via coupled deep-image-priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11026-11035 (2019)
[14] Golub, GH; Heath, M.; Wahba, G., Generalized cross-validation as a method for choosing a good ridge parameter, Technometrics, 21, 2, 215-223 (1979) · Zbl 0461.62059 · doi:10.1080/00401706.1979.10489751
[15] Goyal, B.; Dogra, A.; Agrawal, S.; Sohi, B.; Sharma, A., Image denoising review: from classical to state-of-the-art approaches, Inform. Fusion, 55, 220-244 (2020) · doi:10.1016/j.inffus.2019.09.003
[16] Grasmair, M.: Locally adaptive total variation regularization. In: International Conference on Scale Space and Variational Methods in Computer Vision, pp. 331-342. Springer (2009)
[17] Hansen, PC, Analysis of discrete ill-posed problems by means of the L-curve, SIAM Rev., 34, 4, 561-580 (1992) · Zbl 0770.65026 · doi:10.1137/1034115
[18] He, C.; Hu, C.; Zhang, W.; Shi, B., A fast adaptive parameter estimation for total variation image restoration, IEEE Trans. Image Process., 23, 12, 4954-4967 (2014) · Zbl 1374.94134 · doi:10.1109/TIP.2014.2360133
[19] Immerkaer, J., Fast noise variance estimation, Comput. Vis. Image Underst., 64, 2, 300-302 (1996) · doi:10.1006/cviu.1996.0060
[20] Jin, KH; McCann, MT; Froustey, E.; Unser, M., Deep convolutional neural network for inverse problems in imaging, IEEE Trans. Image Process., 26, 9, 4509-4522 (2017) · Zbl 1409.94275 · doi:10.1109/TIP.2017.2713099
[21] Karl, W.C.: Regularization in image restoration and reconstruction. In: Handbook of Image and Video Processing, pp. 183-V. Elsevier (2005)
[22] Kingma, D.P., Ba, J.: ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
[23] Kobler, E., Effland, A., Kunisch, K., Pock, T.: Total deep variation for linear inverse problems. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)
[24] Kokil, P.; Pratap, T., Additive white gaussian noise level estimation for natural images using linear scale-space features, Circ. Syst. Signal Process., 40, 1, 353-374 (2021) · doi:10.1007/s00034-020-01475-x
[25] Krull, A., Buchholz, T.O., Jug, F.: Noise2void-learning denoising from single noisy images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2129-2137 (2019)
[26] Lehtinen, J., Munkberg, J., Hasselgren, J., Laine, S., Karras, T., Aittala, M., Aila, T.: Noise2Noise: learning image restoration without clean data. In: International Conference on Machine Learning, pp. 2965-2974. PMLR (2018)
[27] Lin, T., Jin, C., Jordan, M.I.: On Gradient Descent Ascent for Nonconvex-Concave Minimax problems. arXiv preprint arXiv:1906.00331 (2021)
[28] Lin, Y.; Wohlberg, B.; Guo, H., UPRE method for total variation parameter selection, Signal Process., 90, 8, 2546-2551 (2010) · Zbl 1194.94109 · doi:10.1016/j.sigpro.2010.02.025
[29] Liu, J., Sun, Y., Xu, X., Kamilov, U.S.: Image restoration using Total Variation regularized Deep Image Prior. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7715-7719. IEEE (2019)
[30] Lucas, A.; Iliadis, M.; Molina, R.; Katsaggelos, AK, Using deep neural networks for inverse problems in imaging: beyond analytical methods, IEEE Signal Process. Mag., 35, 1, 20-36 (2018) · doi:10.1109/MSP.2017.2760358
[31] Mataev, G., Milanfar, P., Elad, M.: DeepRED: Deep Image Prior powered by RED. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 1-10 (2019)
[32] McCann, MT; Jin, KH; Unser, M., Convolutional neural networks for inverse problems in imaging: a review, IEEE Signal Process. Mag., 34, 6, 85-95 (2017) · doi:10.1109/MSP.2017.2739299
[33] Ng, MK; Weiss, P.; Yuan, X., Solving constrained total-variation image restoration and reconstruction problems via alternating direction methods, SIAM J. Sci. Comput., 32, 5, 2710-2736 (2010) · Zbl 1217.65071 · doi:10.1137/090774823
[34] Romano, Y.; Elad, M.; Milanfar, P., The little engine that could: regularization by denoising (red), SIAM J. Imag. Sci., 10, 4, 1804-1844 (2017) · Zbl 1401.62101 · doi:10.1137/16M1102884
[35] Rudin, LI; Osher, S.; Fatemi, E., Nonlinear total variation based noise removal algorithms, Physica D, 60, 1-4, 259-268 (1992) · Zbl 0780.49028 · doi:10.1016/0167-2789(92)90242-F
[36] Sagheer, SVM; George, SN, A review on medical image denoising algorithms, Biomed. Signal Process. Control, 61, 102036 (2020) · doi:10.1016/j.bspc.2020.102036
[37] Scherzer, O., Grasmair, M., Grossauer, H., Haltmeier, M., Lenzen, F.: Variational methods in imaging. Applied mathematical sciences ; v. 167. Springer, New York (2009) · Zbl 1177.68245
[38] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep Image Prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446-9454 (2018)
[39] Van Veen, D., Jalal, A., Soltanolkotabi, M., Price, E., Vishwanath, S., Dimakis, A.G.: Compressed sensing with deep image prior and learned regularization. arXiv preprint arXiv:1806.06438 (2018)
[40] Wen, YW; Yip, AM, Adaptive parameter selection for total variation image deconvolution, Numer. Math. Theor. Meth. Appl, 2, 4, 427-438 (2009) · Zbl 1212.65255 · doi:10.4208/nmtma.2009.m9005s
[41] Willemink, MJ; Koszek, WA; Hardell, C.; Wu, J.; Fleischmann, D.; Harvey, H.; Folio, LR; Summers, RM; Rubin, DL; Lungren, MP, Preparing medical imaging data for machine learning, Radiology, 295, 1, 4-15 (2020) · doi:10.1148/radiol.2020192224
[42] Yan, T.; Wong, PK; Ren, H.; Wang, H.; Wang, J.; Li, Y., Automatic distinction between COVID-19 and common pneumonia using multi-scale convolutional neural network on chest CT scans, Chaos, Solit. Fract., 140, 110153 (2020) · doi:10.1016/j.chaos.2020.110153
[43] Zanni, L.; Benfenati, A.; Bertero, M.; Ruggiero, V., Numerical methods for parameter estimation in poisson data inversion, J. Math. Imag. Vis., 52, 3, 397-413 (2015) · Zbl 1327.65117 · doi:10.1007/s10851-014-0553-9
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.