Bilevel optimization, deep learning and fractional Laplacian regularization with applications in tomography. (English) Zbl 07371382
Summary: In this work we consider a generalized bilevel optimization framework for solving inverse problems. We introduce fractional Laplacian as a regularizer to improve the reconstruction quality, and compare it with the total variation regularization. We emphasize that the key advantage of using fractional Laplacian as a regularizer is that it leads to a linear operator, as opposed to the total variation regularization which results in a nonlinear degenerate operator. Inspired by residual neural networks, to learn the optimal strength of regularization and the exponent of fractional Laplacian, we develop a dedicated bilevel optimization neural network with a variable depth for a general regularized inverse problem. We illustrate how to incorporate various regularizer choices into our proposed network. As an example, we consider tomographic reconstruction as a model problem and show an improvement in reconstruction quality, especially for limited data, via fractional Laplacian regularization. We successfully learn the regularization strength and the fractional exponent via our proposed bilevel optimization neural network. We observe that the fractional Laplacian regularization outperforms total variation regularization. This is specially encouraging, and important, in the case of limited and noisy data.
MSC:
| 65Rxx | Numerical methods for integral equations, integral transforms |
| 68Txx | Artificial intelligence |
| 94Axx | Communication, information |
| 49Kxx | Optimality conditions |
| 65Fxx | Numerical linear algebra |
Keywords:
bilevel optimization neural network; fractional Laplacian regularization; deep residual learning; imaging science; tomographic reconstruction; inverse problemsReferences:
| [1] | Girard D A 1987 Optimal regularized reconstruction in computerized tomography SIAM J. Sci. Stat. Comput.8 934-50 · Zbl 0638.65094 · doi:10.1137/0908076 |
| [2] | Hamalainen K, Kallonen A, Kolehmainen V, Lassas M, Niinimaki K and Siltanen S 2013 Sparse tomography SIAM J. Sci. Comput.35 B644-65 · Zbl 1275.65089 · doi:10.1137/120876277 |
| [3] | Hsieh J, Nett B, Yu Z, Sauer K, Thibault J-B and Bouman C A 2013 Recent advances in CT image reconstruction Current Radiology Reports1 39-51 · doi:10.1007/s40134-012-0003-7 |
| [4] | Lassas M and Siltanen S 2004 Can one use total variation prior for edge-preserving Bayesian inversion? Inverse Problems20 1537 · Zbl 1062.62260 · doi:10.1088/0266-5611/20/5/013 |
| [5] | Niinimaki K, Lassas M, Hamalainen K, Kallonen A, Kolehmainen V, Niemi E and Siltanen S 2016 Multiresolution parameter choice method for total variation regularized tomography SIAM J. Imag. Sci.9 938-74 · Zbl 1346.65072 · doi:10.1137/15m1034076 |
| [6] | Rudin L, Osher S and Fatemi E 1992 Nonlinear total variation based noise removal algorithms Phys. Nonlinear Phenom.60 259-68 · Zbl 0780.49028 · doi:10.1016/0167-2789(92)90242-f |
| [7] | Shen J and Chan T F 2002 Mathematical models for local nontexture inpaintings SIAM J. Appl. Math.62 1019-43 · Zbl 1050.68157 · doi:10.1137/s0036139900368844 |
| [8] | Tikhonov A N and Arsenin V Y 1977 Solutions of Ill-Posed Problems(Scripta Series in Mathematics) ed F John (Washington, D.C: V. H. Winston & Sons) · Zbl 0354.65028 |
| [9] | Hammernik K, Klatzer T, Kobler E, Recht M P, Sodickson D K, Pock T and Knoll F 2018 Learning a variational network for reconstruction of accelerated mri data Magn. Reson. Med.79 3055-71 · doi:10.1002/mrm.26977 |
| [10] | Yang Y, Sun J, Li H and Xu Z 2016 Deep admm-net for compressive sensing mri (USA,)Proc. of the 30th Int. Conf. on Neural Information Processing Systems, NIPS’16 pp 10-8 |
| [11] | Zhang K, Zuo W, Gu S and Zhang L 2017 Learning deep CNN denoiser prior for image restoration IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2808-17 · doi:10.1109/CVPR.2017.300 |
| [12] | Lucas A, Iliadis M, Molina R and Katsaggelos A K 2018 Using deep neural networks for inverse problems in imaging: Beyond analytical methods IEEE Signal Process. Mag.35 20-36 · doi:10.1109/msp.2017.2760358 |
| [13] | McCann M T, Jin K H and Unser M 2017 Convolutional Neural Networks for Inverse Problems in Imaging: A Review IEEE Signal Process. Mag.34 85-95 · doi:10.1109/msp.2017.2739299 |
| [14] | E W 2019 Machine learning: mathematical theory and scientific applications Not. AMS66 1813-20 · Zbl 1439.68019 · doi:10.1090/noti1994 |
| [15] | Glorot X and Bengio Y Understanding the difficulty of training deep feedforward neural networks (Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010)Proc. of the 13th Int. Conf. on Artificial Intelligence and Statistics ed Y W Teh and M Titterington pp 249-56 |
| [16] | Qiu J, Wu Q, Ding G, Xu Y and Feng S 2016 A survey of machine learning for big data processing EURASIP Journal on Advances in Signal Processing2016 67 · doi:10.1186/s13634-016-0355-x |
| [17] | Ruthotto L and Haber E 2019 Deep Neural Networks Motivated by Partial Differential Equations J. Math. Imag. Vision62 352-64 · Zbl 1434.68522 · doi:10.1007/s10851-019-00903-1 |
| [18] | Wigderson A 2019 Mathematics and Computation (Princeton, NJ: Princeton University Press) · Zbl 1455.68004 |
| [19] | Antil H and Bartels S 2017 Spectral approximation of fractional pdes in image processing and phase field modeling Comput. Methods Appl. Math.17 661-78 · Zbl 1431.65222 · doi:10.1515/cmam-2017-0039 |
| [20] | Kak A C, Slaney M and Wang G 2002 Principles of computerized tomographic imaging Med. Phys.29 107 · doi:10.1118/1.1455742 |
| [21] | Jin K H, McCann M T, Froustey E and Unser M 2017 Deep convolutional neural network for inverse problems in imaging IEEE Trans. Image Process.26 4509-22 · Zbl 1409.94275 · doi:10.1109/tip.2017.2713099 |
| [22] | Shan H, Padole A, Homayounieh F, Kruger U, Khera R D, Nitiwarangkul C, Kalra M K and Wang G 2019 Competitive performance of a modularized deep neural network compared to commercial algorithms for low-dose ct image reconstruction Nature Machine Intelligence1 269 · doi:10.1038/s42256-019-0057-9 |
| [23] | Calatroni L, Cao C, De Los Reyes J C, Schönlieb C-B and Valkonen T 2016 Bilevel approaches for learning of variational imaging models Variational Methods (Berlin: Walter de Gruyter GmbH) pp 252-90 · doi:10.1515/9783110430394-008 |
| [24] | Caffarelli L and Silvestre L 2007 An extension problem related to the fractional Laplacian Comm. Partial Differential Equations32 1245-60 · Zbl 1143.26002 · doi:10.1080/03605300600987306 |
| [25] | Stinga P R and Torrea J L 2010 Extension problem and Harnack’s inequality for some fractional operators Comm. Part. Diff. Eqs.35 2092-122 · Zbl 1209.26013 · doi:10.1080/03605301003735680 |
| [26] | Chung J and Español M I 2017 Learning regularization parameters for general-form Tikhonov Inverse Problems33 074004 · Zbl 1414.68048 · doi:10.1088/1361-6420/33/7/074004 |
| [27] | Hansen P C 1988 Regularization, gsvd and truncated gsvd (generalized singular value decomposition) BIT Numerical Mathematics29 491-504 · Zbl 0682.65021 · doi:10.1007/BF02219234 |
| [28] | Antil H and Rautenberg C 2019 Sobolev spaces with non-muckenhoupt weights, fractional elliptic operators, and applications SIAM J. Math. Anal.51 2479-503 · Zbl 1420.35487 · doi:10.1137/18m1224970 |
| [29] | Weiss C J, van Bloemen Waanders B G and Antil H 2020 Fractional Operators Applied to Geophysical Electromagnetics Geophys. J. Int.220 1242-59 |
| [30] | Antil H, Berry T and Harlim J 2018 Fractional diffusion maps (arXiv:1810.03952) |
| [31] | Bueno-Orovio A, Kay D, Grau V, Rodriguez B and Burrage K 2014 Fractional diffusion models of cardiac electrical propagation: role of structural heterogeneity in dispersion of repolarization J. R. Soc. Interface11 20140352 · doi:10.1098/rsif.2014.0352 |
| [32] | Antil H, Khatri R and Warma M 2019 External optimal control of nonlocal PDEs Inverse Problems35 084003-35 · Zbl 1461.35221 · doi:10.1088/1361-6420/ab1299 |
| [33] | Antil H, Verma D and Warma M 2020 External optimal control of fractional parabolic PDEs ESAIM Control Optim. Calc. Var.26 20 · Zbl 1444.35144 · doi:10.1051/cocv/2020005 |
| [34] | Bougleux S, Elmoataz A and Melkemi M 2009 Local and nonlocal discrete regularization on weighted graphs for image and mesh processing Int. J. Comput. Vis.84 220-36 · Zbl 1481.94008 · doi:10.1007/s11263-008-0159-z |
| [35] | Liu W, Ma X, Zhou Y, Tao D and Cheng J 2019 p-Laplacian regularization for scene recognition IEEE Transactions on Cybernetics49 2927-40 · doi:10.1109/tcyb.2018.2833843 |
| [36] | Magiera J, Ray D, Hesthaven J S and Rohde C 2020 Constraint-aware neural networks for Riemann problems Int. J. Comput. Vis.409 109345 · Zbl 1435.76046 |
| [37] | Hintermüller M and Rautenberg C N 2017 Optimal selection of the regularization function in a weighted total variation model. Part I: Modelling and theory J. Math. Imag. Vis.59 498-514 · Zbl 1382.94015 · doi:10.1007/s10851-017-0744-2 |
| [38] | Hintermüller M, Rautenberg C N, Wu T and Langer A 2017 Optimal selection of the regularization function in a weighted total variation model. Part II: Algorithm, its analysis and numerical tests J. Math. Imag. Vis.59 515-33 · Zbl 1382.94016 · doi:10.1007/s10851-017-0736-2 |
| [39] | Starck J-L, Candès E J and Donoho D L 2002 The curvelet transform for image denoising IEEE Trans. Image Process.11 670-84 · Zbl 1288.94011 · doi:10.1109/TIP.2002.1014998 |
| [40] | Hansen P C 1994 Regularization tools: a matlab package for analysis and solution of discrete ill-posed problems Numer. Algorithms6 1-35 · Zbl 0789.65029 · doi:10.1007/bf02149761 |
| [41] | Ambrosio L, Fusco N and Pallara D 2000 Functions of Bounded Variation and Free Discontinuity Problems (New York: Oxford University Press) · Zbl 0957.49001 |
| [42] | Bartels S and Milicevic M 2017 Alternating direction method of multipliers with variable step sizes (arXiv:1704.06069) |
| [43] | Kinderlehrer D and Stampacchia G 1980 An Introduction to Variational Inequalities and Their Applications (New York: Academic) · Zbl 0457.35001 |
| [44] | Di Z, Leyffer S and Wild S M 2016 Optimization-based approach for joint x-ray fluorescence and transmission tomographic inversion SIAM J. Imag. Sci.9 1 · Zbl 1381.94015 · doi:10.1137/15m1021404 |
| [45] | Radon J 1986 On the determination of functions from their integral values along certain manifolds IEEE Trans. Med. Imaging5 170-6 · doi:10.1109/tmi.1986.4307775 |
| [46] | Colton D and Kress R 2013 Inverse Acoustic and Electromagnetic Scattering Theory(Appl. Math. Sci of vol 93) 3rd edn (New York: Springer) · Zbl 1266.35121 · doi:10.1007/978-1-4614-4942-3 |
| [47] | Nash S G 2000 A survey of truncated-Newton methods J. of Comp. and App. Math.124 45-59 · Zbl 0969.65054 · doi:10.1016/s0377-0427(00)00426-x |
| [48] | Hansen P C and O’Leary D P 1993 The use of the l-curve in the regularization of discrete ill-posed problems SIAM J. Sci. Comput.14 1487-503 · Zbl 0789.65030 · doi:10.1137/0914086 |
| [49] | Vogel C R 1996 Non-convergence of the L-curve regularization parameter selection method Inverse Problems12 535-47 · Zbl 0867.65025 · doi:10.1088/0266-5611/12/4/013 |
| [50] | He K, Zhang X, Ren S and Sun J 2016 Deep residual learning for image recognition IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) pp 770-8 |
| [51] | Wu S, Zhong S and Liu Y 2018 Deep residual learning for image steganalysis Multimed. Tool. Appl.77 10437-53 · doi:10.1007/s11042-017-4440-4 |
| [52] | Chen H, Dou Q, Yu L, Qin J and Voxresnet P-A H 2018 Deep voxelwise residual networks for brain segmentation from 3d mr images NeuroImage170 446-55 · doi:10.1016/j.neuroimage.2017.04.041 |
| [53] | Lee D, Yoo J, Tak S and Ye J C 2018 Deep residual learning for accelerated mri using magnitude and phase networks IEEE Trans. Biomed. Eng.65 1985-95 · doi:10.1109/TBME.2018.2821699 |
| [54] | Bischke B, Bhardwaj P, Gautam A, Helber P, Borth D and Dengel A 2017 Detection of flooding events in social multimedia and satellite imagery using deep neural networks Working Notes Proceedings of the MediaEval 2017(MediaEval Benchmark, September) vol 13-15 (Dublin: MediaEval) |
| [55] | Tai Y, Yang J and Liu X 2017 Image super-resolution via deep recursive residual network IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) pp 2790-8 |
| [56] | Zhang Q, Yuan Q, Zeng C, Li X and Wei Y 2018 Missing data reconstruction in remote sensing image with a unified spatial-temporal-spectral deep convolutional neural network IEEE Trans. Geosci. Remote Sens.56 4274-88 · doi:10.1109/tgrs.2018.2810208 |
| [57] | Kelley C T 1999 Iterative methods for optimization Frontiers in Applied Mathematics (Philadelphia: SIAM) · Zbl 0934.90082 · doi:10.1137/1.9781611970920 |
| [58] | Bottou L, Curtis F E and Nocedal J 2018 Optimization methods for large-scale machine learning SIAM Rev.60 223-311 · Zbl 1397.65085 · doi:10.1137/16m1080173 |
| [59] | Goodfellow I, Bengio Y and Courville A 2016 Deep Learning (Cambridge, MA: MIT Press) · Zbl 1373.68009 |
| [60] | Hastie T, Tibshirani R and Friedman J 2009 The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Berlin: Springer) · Zbl 1273.62005 · doi:10.1007/978-0-387-84858-7 |
| [61] | Higham C F and Higham D J 2019 Deep learning: an introduction for applied mathematicians SIAM Rev.61 860-91 · Zbl 1440.68214 · doi:10.1137/18m1165748 |
| [62] | Sjöberg J 1992 Overtraining regularization, and searching for minimum in neural networks IFAC Proceedings Volumes25 73-8 · doi:10.1016/s1474-6670(17)50715-6 |
| [63] | Antil H, Pfefferer J and Rogovs S 2018 Fractional operators with inhomogeneous boundary conditions: analysis, control, and discretization Commun. Math. Sci.16 1395-426 · Zbl 06996271 · doi:10.4310/cms.2018.v16.n5.a11 |
| [64] | Clarke F H 1975 Generalized gradients and applications Trans. Amer. Math. Soc.205 247-62 · Zbl 0307.26012 · doi:10.1090/s0002-9947-1975-0367131-6 |
| [65] | Austin A P, Di Z, Leyffer S and Wild S M 2019 Simultaneous sensing error recovery and tomographic inversion using an optimization-based approach SIAM J. Sci. Comput.41 B497-521 · Zbl 1420.65141 · doi:10.1137/18m121993x |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
