×

NF-ULA: normalizing flow-based unadjusted Langevin algorithm for imaging inverse problems. (English) Zbl 1541.62065

Summary: Bayesian methods for solving inverse problems are a powerful alternative to classical methods since the Bayesian approach offers the ability to quantify the uncertainty in the solution. In recent years, data-driven techniques for solving inverse problems have also been remarkably successful, due to their superior representation ability. In this work, we incorporate data-based models into a class of Langevin-based sampling algorithms for Bayesian inference in imaging inverse problems. In particular, we introduce NF-ULA (normalizing flow-based unadjusted Langevin algorithm), which involves learning a normalizing flow (NF) as the image prior. We use NF to learn the prior because a tractable closed-form expression for the log prior enables the differentiation of it using autograd libraries. Our algorithm only requires a normalizing flow-based generative network, which can be pretrained independently of the considered inverse problem and the forward operator. We perform theoretical analysis by investigating the well-posedness and nonasymptotic convergence of the resulting NF-ULA algorithm. The efficacy of the proposed NF-ULA algorithm is demonstrated in various image restoration problems such as image deblurring, image inpainting, and limited-angle X-ray computed tomography reconstruction. NF-ULA is found to perform better than competing methods for severely ill-posed inverse problems.

MSC:

62F15 Bayesian inference
49N45 Inverse problems in optimal control
94A08 Image processing (compression, reconstruction, etc.) in information and communication theory
Full Text: DOI

References:

[1] Adler, J. and Öktem, O., Learned primal-dual reconstruction, IEEE Trans. Med. Imaging, 37 (2018), pp. 1322-1332.
[2] Aguerrebere, C., Almansa, A., Delon, J., Gousseau, Y., and Musé, P., A Bayesian hyperprior approach for joint image denoising and interpolation, with an application to HDR imaging, IEEE Trans. Comput. Imaging, 3 (2017), pp. 633-646.
[3] Altekrüger, F., Denker, A., Hagemann, P., Hertrich, J., Maass, P., and Steidl, G., Patchnr: Learning from very few images by patch normalizing flow regularization, Inverse Problems, 39 (2023), 064006.
[4] Altekrüger, F., Hagemann, P., and Steidl, G., Conditional generative models are provably robust: Pointwise guarantees for Bayesian inverse problems, Trans. Mach. Learn. Res., (2023).
[5] Amos, B., Xu, L., and Kolter, J. Z., Input convex neural networks, Proc. Mach. Learn. Res. (PMLR), 70 (2017), pp. 146-155.
[6] Ardizzone, L., Bungert, T., Draxler, F., Köthe, U., Kruse, J., Schmier, R., and Sorrenson, P., vislearn/FrEIA, https://github.com/vislearn/FrEIA.
[7] Arridge, S., Maass, P., Öktem, O., and Schönlieb, C.-B., Solving inverse problems using data-driven models, Acta Numer., 28 (2019), pp. 1-174. · Zbl 1429.65116
[8] Asim, M., Daniels, M., Leong, O., Ahmed, A., and Hand, P., Invertible generative models for inverse problems: Mitigating representation error and dataset bias, Proc. Mach. Learn. Res. (PMLR), 119 (2020), pp. 399-409.
[9] Bauschke, H. H. and Combettes, P. L.,Convex Analysis and Monotone Operator Theory in Hilbert Spaces, , Springer, New York, 2011. · Zbl 1218.47001
[10] Benning, M. and Burger, M., Modern regularization methods for inverse problems, Acta Numer., 27 (2018), pp. 1-111. · Zbl 1431.65080
[11] Blake, A., Kohli, P., and Rother, C., Markov Random Fields for Vision and Image Processing, MIT Press, Cambridge, MA, 2011. · Zbl 1236.68001
[12] Blei, D. M., Kucukelbir, A., and McAuliffe, J. D., Variational inference: A review for statisticians, J. Amer. Stat. Assoc., 112 (2017), pp. 859-877.
[13] Chambolle, A., Caselles, V., Cremers, D., Novaga, M., and Pock, T., An introduction to total variation for image analysis, Theoret. Found. Numer. Methods Sparse Recovery, 9 (2010), 227.
[14] Cheng, X., Chatterji, N. S., Abbasi-Yadkori, Y., Bartlett, P. L., and Jordan, M. I., Sharp convergence rates for Langevin dynamics in the nonconvex setting, J. Mach. Learn. Res., (2019).
[15] Coeurdoux, F., Dobigeon, N., and Chainais, P., Normalizing Flow Sampling with Langevin Dynamics in the Latent Space, preprint, arXiv:2305.12149, 2023.
[16] Coeurdoux, F., Dobigeon, N., and Chainais, P., Plug-and-play split Gibbs sampler: Embedding deep generative priors in Bayesian inference, in International Conference on Learning Representations, , 2024.
[17] Combettes, P. L. and Pesquet, J.-C., Proximal splitting methods in signal processing, in Fixed-Point Algorithms for Inverse Problems in Science and Engineering, Springer, New York, 2011, pp. 185-212. · Zbl 1242.90160
[18] Combettes, P. L. and Wajs, V. R., Signal recovery by proximal forward-backward splitting, Multiscale Model. Simul., 4 (2005), pp. 1168-1200. · Zbl 1179.94031
[19] Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., and Bharath, A. A., Generative adversarial networks: An overview, IEEE Signal Process. Mag., 35 (2018), pp. 53-65.
[20] Dalalyan, A. S., Theoretical guarantees for approximate sampling from smooth and log-concave densities, J. R. Stat. Soc. Ser. B. Stat. Methodol., 79 (2017), pp. 651-676. · Zbl 1411.62030
[21] Dalalyan, A. S. and Karagulyan, A., User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient, Stochastic Process. Appl., 129 (2019), pp. 5278-5311. · Zbl 1428.62316
[22] De Bortoli, V. and Durmus, A., Convergence of Diffusions and Their Discretizations: From Continuous to Discrete Processes and Back, preprint, arXiv:1904.09808, 2019.
[23] Dhariwal, P. and Nichol, A., Diffusion models beat GANs on image synthesis, Adv. Neural Inf. Process. Syst., 34 (2021), pp. 8780-8794.
[24] Dinh, L., Krueger, D., and Bengio, Y., Nice: Non-linear independent components estimation, in International Conference on Learning Representations (Workshop), , 2015.
[25] Dinh, L., Sohl-Dickstein, J., and Bengio, S., Density estimation using real NVP, in International Conference on Learning Representations, , 2017.
[26] Douc, R., Moulines, E., Priouret, P., and Soulier, P., Markov Chains, Springer, Cham, Switzerland, 2018. · Zbl 1429.60002
[27] Durmus, A., Majewski, S., and Miasojedow, B., Analysis of Langevin Monte Carlo via convex optimization, J. Mach. Learn. Res., 20 (2019), pp. 2666-2711. · Zbl 1491.65009
[28] Durmus, A. and Moulines, E., Nonasymptotic convergence analysis for the unadjusted Langevin algorithm, Ann. Appl. Probab., 27 (2017), pp. 1551-1587. · Zbl 1377.65007
[29] Durmus, A., Moulines, E., and Pereyra, M., Efficient Bayesian computation by proximal Markov chain Monte Carlo: When Langevin meets Moreau, SIAM J. Imaging Sci., 11 (2018), pp. 473-506. · Zbl 1401.65016
[30] Efron, B., Tweedie’s formula and selection bias, J. Amer. Statist. Assoc., 106 (2011), pp. 1602-1614. · Zbl 1234.62007
[31] Erdogdu, M. A. and Hosseinzadeh, R., On the convergence of Langevin Monte Carlo: The interplay between tail growth and smoothness, Proc. Mach. Learn. Res. (PMLR), 134 (2021), pp. 1776-1822.
[32] Gilks, W. R., Richardson, S., and Spiegelhalter, D., Markov Chain Monte Carlo in Practice, CRC Press, Boca Raton, FL, 1995.
[33] Gilton, D., Ongie, G., and Willett, R., Learned patch-based regularization for inverse problems in imaging, in 2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), IEEE, Piscataway, NJ, 2019, pp. 211-215.
[34] Gilton, D., Ongie, G., and Willett, R., Deep equilibrium architectures for inverse problems in imaging, IEEE Trans. Comput. Imaging, 7 (2021), pp. 1123-1133.
[35] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y., Generative adversarial nets, Adv. Neural Inf. Process. Syst., 27 (2014).
[36] Hagemann, P., Hertrich, J., and Steidl, G., Stochastic normalizing flows for inverse problems: A Markov chains viewpoint, SIAM/ASA J. Uncertain. Quantif., 10 (2022), pp. 1162-1190. · Zbl 1498.62067
[37] Hagemann, P. and Neumayer, S., Stabilizing invertible neural networks using mixture models, Inverse Problems, 37 (2021), 085002. · Zbl 07375643
[38] Hertrich, J., Neumayer, S., and Steidl, G., Convolutional proximal neural networks and plug-and-play algorithms, Linear Algebra Appl., 631 (2021), pp. 203-234. · Zbl 1534.68203
[39] Ho, J., Jain, A., and Abbeel, P., Denoising diffusion probabilistic models, Adv. Neural Inf. Process. Syst., 33 (2020), pp. 6840-6851.
[40] Hoffman, M. D., Blei, D. M., Wang, C., and Paisley, J., Stochastic variational inference, J. Mach. Learn. Res., 14 (2013), pp. 1303-1347. · Zbl 1317.68163
[41] Houdard, A., Bouveyron, C., and Delon, J., High-dimensional mixture models for unsupervised image denoising (HDMI), SIAM J. Imaging Sci., 11 (2018), pp. 2815-2846. · Zbl 1475.94018
[42] Hurault, S., Leclaire, A., and Papadakis, N., Gradient step denoiser for convergent plug-and-play, in International Conference on Learning Representations, , https://openreview.net/forum?id=fPhKeld3Okz, 2022.
[43] Ikeda, N. and Watanabe, S., Stochastic Differential Equations and Diffusion Processes, Elsevier, Burlington, MA, 2014.
[44] Jaini, P., Kobyzev, I., Yu, Y., and Brubaker, M., Tails of Lipschitz triangular flows, Proc. Mach. Learn. Res. (PMLR), 119 (2020), pp. 4673-4681.
[45] Jin, K. H., McCann, M. T., Froustey, E., and Unser, M., Deep convolutional neural network for inverse problems in imaging, IEEE Trans. Image Process., 26 (2017), pp. 4509-4522. · Zbl 1409.94275
[46] Kaipio, J. and Somersalo, E., Statistical and Computational Inverse Problems, , Springer, New York, 2006.
[47] Kamilov, U. S., Bouman, C. A., Buzzard, G. T., and Wohlberg, B., Plug-and-play methods for integrating physical and learned models in computational imaging: Theory, algorithms, and applications, IEEE Signal Process. Mag., 40 (2023), pp. 85-97.
[48] Karatzas, I., Karatzas, I., Shreve, S., and Shreve, S. E., Brownian Motion and Stochastic Calculus, , Springer, New York, 1991. · Zbl 0734.60060
[49] Karras, T., Laine, S., and Aila, T., A style-based generator architecture for generative adversarial networks, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, , IEEE Computer Society, Los Alamitos, CA, 2019, pp. 4401-4410.
[50] Kingma, D. P. and Dhariwal, P., Glow: Generative flow with invertible \(1\times1\) convolutions, Adv. Neural Inf. Process. Syst., 31 (2018), pp. 10236-10245.
[51] Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M., Improved variational inference with inverse autoregressive flow, Adv. Neural Inf. Process. Syst., 29 (2016), pp. 4743-4751.
[52] Kingma, D. P. and Welling, M., Auto-encoding variational bayes, in International Conference on Learning Representations, , 2014.
[53] Kobler, E., Effland, A., Kunisch, K., and Pock, T., Total deep variation: A stable regularization method for inverse problems, IEEE Trans. Pattern Anal. Mach. Intell., 44 (2021), pp. 9163-9180.
[54] Kobyzev, I., Prince, S. J., and Brubaker, M. A., Normalizing flows: An introduction and review of current methods, IEEE Trans. Pattern Anal. Mach. Intell., 43 (2020), pp. 3964-3979.
[55] Latz, J., On the well-posedness of Bayesian inverse problems, SIAM/ASA J. Uncertain. Quantif., 8 (2020), pp. 451-482. · Zbl 1437.49050
[56] Laumont, R., Bortoli, V. D., Almansa, A., Delon, J., Durmus, A., and Pereyra, M., Bayesian imaging using plug & play priors: When Langevin meets Tweedie, SIAM J. Imaging Sci., 15 (2022), pp. 701-737. · Zbl 1515.65144
[57] Laumont, R., Bortoli, V. D., Almansa, A., Delon, J., Durmus, A., and Pereyra, M., Supplementary materials: Bayesian imaging using plug & play priors: When Langevin meets Tweedie, doi:10.1137/21M1406349/suppl_file/M140634_01.pdf, 2022.
[58] Lehec, J., The Langevin Monte Carlo algorithm in the non-smooth log-concave case, Ann. Appl. Probab., 33 (2023), pp. 4858-4874. · Zbl 07789649
[59] Lempitsky, V., Vedaldi, A., and Ulyanov, D., Deep image prior, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, , IEEE Computer Society, Los Alamitos, CA, 2018, pp. 9446-9454.
[60] Leuschner, J., Schmidt, M., Baguer, D. O., and Maass, P., Lodopab-ct, a benchmark dataset for low-dose computed tomography reconstruction, Scientific Data, 8 (2021), 109.
[61] Liu, Q. and Wang, D., Stein variational gradient descent: A general purpose bayesian inference algorithm, Adv. Neural Inf. Process. Syst., 29 (2016), pp. 2378-2386.
[62] Louchet, C. and Moisan, L., Posterior expectation of the total variation model: Properties and experiments, SIAM J. Imaging Sci., 6 (2013), pp. 2640-2684. · Zbl 1279.68333
[63] Lunz, S., Öktem, O., and Schönlieb, C.-B., Adversarial regularizers in inverse problems, Adv. Neural Inf. Process. Syst., 31 (2018), pp. 8516-8525.
[64] Luu, T. D., Fadili, J., and Chesneau, C., Sampling from non-smooth distributions through Langevin diffusion, Methodol. Comput. Appl. Probab., 23 (2021), pp. 1173-1201. · Zbl 1477.60009
[65] Majka, M. B., Mijatovic, A., and Szpruch, L., Nonasymptotic bounds for sampling algorithms without log-concavity, Ann. Appl. Probab., 30 (2020), pp. 1534-1581, doi:10.1214/19-AAP1535. · Zbl 1466.65008
[66] Meyn, S. P. and Tweedie, R. L., Stability of Markovian processes III: Foster-Lyapunov criteria for continuous-time processes, Adv. Appl. Probab., 25 (1993), pp. 518-548. · Zbl 0781.60053
[67] Monga, V., Li, Y., and Eldar, Y. C., Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing, IEEE Signal Process. Mag., 38 (2021), pp. 18-44.
[68] Mou, W., Flammarion, N., Wainwright, M. J., and Bartlett, P. L., An efficient sampling algorithm for non-smooth composite potentials, J. Mach. Learn. Res., 23 (2022), pp. 1-50.
[69] Mukherjee, S., Carioni, M., Öktem, O., and Schönlieb, C.-B., End-to-end reconstruction meets data-driven regularization for inverse problems, Adv. Neural Inf. Process. Syst., 34 (2021), pp. 21413-21425.
[70] Mukherjee, S., Dittmer, S., Shumaylov, Z., Lunz, S., Öktem, O., and Schönlieb, C.-B., Learned Convex Regularizers for Inverse Problems, preprint, arXiv:2008.02839, 2020.
[71] Neal, R., Bayesian learning via stochastic dynamics, Adv. Neural Inf. Process. Syst., 5 (1992), pp. 475-482.
[72] Ongie, G., Jalal, A., Metzler, C. A., Baraniuk, R. G., Dimakis, A. G., and Willett, R., Deep learning techniques for inverse problems in imaging, IEEE J. Sel. Areas Inf. Theory, 1 (2020), pp. 39-56.
[73] Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C. C., and Luo, P., Exploiting deep generative prior for versatile image restoration and manipulation, IEEE Trans. Pattern Anal. Mach. Intell., 44 (2021), pp. 7474-7489.
[74] Papamakarios, G., Nalisnick, E. T., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B., Normalizing flows for probabilistic modeling and inference, J. Mach. Learn. Res., 22 (2021), pp. 1-64. · Zbl 1543.60021
[75] Papamakarios, G., Pavlakou, T., and Murray, I., Masked autoregressive flow for density estimation, Adv. Neural Inf. Process. Syst., 30 (2017), pp. 2335-2344.
[76] Pereyra, M., Proximal Markov chain Monte Carlo algorithms, Stat. Comput., 26 (2016), pp. 745-760. · Zbl 1505.62315
[77] Pesquet, J.-C., Repetti, A., Terris, M., and Wiaux, Y., Learning maximally monotone operators for image recovery, SIAM J. Imaging Sci., 14 (2021), pp. 1206-1237. · Zbl 1479.47058
[78] Prost, J., Houdard, A., Almansa, A., and Papadakis, N., Learning local regularization for variational image restoration, in Scale Space and Variational Methods in Computer Vision: 8th International Conference, SSVM 2021, Virtual Event, Proceedings, , Springer, Cham, Switzerland, 2021, pp. 358-370.
[79] Rezende, D. and Mohamed, S., Variational inference with normalizing flows, Proc. Mach. Learn. Res. (PMLR), 37 (2015), pp. 1530-1538.
[80] Roberts, G. O. and Tweedie, R. L., Exponential convergence of Langevin distributions and their discrete approximations, Bernoulli, 2 (1996), pp. 341-363. · Zbl 0870.60027
[81] Romano, Y., Elad, M., and Milanfar, P., The little engine that could: Regularization by denoising (RED), SIAM J. Imaging Sci., 10 (2017), pp. 1804-1844. · Zbl 1401.62101
[82] Ronchetti, M., Torchradon: Fast Differentiable Routines for Computed Tomography, preprint, arXiv:2009.14788, 2020.
[83] Ronneberger, O., Fischer, P., and Brox, T., U-net: Convolutional networks for biomedical image segmentation, in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, 2015, Proceedings, Part III 18, , Springer, Cham, Switzerland, 2015, pp. 234-241.
[84] Ryu, E., Liu, J., Wang, S., Chen, X., Wang, Z., and Yin, W., Plug-and-play methods provably converge with properly trained denoisers, Proc. Mach. Learn. Res. (PMLR), 35 (2019), pp. 5546-5557.
[85] Salim, A., Kovalev, D., and Richtárik, P., Stochastic proximal Langevin algorithm: Potential splitting and nonasymptotic rates, Adv. Neural Inf. Process. Syst., 32 (2019), pp. 6653-6664.
[86] Salmona, A., De Bortoli, V., Delon, J., and Desolneux, A., Can push-forward generative models fit multimodal distributions?, Adv. Neural Inf. Process. Syst., 35 (2022), pp. 10766-10779.
[87] Song, Y., Shen, L., Xing, L., and Ermon, S., Solving inverse problems in medical imaging with score-based generative models, in International Conference on Learning Representations, , 2022.
[88] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B., Score-based generative modeling through stochastic differential equations, in International Conference on Learning Representations, , 2021.
[89] Sprungk, B., On the local Lipschitz stability of Bayesian inverse problems, Inverse Problems, 36 (2020), 055015. · Zbl 1469.62216
[90] Sreehari, S., Venkatakrishnan, S. V., Wohlberg, B., Buzzard, G. T., Drummy, L. F., Simmons, J. P., and Bouman, C. A., Plug-and-play priors for bright field electron tomography and sparse interpolation, IEEE Trans. Comput. Imaging, 2 (2016), pp. 408-423.
[91] Steidl, G., Hagemann, P. L., and Hertrich, J., Generalized Normalizing Flows via Markov Chains, , Cambridge University Press, Cambridge, 2022. · Zbl 1532.60001
[92] Stramer, O. and Tweedie, R., Langevin-type models i: Diffusions with given stationary distributions and their discretizations, Methodol. Comput. Appl. Probab., 1 (1999), pp. 283-306. · Zbl 0947.60071
[93] Stuart, A. M., Inverse problems: A Bayesian perspective, Acta Numer., 19 (2010), pp. 451-559. · Zbl 1242.65142
[94] Tan, H. Y., Mukherjee, S., Tang, J., and Schönlieb, C.-B., Provably Convergent Plug-and-Play Quasi-Newton Methods, preprint, arXiv:2303.07271, 2023.
[95] Tarantola, A., Inverse Problem Theory and Methods for Model Parameter Estimation, SIAM, Philadelphia, 2005. · Zbl 1074.65013
[96] Venkatakrishnan, S. V., Bouman, C. A., and Wohlberg, B., Plug-and-play priors for model based reconstruction, in 2013 IEEE Global Conference on Signal and Information Processing, , IEEE, Piscataway, NJ, 2013, pp. 945-948.
[97] Verine, A., Negrevergne, B., Rossi, F., and Chevaleyre, Y., On the expressivity of bi-Lipschitz normalizing flows, Proc. Mach. Learn. Res. (PMLR), 189 (2022), pp. 1054-1069.
[98] Villani, C., Optimal Transport: Old and New, , Springer, Berlin, 2009. · Zbl 1156.53003
[99] Whang, J., Lei, Q., and Dimakis, A., Solving inverse problems with a flow-based noise model, Proc. Mach. Learn. Res. (PMLR), 139 (2021), pp. 11146-11157.
[100] Wu, H., Köhler, J., and Noé, F., Stochastic normalizing flows, Adv. Neural Inf. Process. Syst., 33 (2020), pp. 5933-5944.
[101] Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Shao, Y., Zhang, W., Cui, B., and Yang, M.-H., Diffusion models: A comprehensive survey of methods and applications, ACM Comput. Surv., 56 (2023), 105.
[102] Yu, G., Sapiro, G., and Mallat, S., Solving inverse problems with piecewise linear estimators: From Gaussian mixture models to structured sparsity, IEEE Trans. Image Process., 21 (2011), pp. 2481-2499. · Zbl 1373.94471
[103] Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., and Timofte, R., Plug-and-play image restoration with deep denoiser prior, IEEE Trans. Pattern Anal. Mach. Intell., 44 (2021), pp. 6360-6376.
[104] Zhang, K., Zuo, W., Chen, Y., Meng, D., and Zhang, L., Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Trans. Image Process., 26 (2017), pp. 3142-3155. · Zbl 1409.94754
[105] Zoran, D. and Weiss, Y., From learning models of natural image patches to whole image restoration, in 2011 International Conference on Computer Vision, , IEEE, Piscataway, NJ, 2011, pp. 479-486.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.