Image reconstruction in dynamic inverse problems with temporal models. (English) Zbl 07914383
Chen, Ke (ed.) et al., Handbook of mathematical models and algorithms in computer vision and imaging. Mathematical imaging and vision. Springer Reference. Cham: Springer. 1707-1737 (2023).
Summary: This paper surveys variational approaches for image reconstruction in dynamic inverse problems. Emphasis is on variational methods that rely on parametrized temporal models. These are encoded here as diffeomorphic deformations with time-dependent parameters or as motion-constrained reconstructions where the motion model is given by a differential equation. The survey also includes recent developments in integrating deep learning for solving these computationally demanding variational methods. Examples are given for 2D dynamic tomography, but methods apply to general inverse problems.
For the entire collection see [Zbl 1527.94003].
For the entire collection see [Zbl 1527.94003].
MSC:
| 94A08 | Image processing (compression, reconstruction, etc.) in information and communication theory |
| 68U10 | Computing methodologies for image processing |
| 68T07 | Artificial neural networks and deep learning |
Keywords:
image registration; indirect registration; inverse problems; regularization; tomography; image reconstruction; deep learningReferences:
| [1] | Arguillere, S., Trélat, E., Trouvé, A., Younes, L.: Shape deformation analysis from the optimal control viewpoint. Journal de Mathématiques Pures et Appliqués 104(1), 139-178 (2015) · Zbl 1319.49064 |
| [2] | Arridge, S., Hauptmann, A.: Networks for nonlinear diffusion problems in imaging. J. Math. Imag. Vis. 62(3), 471-487 (2020). https://doi.org/10.1007/s10851-019-00901-3 · Zbl 1434.68496 · doi:10.1007/s10851-019-00901-3 |
| [3] | Aviles-Rivero, A.I., Williams, G., Graves, M.J., Schönlieb, C.B.: Compressed sensing plus motion (CS+M): a new perspective for improving undersampled mr image reconstruction. ArXiv preprint 1810.10828 (2018) |
| [4] | Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: VoxelMorph: a learning framework for deformable medical image registration. IEEE Trans. Med. Imag. 38(8), 1788-1800 (2019) |
| [5] | Beg, F.M., Miller, M.I., Trouvé, A., Younes, L.: Computing large deformation metric mappings via geodesic flow of diffeomorphisms. Int. J. Comput. Vis. 61(2), 139-157 (2005) · Zbl 1477.68459 |
| [6] | Benning, M., Burger, M.: Modern regularization methods for inverse problems. Acta Numer. 27, 1-111 (2018) · Zbl 1431.65080 |
| [7] | Bertero, M., Lantéri, H., Zanni, L.: Iterative image reconstruction: a point of view. In: Censor, Y., Jiang, M., Louis, A.K. (eds.) Interdisciplinary Workshop on Mathematical Methods in Biomedical Imaging and Intensity-Modulated Radiation (IMRT), Pisa, pp. 37-63 (2008) |
| [8] | Bubba, T.A., März, M., Purisha, Z., Lassas, M., Siltanen, S.: Shearlet-based regularization in sparse dynamic tomography. In: Wavelets and Sparsity XVII, vol. 10394, p. 103940Y. International Society for Optics and Photonics, Bellinghams (2017) |
| [9] | Burger, M., Dirks, H., Frerking, L., Hauptmann, A., Helin, T., Siltanen, S.: A variational reconstruction method for undersampled dynamic x-ray tomography based on physical motion models. Inverse Probl. 33(12), 124008 (2017) · Zbl 1381.92041 |
| [10] | Burger, M., Dirks, H., Schönlieb, C.B.: A variational model for joint motion estimation and image reconstruction. SIAM J. Imag. Sci. 11(1), 94-128 (2018) · Zbl 1437.94009 |
| [11] | Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imag. Vis. 40(1), 120-145 (2011) · Zbl 1255.68217 |
| [12] | Charon, N., Charlier, B., Trouvé, A.: Metamorphoses of functional shapes in Sobolev spaces. Found. Comput. Math. 18(6), 1535-1596 (2018). https://doi.org/10.1007/s10208-018-9374-3 · Zbl 1403.49021 · doi:10.1007/s10208-018-9374-3 |
| [13] | Chen, C., Öktem, O.: Indirect image registration with large diffeomorphic deformations. SIAM J. Imag. Sci. 11(1), 575-617 (2018) · Zbl 1401.94013 |
| [14] | Chen, B., Abascal, J., Soleimani, M.: Extended joint sparsity reconstruction for spatial and temporal ERT imaging. Sensors 18(11), 4014 (2018) |
| [15] | Chen, C., Gris, B., Öktem, O.: A new variational model for joint image reconstruction and motion estimation in spatiotemporal imaging. SIAM J. Imag. Sci. 12(4), 1686-1719 (2019) · Zbl 1434.68585 |
| [16] | De Schryver, T., Dierick, M., Heyndrickx, M., Van Stappen, J., Boone, M.A., Van Hoorebeke, L., Boone, M.N.: Motion compensated micro-CT reconstruction for in-situ analysis of dynamic processes. Sci. Rep. 8, 7655 (10pp) (2018) |
| [17] | Dirks, H.: Variational methods for joint motion estimation and image reconstruction. Phd thesis, Institute for Computational and Applied Mathematics, University of Münster (2015) · Zbl 1321.68002 |
| [18] | Djurabekova, N., Goldberg, A., Hauptmann, A., Hawkes, D., Long, G., Lucka, F., Betcke, M.: Application of proximal alternating linearized minimization (PALM) and inertial PALM to dynamic 3D CT. In: 15th International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine, vol. 11072, p. 1107208. International Society for Optics and Photonics, Bellingham (2019) |
| [19] | Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., Brox, T.: Flownet: learning optical flow with convolutional networks. In: IEEE International Conference on Computer Vision, pp. 2758-2766 (2015) |
| [20] | Feng, L., Grimm, R., Block, K.T., Chandarana, H., Kim, S., Xu, J., Axel, L., Sodickson, D.K., Otazo, R.: Golden-angle radial sparse parallel MRI: combination of compressed sensing, parallel imaging, and golden-angle radial sampling for fast and flexible dynamic volumetric MRI. Magn. Reson. Med. 72(3), 707-717 (2014) |
| [21] | Frerking, L.: Variational methods for direct and indirect tracking in dynamic imaging. Phd thesis, Institute for Computational and Applied Mathematics, University of Münsternster (2016) · Zbl 1410.94004 |
| [22] | Fu, Y., Lei, Y., Wang, T., Curran, W.J., Liu, T., Yang, X.: Deep learning in medical image registration: a review. ArXiv preprint 1912.12318 (2019) |
| [23] | Glover, G.H.: Overview of functional magnetic resonance imaging. Neurosurg. Clin. 22(2), 133-139 (2011) |
| [24] | Grasmair, M.: Generalized Bregman distances and convergence rates for non-convex regularization methods. Inverse Probl. 26(11), 115014 (2010) · Zbl 1228.65086 |
| [25] | Grenander, U., Miller, M.: Pattern Theory. From Representation to Inference. Oxford University Press, Oxford (2007) · Zbl 1259.62089 |
| [26] | Gris, B., Chen, C., Öktem, O.: Image reconstruction through metamorphosis. Inverse Probl. 36(2), 025001 (27pp) (2020) · Zbl 1447.94005 |
| [27] | Hakkarainen, J., Purisha, Z., Solonen, A., Siltanen, S.: Undersampled dynamic x-ray tomography with dimension reduction kalman filter. IEEE Trans. Comput. Imag. 5(3), 492-501 (2019). https://doi.org/10.1109/TCI.2019.2896527 · doi:10.1109/TCI.2019.2896527 |
| [28] | Haskins G., Kruger, U., Yan, P.: Deep learning in medical image registration: a survey. Mach. Vis. Appl. 31(8) (2020) |
| [29] | Hauptmann, A., Arridge, S., Lucka, F., Muthurangu, V., Steeden, S.A.: Real-time cardiovascular mr with spatio-temporal artifact suppression using deep learning-proof of concept in congenital heart disease. Magn. Reson. Med. 81(2), 1143-1156 (2019) |
| [30] | Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1-3), 185-203 (1981) · Zbl 1497.68488 |
| [31] | Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2462-2470 (2017) |
| [32] | Kofler, A., Dewey, M., Schaeffter, T., Wald, C., Kolbitsch, C.: Spatio-temporal deep learning-based undersampling artefact reduction for 2D radial cine MRI with limited training data. IEEE Trans. Med. Imag. 39(3), 703-717 (2019). https://doi.org/10.1109/TMI.2019.2930318 · doi:10.1109/TMI.2019.2930318 |
| [33] | Kuang, D., Schmah, T.: FAIM -a ConvNet method for unsupervised 3D medical image registration. ArXiv preprint 1811.09243 (2018) |
| [34] | Kushnarev, S., Qiu, A., Younes, L. (eds.): Mathematics of Shapes and Applications. World Scientific, Singapore (2020) · Zbl 1436.58001 |
| [35] | Kwong, Y., Mel, A.O., Wheeler, G., Troupis, J.M.: Four-dimensional computed tomography (4DCT): a review of the current status and applications. J. Med. Imag. Radiat. Oncol. 59(5), 545-554 (2015) |
| [36] | Lang, L.F., Dutta, N., Scarpa, E., Sanson, B., Schönlieb, C.B., Étienne, J.: Joint motion estimation and source identification using convective regularisation with an application to the analysis of laser nanoablations. bioRxiv 686261 (2019a) |
| [37] | Lang, L.F., Neumayer, S., Öktem, O., Schönlieb, C.B.: Template-based image reconstruction from sparse tomographic data. Appl. Math. Optim. (2019b). https://doi.org/10.1007/s00245-019-09573-2 · Zbl 1461.49048 · doi:10.1007/s00245-019-09573-2 |
| [38] | Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A.W.M., van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60-88 (2017) |
| [39] | Liu, J., Aviles-Rivero, A.I., Ji, H., Schönlieb, C.B.: Rethinking medical image reconstruction via shape prior, going deeper and faster: deep joint indirect registration and reconstruction. To appear in Medical Image Analysis, preprint on arxiv 1912.07648 (2019) |
| [40] | Long, Z., Lu, Y., Dong, B.: Pde-net 2.0: learning pdes from data with a numeric-symbolic hybrid deep network. J. Comput. Phys. 399, 108925 (2019) · Zbl 1454.65131 |
| [41] | Lucka, F., Huynh, N., Betcke, M., Zhang, E., Beard, P., Cox, B., Arridge, S.: Enhancing compressed sensing 4D photoacoustic tomography by simultaneous motion estimation. SIAM J. Imag. Sci. 11(4), 2224-2253 (2018) · Zbl 1420.92060 |
| [42] | Lustig, M., Santos, J.M., Donoho, D.L., Pauly, J.M.: kt SPARSE: high frame rate dynamic MRI exploiting spatio-temporal sparsity. In: 13th Annual Meeting of ISMRM, Seattle, vol. 2420 (2006) |
| [43] | Miller, M.I., Trouvé, A., Younes, L.: Geodesic shooting for computational anatomy. J. Math. Imag. Vis. 24(2), 209-228 (2006) · Zbl 1478.92084 |
| [44] | Mokso, R., Schwyn, D.A., Walker, S.M., Doube, M., Wicklein, M., Müller, T., Stampanoni, M., Taylor, G.K., Krapp, H.G.: Four-dimensional in vivo x-ray microscopy with projection-guided gating. Sci. Rep. 5, 8727 (6pp) (2014) |
| [45] | Mussabayeva, A., Pisov, M., Kurmukov, A., Kroshnin, A., Denisova, Y., Shen, L., Cong, S., Wang, L., Gutman, B.: Diffeomorphic metric learning and template optimization for registration-based predictive models. In: Zhu, D., Yan, J., Huang, H., Shen, L., Thompson, P.M., Westin, C.F., Pennec, X., Joshi, S., Nielsen, M., Fletcher, T., Durrleman, S., Sommer, S. (eds.) Multimodal Brain Image Analysis and Mathematical Foundations of Computational Anatomy (MBIA 2019/MFCA 2019). Lecture Notes in Computer Science, vol. 11846, pp. 151-161. Springer Nature Switzerland, Cham (2019) |
| [46] | Niemi, E., Lassas, M., Kallonen, A., Harhanen, L., Hämäläinen, K., Siltanen, S.: Dynamic multi-source x-ray tomography using a spacetime level set method. J. Comput. Phys. 291, 218-237 (2015) · Zbl 1349.92083 |
| [47] | Niethammer, M., Kwitt, R., Vialard, F.X.: Metric learning for image registration. In: Computer Vision and Pattern Recognition (CVPR 2019) (2019) |
| [48] | Pennec, X., Sommer, S., Fletcher, T. (eds.): Riemannian Geometric Statistics in Medical Image Analysis. Academic Press, Cambridge (2020) · Zbl 1428.92004 |
| [49] | Pouchol, C., Verdier, O., Öktem, O.: Spatiotemporal PET reconstruction using ML-EM with learned diffeomorphic deformation. In: Knoll, F., Maier, A., Rueckert, D., Ye, J.C. (eds.) Machine Learning for Medical Image Reconstruction. Second International Workshop, MLMIR 2019, Held in Conjunction with MICCAI 2019. Lecture Notes in Computer Science, vol. 11905, pp. 151-162. Springer (2019). Selected for oral presentation |
| [50] | Qin, C., Bai, W., Schlemper, J., Petersen, S.E., Piechnik, S.K., Neubauer, S., Rueckert, D.: Joint learning of motion estimation and segmentation for cardiac mr image sequences. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 472-480. Springer (2018) |
| [51] | Rahmim, A., Lodge, M.A., Karakatsanis, N.A., Panin, V.Y., Zhou, Y., McMillan, A., Cho, S., Zaidi, H., Casey, M.E., Wahl, R.L.: Dynamic whole-body PET imaging: principles, potentials and applications. Eur. J. Nucl. Med. Mol. Imag. 46, 501-518 (2019) |
| [52] | Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D: Nonlinear Phenom. 60(1-4), 259-268 (1992) · Zbl 0780.49028 |
| [53] | Ruhlandt, A., Töpperwien, M., Krenkel, M., Mokso, R., Salditt, T.: Four dimensional material movies: high speed phase-contrast tomography by backprojection along dynamically curved paths. Sci. Rep. 7, 6487 (9pp) (2017) |
| [54] | Salman, H., Yadollahpour, P., Fletcher, T., Batmanghelich, K.: Deep diffeomorphic normalizing flows. ArXiv preprint 1810.03256 (2018) |
| [55] | Scherzer, O., Grasmair, M., Grossauer, H., Haltmeier, M., Lenzen, F.: Variational Methods in Imaging. Applied Mathematical Sciences, vol. 167. Springer, New York (2009) · Zbl 1177.68245 |
| [56] | Schlemper, J., Caballero, J., Hajnal, J.V., Price, A.N., Rueckert, D.: A deep cascade of convolu-tional neural networks for dynamic mr image reconstruction. IEEE Trans. Med. Imag. 37(2), 491-503 (2017) |
| [57] | Schmitt, U., Louis, A.K.: Efficient algorithms for the regularization of dynamic inverse problems: I. Theory. Inverse Probl. 18(3), 645 (2002) · Zbl 1003.65049 |
| [58] | Schmitt, U., Louis, A.K., Wolters, C., Vauhkonen, M.: Efficient algorithms for the regularization of dynamic inverse problems: II. Applications. Inverse Probl. 18(3), 659 (2002) · Zbl 1003.65050 |
| [59] | Shen, D., Wu, G., Suk, H.I.: Deep learning in medical image analysis. Ann. Rev. Biomed. Eng. 19, 221-248 (2017) |
| [60] | Steeden, J.A., Kowalik, G.T., Tann, O., Hughes, M., Mortensen, K.H., Muthurangu, V.: Real-time assessment of right and left ventricular volumes and function in children using high spatiotemporal resolution spiral bssfp with compressed sensing. J. Cardiovasc. Magn. Reson. 20(1), 79 (2018) |
| [61] | Trouvé, A., Younes, L.: Metamorphoses through Lie group action. Found. Comput. Math. 5(2), 173-198 (2005) · Zbl 1099.68116 |
| [62] | Trouvé, A., Younes, L.: Shape spaces. In: Otmar, S. (ed.) Handbook of Mathematical Methods in Imaging, pp. 1759-1817. Springer, New York (2015) · Zbl 1331.68261 |
| [63] | Yang, G., Hipwell, J.H., Hawkes, D.J., Arridge, S.R.: Numerical methods for coupled reconstruc-tion and registration in digital breast tomosynthesis. Ann. Br. Mach. Vis. Assoc. 2013(9), 1-38 (2013) |
| [64] | Yang, X., Kwitt, R., Styner, M., Niethammer, M.: Quicksilver: fast predictive image registration-a deep learning approach. NeuroImage 158, 378-396 (2017) |
| [65] | Younes, L.: Shapes and Diffeomorphisms. Applied Mathematical Sciences, vol. 171, 2nd edn. Springer, Heidelberg (2019) · Zbl 1423.53002 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
