Task adapted reconstruction for inverse problems. (English) Zbl 07543696
Summary: The paper considers the problem of performing a post-processing task defined on a model parameter that is only observed indirectly through noisy data in an ill-posed inverse problem. A key aspect is to formalize the steps of reconstruction and post-processing as appropriate estimators (non-randomized decision rules) in statistical estimation problems. The implementation makes use of (deep) neural networks to provide a differentiable parametrization of the family of estimators for both steps. These networks are combined and jointly trained against suitable supervised training data in order to minimize a joint differentiable loss function, resulting in an end-to-end task adapted reconstruction method. The suggested framework is generic, yet adaptable, with a plug-and-play structure for adjusting both the inverse problem and the post-processing task at hand. More precisely, the data model (forward operator and statistical model of the noise) associated with the inverse problem is exchangeable, e.g., by using neural network architecture given by a learned iterative method. Furthermore, any post-processing that can be encoded as a trainable neural network can be used. The approach is demonstrated on joint tomographic image reconstruction, classification and joint tomographic image reconstruction segmentation.
Keywords:
inverse problems; image reconstruction; tomography; deep learning; feature reconstruction; segmentation; classificationReferences:
| [1] | Adler, J.; Öktem, O., Solving ill-posed inverse problems using iterative deep neural networks, Inverse Problems, 33 (2017) · Zbl 1394.92070 · doi:10.1088/1361-6420/aa9581 |
| [2] | Adler, J.; Öktem, O., Learned primal-dual reconstruction, IEEE Trans. Med. Imaging, 37, 1322-1332 (2018) · doi:10.1109/tmi.2018.2799231 |
| [3] | Adler, J.; Ringh, A.; Öktem, O.; Karlsson, J., Learning to solve inverse problems using Wasserstein loss (2017) |
| [4] | Andrade-Loarca, H.; Kutyniok, G.; Öktem, O.; Petersen, P. C., Extraction of digital wavefront sets using applied harmonic analysis and deep neural networks, SIAM J. Imaging Sci., 12, 1936-1966 (2019) · Zbl 1439.35009 · doi:10.1137/19m1237594 |
| [5] | Arridge, S.; Maass, P.; Öktem, O.; Schönlieb, C-B, Solving inverse problems using data-driven models, Acta Numer., 28, 1-174 (2019) · Zbl 1429.65116 · doi:10.1017/s0962492919000059 |
| [6] | Badrinarayanan, V.; Kendall, A.; Cipolla., R., SegNet: a deep convolutional encoder-decoder architecture for image segmentation, IEEE Trans. Pattern Anal. Mach. Intell., 39, 2481-2495 (2017) · doi:10.1109/tpami.2016.2644615 |
| [7] | Balakrishnan, G.; Zhao, A.; Sabuncu, M. R.; Guttag, J.; Dalca, A. V., VoxelMorph: a learning framework for deformable medical image registration, IEEE Trans. Med. Imaging, 38, 1788-1800 (2019) · doi:10.1109/tmi.2019.2897538 |
| [8] | Benning, M.; Burger, M., Modern regularization methods for inverse problems, Acta Numer., 27, 1-111 (2018) · Zbl 1431.65080 · doi:10.1017/s0962492918000016 |
| [9] | Boink, Y. E.; Manohar, S.; Brune, C., A partially-learned algorithm for joint photo-acoustic reconstruction and segmentation, IEEE Trans. Med. Imaging, 39, 129-139 (2019) · doi:10.1109/tmi.2019.2922026 |
| [10] | Burger, M.; Dirks, H.; Schönlieb, C-B, A variational model for joint motion estimation and image reconstruction, SIAM J. Imaging Sci., 11, 94-128 (2018) · Zbl 1437.94009 · doi:10.1137/16m1084183 |
| [11] | Burger, M.; Rossmanith, C.; Zhang, X., Simultaneous reconstruction and segmentation for dynamic spect imaging, Inverse Problems, 32 (2016) · Zbl 1402.92266 · doi:10.1088/0266-5611/32/10/104002 |
| [12] | Chen, C.; Gris, B.; Öktem, O., A new variational model for joint image reconstruction and motion estimation in spatiotemporal imaging, SIAM J. Imaging Sci., 12, 1686-1719 (2019) · Zbl 1434.68585 · doi:10.1137/18m1234047 |
| [13] | Chen, C.; Öktem, O., Indirect image registration with large diffeomorphic deformations, SIAM J. Imaging Sci., 11, 575-617 (2018) · Zbl 1401.94013 · doi:10.1137/17m1134627 |
| [14] | Dahl, R.; Norouzi, M.; Shlens, J., Pixel recursive super resolution (2017) |
| [15] | Dashti, M.; Stuart, A. M.; Ghanem, R.; Higdon, D.; Owhadi, H., The Bayesian approach to inverse problems, Handbook of Uncertainty Quantification (2017), New York: Springer, New York · doi:10.1007/978-3-319-12385-1_7 |
| [16] | Evans, S. N.; Stark, P. B., Inverse problems as statistics, Inverse Problems, 18, R1-R55 (2002) · Zbl 1039.62007 · doi:10.1088/0266-5611/18/4/201 |
| [17] | Farabet, C.; Couprie, C.; Najman, L.; LeCun, Y., Learning hierarchical features for scene labeling, IEEE Trans. Pattern Anal. Mach. Intell., 35, 1915-1929 (2013) · doi:10.1109/tpami.2012.231 |
| [18] | Ghosal, S.; Ray, N., Deep deformable registration: enhancing accuracy by fully convolutional neural net, Pattern Recognit. Lett., 94, 81-86 (2017) · doi:10.1016/j.patrec.2017.05.022 |
| [19] | Gris, B.; Chen, C.; Öktem, O., Image reconstruction through metamorphosis, Inverse Problems, 36 (2020) · Zbl 1447.94005 · doi:10.1088/1361-6420/ab5832 |
| [20] | Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K. O., On calibration of modern neural networks (2017) |
| [21] | Guo, Y.; Liu, Y.; Georgiou, T.; Lew, M. S., A review of semantic segmentation using deep neural networks, Int. J. Multimed. Inf. Retr., 7, 87-93 (2018) · doi:10.1007/s13735-017-0141-z |
| [22] | Hauptmann, A.; Öktem, O.; Schönlieb, C-B; Chen, K.; Schönlieb, C-B; Tai, X-C; Younes, L., Image reconstruction in dynamic inverse problems with temporal models, Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging (2020), Berlin: Springer, Berlin |
| [23] | He, K.; Zhang, X.; Ren, S.; Sun, J., Deep residual learning for image recognition, 770-778 (2016) |
| [24] | Helin, T.; Burger, M., Maximum a posteriori probability estimates in infinite-dimensional Bayesian inverse problems, Inverse Problems, 31 (2015) · Zbl 1325.62058 · doi:10.1088/0266-5611/31/8/085009 |
| [25] | Hohage, T.; Werner, F., Inverse problems with Poisson data: statistical regularization theory, applications and algorithms, Inverse Problems, 32 (2016) · Zbl 1372.65163 · doi:10.1088/0266-5611/32/9/093001 |
| [26] | Hohm, K.; Storath, M.; Weinmann, A., An algorithmic framework for Mumford-Shah regularization of inverse problems in imaging, Inverse Problems, 31 (2015) · Zbl 1329.35351 · doi:10.1088/0266-5611/31/11/115011 |
| [27] | Iizuka, S.; Simo-Serra, E.; Ishikawa, H., Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification, ACM Trans. Graph., 35, 1-11 (2016) · doi:10.1145/2897824.2925974 |
| [28] | Johnson, J.; Alahi, A.; Fei-Fei, L.; Leibe, B.; Matas, J.; Sebe, N.; Welling, M., Perceptual losses for real-time style transfer and super-resolution, 694-711 (2016), Berlin: Springer, Berlin |
| [29] | Kaipio, J. P.; Somersalo, E., Statistical and Computational Inverse Problems (2005), Berlin: Springer, Berlin · Zbl 1068.65022 |
| [30] | Karpathy, A.; Fei-Fei, L., Deep visual-semantic alignments for generating image descriptions, IEEE Trans. Pattern Anal. Mach. Intell., 39, 664-676 (2017) · doi:10.1109/tpami.2016.2598339 |
| [31] | Klann, E.; Ramlau, R.; Ramlau, R.; Ring, W., A Mumford-Shah level-set approach for the inversion and segmentation of SPECT/CT data, Inverse Problems Imaging, 5, 137-166 (2011) · Zbl 1213.94015 · doi:10.3934/ipi.2011.5.137 |
| [32] | Krishnan, V. P.; Quinto, E. T.; Scherzer, O., Microlocal analysis in tomography, Handbook of Mathematical Methods in Imaging, 847-902 (2015), Berlin: Springer, Berlin · Zbl 1395.94042 · doi:10.1007/978-1-4939-0790-8_36 |
| [33] | Krizhevsky, A.; Sutskever, I.; Hinton, G. E., ImageNet classification with deep convolutional neural networks, 1097-1105 (2012) |
| [34] | Kutyniok, G.; Labate, D., Shearlets: Multiscale Analysis for Multivariate Data (2012), Berlin: Springer, Berlin · Zbl 1237.42001 |
| [35] | Lambin, P., Radiomics: the bridge between medical imaging and personalized medicine, Nat. Rev. Clin. Oncol., 14 (2017) · doi:10.1038/nrclinonc.2017.141 |
| [36] | Lang, L. F.; Neumayer, S.; Öktem, O.; Schönlieb, C-B, Template-based image reconstruction from sparse tomographic data, Appl. Math. Optim., 82, 1081-1109 (2019) · Zbl 1461.49048 · doi:10.1007/s00245-019-09573-2 |
| [37] | LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P., Gradient-based learning applied to document recognition, Proc. IEEE, 86, 2278-2324 (1998) · doi:10.1109/5.726791 |
| [38] | Liese, F.; Miescke, K-J, Statistical Decision Theory: Estimation, Testing, and Selection (2008), New York: Springer, New York · Zbl 1154.62008 |
| [39] | Liu, J.; Aviles-Rivero, A. I.; Ji, H.; Schönlieb, C-B, Rethinking medical image reconstruction via shape prior, going deeper and faster: deep joint indirect registration and reconstruction (2019) |
| [40] | Long, J.; Shelhamer, E.; Darrell, T., Fully convolutional networks for semantic segmentation, 3431-3440 (2015) |
| [41] | Louis, A. K., Feature reconstruction in inverse problems, Inverse Problems, 27 (2011) · Zbl 1229.47141 · doi:10.1088/0266-5611/27/6/065010 |
| [42] | Lucka, F.; Huynh, N.; Betcke, M.; Zhang, E.; Beard, P.; Cox, B.; Arridge, S., Enhancing compressed sensing 4D photoacoustic tomography by simultaneous motion estimation, SIAM J. Imaging Sci., 11, 2224-2253 (2018) · Zbl 1420.92060 · doi:10.1137/18m1170066 |
| [43] | Lunz, S.; Öktem, O.; Schönlieb, C-B, Adversarial regularizers in inverse problems (2018) |
| [44] | Mataev, G.; Elad, M.; DeepRED, P. M., Deep image prior powered by RED (2019) |
| [45] | Mohammad-Djafari, A., Gauss-Markov-Potts priors for images in computer tomography resulting to joint optimal reconstruction and segmentation, Int. J. Tomogr. Stat., 11, 76-92 (2009) |
| [46] | Monga, V.; Li, Y.; Eldar, Y. C., Algorithm unrolling: interpretable, efficient deep learning for signal and image processing (2019) |
| [47] | Nickl, R., On Bayesian inference for some statistical inverse problems with partial differential equations, Bernoulli News, 24, 5-9 (2017) |
| [48] | Noh, H.; Hong, S.; Han, B., Learning deconvolution network for semantic segmentation, 1520-1528 (2015) |
| [49] | Pouchol, C.; Verdier, O.; Öktem, O.; Knoll, F.; Maier, A.; Rueckert, D.; Ye, J. C., Spatiotemporal PET reconstruction using ML-EM with learned diffeomorphic deformation, Machine Learning for Medical Image Reconstructio (MLMIR 2019), 151-162 (2019), Berlin: Springer, Berlin |
| [50] | Ramlau, R.; Ring, W., A Mumford-Shah level-set approach for the inversion and segmentation of x-ray tomography data, J. Comput. Phys., 221, 539-557 (2007) · Zbl 1114.68077 · doi:10.1016/j.jcp.2006.06.041 |
| [51] | Romano, Y.; Isidoro, J.; Milanfar, P., RAISR: rapid and accurate image super resolution, IEEE Trans. Comput. Imaging, 3, 110-125 (2017) · doi:10.1109/tci.2016.2629284 |
| [52] | Romanov, M.; Dahl, A. B.; Dong, Y.; Hansen, P. C., Simultaneous tomographic reconstruction and segmentation with class priors, Inverse Problems Sci. Eng., 24, 1432-1453 (2016) · Zbl 1349.92084 · doi:10.1080/17415977.2015.1124428 |
| [53] | Ronneberger, O.; Fischer, P.; Brox, T.; U-Net; Navab, N.; Hornegger, J.; Wells, W. M.; Frangi, A. F., U-net: convolutional networks for biomedical image segmentation, Medical Image Computing and Computer Assisted Intervention (MICCAI 2015), 234-241 (2015), Berlin: Springer, Berlin · doi:10.1007/978-3-319-24574-4_28 |
| [54] | Rubin, G. D., Computed tomography: revolutionizing the practice of medicine for 40 years, Radiology, 273, 45-74 (2014) · doi:10.1148/radiol.14141356 |
| [55] | Rubinstein, R.; Bruckstein, A. M.; Elad, M., Dictionaries for sparse representation modeling, Proc. IEEE, 98, 1045-1057 (2010) · doi:10.1109/jproc.2010.2040551 |
| [56] | Schmoderer, T.; Aviles-Rivero, A. I.; Corona, V.; Debroux, N.; Schönlieb, C-B, Learning optical flow for fast MRI reconstruction (2020) |
| [57] | Storath, M.; Weinmann, A.; Frikel, J.; Unser, M., Joint image reconstruction and segmentation using the Potts model, Inverse Problems, 31 (2015) · Zbl 1342.94029 · doi:10.1088/0266-5611/31/2/025003 |
| [58] | Sun, L.; Fan, Z.; Huang, Y.; Ding, X.; Paisley, J., Joint CS-MRI reconstruction and segmentation with a unified deep network, 492-504 (2018) |
| [59] | Syu, N-S; Chen, Y-S; Chuang, Y-Y, Learning deep convolutional networks for demosaicing (2018) |
| [60] | Thoma, M., A survey of semantic segmentation (2016) |
| [61] | van Timmeren, J. E.; Cester, D.; Tanadini-Lang, S.; Alkadhi, H.; Baessler, B., Radiomics in medical imaging-‘how-to’ guide and critical reflection, Insights Imaging, 11, 91 (2020) · doi:10.1186/s13244-020-00887-2 |
| [62] | Vinyals, O.; Toshev, A.; Bengio, S.; Erhan, D., Show and tell: a neural image caption generator, 3156-3164 (2015) · doi:10.1109/cvpr.2015.7298935 |
| [63] | Wang, G.; Ye, J. C.; Mueller, K.; Fessler, J. A., Image reconstruction is a new frontier of machine learning, IEEE Trans. Med. Imaging, 37, 1289-1296 (2018) · doi:10.1109/tmi.2018.2833635 |
| [64] | Wolterink, J. M.; Dinkla, A. M.; Savenije, M. H F.; Seevinck, P. R.; van den Berg, C. A T.; Išgum, I.; Tsaftaris, S.; Gooya, A.; Frangi, A.; Prince, J., Deep MR to CT synthesis using unpaired data, Simulation and Synthesis in Medical Imaging, 14-23 (2017), Cham: Springer, Cham · doi:10.1007/978-3-319-68127-6_2 |
| [65] | Wu, D.; Kim, K.; Dong, B.; Li, Q., End-to-end abnormality detection in medical imaging (2017) |
| [66] | Xie, J.; Xu, L.; Chen, E., Image denoising and inpainting with deep neural networks, 341-349 (2012) |
| [67] | Yang, X.; Kwitt, R.; Styner, M.; Niethammer, M., Quicksilver: fast predictive image registration-a deep learning approach, NeuroImage, 158, 378-396 (2017) · doi:10.1016/j.neuroimage.2017.07.008 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
