Abstract
Ground-based solar-image restoration is a computationally expensive procedure that involves nonlinear optimization techniques. The presence of atmospheric turbulence produces perturbations in individual images that make it necessary to apply blind deconvolution techniques. These techniques rely on the observation of many short-exposure frames that are used to simultaneously infer the instantaneous state of the atmosphere and the unperturbed object. We have recently explored the use of machine learning to accelerate this process, with promising results. We build upon this previous work to propose several interesting improvements that lead to better models. Also, we propose a new method to accelerate the restoration based on algorithm unrolling. In this method, the image-restoration problem is solved with a gradient-descent method that is unrolled and accelerated, aided by a few small neural networks. The role of the neural networks is to correct the estimation of the solution at each iterative step. The model is trained to perform the optimization in a small fixed number of steps with a curated dataset. Our findings demonstrate that both methods significantly reduce the restoration time compared to the standard optimization procedure. Furthermore, we showcase that these models can be trained in an unsupervised manner using observed images from three different instruments. Remarkably, they also exhibit robust generalization capabilities when applied to new datasets. To foster further research and collaboration, we openly provide the trained models, along with the corresponding training and evaluation code, as well as the training dataset, to the scientific community.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Code Availability
The training and evaluation code is freely available in the following repository: https://github.com/aasensio/neural-MFBD. This repository also contains information on how to retrieve the training data.
Notes
Karhunen–Loève (KL) modes are obtained after a suitable rotation of the Zernike basis by diagonalizing the covariance matrix under the assumption of Kolmogorov turbulence (Noll, 1976).
References
Asensio Ramos, A., de la Cruz Rodríguez, J., Pastor Yabar, A.: 2018, Real-time, multiframe, blind deconvolution of solar images. Astron. Astrophys. 620, A73.
Asensio Ramos, A., Olspert, N.: 2021, Learning to do multiframe wavefront sensing unsupervised: applications to blind deconvolution. Astron. Astrophys. 646, A100.
de la Cruz Rodríguez, J., Rouppe van der Voort, L., Socas-Navarro, H., van Noort, M.: 2013, Physical properties of a sunspot chromosphere with umbral flashes. Astron. Astrophys. 556, A115.
Denis, L., Thiébaut, É., Soulez, F., Becker, J.-M., Mourya, R.: 2015, Fast approximations of shift-variant blur. Int. J. Comput. Vis. 115, 253.
Denker, C.J., Verma, M., Wiśniewska, A., Kamlah, R., Kontogiannis, I., Dineva, E., Rendtel, J., Bauer, S.-M., Dionies, M., Önel, H., Woche, M., Kuckein, C., Seelemann, T., Pal, P.S.: 2023, Improved high-resolution fast imager. J. Astron. Telesc. Instrum. Syst. 9, 015001.
Gonsalves, R.A., Chidlaw, R.: 1979, Wavefront sensing by phase retrieval. In: Tescher, A.G. (ed.) Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 207, 32.
Gregor, K., LeCun, Y.: 2010, Learning fast approximations of sparse coding. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, Omnipress, Madison, 399.
He, K., Zhang, X., Ren, S., Sun, J.: 2015, Deep residual learning for image recognition. CoRR. arXiv.
Hendrycks, D., Gimpel, K.: 2016, Gaussian Error Linear Units (GELUs). arXiv:e-prints. arXiv.
Hirsch, M., Sra, S., Schölkopf, B., Harmeling, S.: 2010, Efficient filter flow for space-variant multiframe blind deconvolution. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 607. DOI.
Kingma, D.P., Ba, J.: 2014, Adam: a method for stochastic optimization. arXiv e-prints. arXiv.
Kleint, L., Berkefeld, T., Esteves, M., Sonner, T., Volkmer, R., Gerber, K., Krämer, F., Grassin, O., Berdyugina, S.: 2020, GREGOR: optics redesign and updates from 2018 – 2020. Astron. Astrophys. 641, A27.
Kuckein, C., Denker, C., Verma, M., Balthasar, H., González Manrique, S.J., Louis, R.E., Diercke, A.: 2017, sTools - a data reduction pipeline for the GREGOR Fabry-Pérot Interferometer and the High-resolution Fast Imager at the GREGOR solar telescope. In: Vargas Domínguez, S., Kosovichev, A.G., Antolin, P., Harra, L. (eds.) Fine Structure and Dynamics of the Solar Atmosphere 327, 20. DOI.
Labeyrie, A.: 1970, Attainment of diffraction limited resolution in large telescopes by Fourier analysing speckle patterns in star images. Astron. Astrophys. 6, 85.
Löfdahl, M.G., Scharmer, G.B.: 1994, Wavefront sensing and image restoration from focused and defocused solar images. Astron. Astrophys. 107, 243.
Löfdahl, M.G., Berger, T.E., Shine, R.S., Title, A.M.: 1998, Preparation of a dual wavelength sequence of high-resolution solar photospheric images using phase diversity. Astrophys. J. 495, 965.
Löfdahl, M.G., Bones, P.J., Fiddy, M.A., Millane, R.P.: 2002, Multi-frame blind deconvolution with linear equality constraints. In: Image Reconstruction from Incomplete Data 4792. DOI.
Löfdahl, M.G., Hillberg, T., de la Cruz Rodríguez, J., Vissers, G., Andriienko, O., Scharmer, G.B., Haugan, S.V.H., Fredvik, T.: 2021, SSTRED: data- and metadata-processing pipeline for CHROMIS and CRISP. Astron. Astrophys. 653, A68.
Monga, V., Li, Y., Eldar, Y.C.: 2021, Algorithm unrolling: interpretable, efficient deep learning for signal and image processing. IEEE Signal Process. Mag. 38, 18.
Nagy, J.G., O’Leary, D.P.: 1998, Restoring images degraded by spatially variant blur. SIAM J. Sci. Comput. 19, 1063.
Noll, R.J.: 1976, Zernike polynomials and atmospheric turbulence. J. Opt. Soc. Amer. 66, 207.
Oscoz, A., Rebolo, R., López, R., Pérez-Garrido, A., Pérez, J.A., Hildebrandt, S., Rodríguez, L.F., Piqueras, J.J., Villó, I., González, J.M., Barrena, R., Gómez, G., Garcí a-Hernández, D.A., Montañés, P., Rosenberg, A., Cadavid, E., Calcines, A., Dí az-Sánchez, A., Kohley, R., Martín, Y., Peñate, J., Sánchez, V. (eds.): 2008, FastCam: a new lucky imaging instrument for medium-sized telescopes. SPIE Conf. Ser. 7014, 701447.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: 2019, PyTorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’é-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, Curran Associates, Red Hook, 8024.
Paxman, R.G., Schulz, T.J., Fienup, J.R.: 1992, Joint estimation of object and aberrations by using phase diversity. J. Opt. Soc. Amer. A 9, 1072.
Scharmer, G.B.: 2006, Comments on the optimization of high resolution Fabry-Pérot filtergraphs. Astron. Astrophys. 447, 1111.
Scharmer, G.: 2017, SST/CHROMIS: a new window to the solar chromosphere. In: SOLARNET IV: The Physics of the Sun from the Interior to the Outer Atmosphere, 85.
Scharmer, G.B., Bjelksjo, K., Korhonen, T.K., Lindberg, B., Petterson, B.: 2003, The 1-meter Swedish solar telescope. In: Keil, S.L., Avakyan, S.V. (eds.) Innovative Telescopes and Instrumentation for Solar Astrophysics, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 4853, 341.
Scharmer, G.B., Narayan, G., Hillberg, T., de la Cruz Rodriguez, J., Löfdahl, M.G., Kiselman, D., Sütterlin, P., van Noort, M., Lagg, A.: 2008, CRISP spectropolarimetric imaging of penumbral fine structure. Astrophys. J. Lett. 689, L69.
Schmidt, W., von der Lühe, O., Volkmer, R., Denker, C., Solanki, S.K., Balthasar, H., Bello Gonzalez, N., Berkefeld, T., Collados, M., Fischer, A., Halbgewachs, C., Heidecke, F., Hofmann, A., Kneer, F., Lagg, A., Nicklas, H., Popow, E., Puschmann, K.G., Schmidt, D., Sigwarth, M., Sobotka, M., Soltau, D., Staude, J., Strassmeier, K.G., Waldmann, T.A.: 2012, The 1.5 meter solar telescope GREGOR. Astron. Nachr. 333, 796.
van Noort, M.J., Rouppe van der Voort, L.H.M.: 2008, Stokes imaging polarimetry using image restoration at the Swedish 1-m solar telescope. Astron. Astrophys. 489, 429.
van Noort, M., Rouppe van der Voort, L., Löfdahl, M.G.: 2005, Solar image restoration by use of multi-frame blind de-convolution with multiple objects and phase diversity. Solar Phys. 228, 191.
von der Lühe, O.: 1993, Speckle imaging of solar small scale structure. I - Methods. Astron. Astrophys. 268, 374.
Acknowledgments
The authors thank Michiel van Noort for invaluable advice on solar-image restoration and Nigul Olspert for his work on the initial phases of this work. The authors acknowledge Sergio J. González Manrique, who participated in the GREGOR observing campaigns. The authors are also thankful to Andrea Diercke, the PI of the HiFI validation data, for allowing the use of the data. We also acknowledge Javier Trujillo Bueno for participating in the acquisition of the quiet-Sun data during our 2019 campaign at the SST. The Swedish 1-m Solar Telescope is operated on the island of La Palma by the Institute for Solar Physics of Stockholm University in the Spanish Observatorio del Roque de los Muchachos of the Instituto de Astrofísica de Canarias. The Institute for Solar Physics is supported by a grant for research infrastructures of national importance from the Swedish Research Council (registration number 2017-00625). The authors thankfully acknowledge the technical expertise and assistance provided by the Spanish Supercomputing Network (Red Española de Supercomputación), as well as the computer resources used: the La Palma Supercomputer, located at the Instituto de Astrofísica de Canarias. The 1.5-meter GREGOR solar telescope was built by a German consortium under the leadership of the Leibniz Institute for Solar Physics (KIS) in Freiburg with the Leibniz Institute for Astrophysics Potsdam (AIP), the Institute for Astrophysics Göttingen, and the Max Planck Institute for Solar System Research (MPS) in Göttingen as partners, and with contributions by the Instituto de Astrofísica de Canarias (IAC) and the Astronomical Institute of the Academy of Sciences of the Czech Republic (ASU).
Funding
AAR acknowledges financial support from the Spanish Ministerio de Ciencia, Innovación y Universidades through project PGC2018-102108-B-I00 and FEDER funds. CK acknowledges funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 895955. SEP acknowledges the funding received from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (ERC Advanced grant agreement No. 742265).
Author information
Authors and Affiliations
Contributions
AAR proposed the two neural models, curated the training data, trained the models and analyzed the results. AAR made all figures and wrote the text. SEP observed and reduced the CRISP@SST and CHROMIS@SST data, also producing the MOMFBD restorations. CK observed and reduced the HiFI@GREGOR data, also producing the MOMFBD restorations. SEP and CK also contributed to the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Asensio Ramos, A., Esteban Pozuelo, S. & Kuckein, C. Accelerating Multiframe Blind Deconvolution via Deep Learning. Sol Phys 298, 91 (2023). https://doi.org/10.1007/s11207-023-02185-8
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11207-023-02185-8