EEG decoding method based on multi-feature information fusion for spinal cord injury. (English) Zbl 1525.92045
Summary: To develop an efficient brain-computer interface (BCI) system, electroencephalography (EEG) measures neuronal activities in different brain regions through electrodes. Many EEG-based motor imagery (MI) studies do not make full use of brain network topology. In this paper, a deep learning framework based on a modified graph convolution neural network (M-GCN) is proposed, in which temporal-frequency processing is performed on the data through modified S-transform (MST) to improve the decoding performance of original EEG signals in different types of MI recognition. MST can be matched with the spatial position relationship of the electrodes. This method fusions multiple features in the temporal-frequency-spatial domain to further improve the recognition performance. By detecting the brain function characteristics of each specific rhythm, EEG generated by imaginary movement can be effectively analyzed to obtain the subjects’ intention. Finally, the EEG signals of patients with spinal cord injury (SCI) are used to establish a correlation matrix containing EEG channel information, the M-GCN is employed to decode relation features. The proposed M-GCN framework has better performance than other existing methods. The accuracy of classifying and identifying MI tasks through the M-GCN method can reach 87.456%. After 10-fold cross-validation, the average accuracy rate is 87.442%, which verifies the reliability and stability of the proposed algorithm. Furthermore, the method provides effective rehabilitation training for patients with SCI to partially restore motor function.
MSC:
| 92C55 | Biomedical imaging and signal processing |
| 92B20 | Neural networks for/in biological studies, artificial life and related topics |
| 68T07 | Artificial neural networks and deep learning |
Keywords:
brain-computer interface; electroencephalography; motor imagery; modified graph convolution neural network; spinal cord injuryReferences:
| [1] | Aldayel, M.; Ykhlef, M.; Al-Nafjan, A., Deep learning for EEG-based preference classification in neuromarketing, Applied Sciences, 10, 4, 1-23 (2020) |
| [2] | Ameri, R.; Pouyan, A.; Abolghasemi, V., Projective dictionary pair learning for EEG signal classification in brain computer interface applications, Neurocomputing, 218, 382-389 (2016) |
| [3] | An, X.; Kuang, D.; Guo, X.; Zhao, Y.; He, L., A deep learning method for classification of EEG data based on motor imagery, (Huang, D. S.; Han, K.; Gromiha, M., Intelligent computing in bioinformatics. ICIC 2014. Intelligent computing in bioinformatics. ICIC 2014, Lecture notes in computer science, vol. 8590 (2014)), 203-210 · Zbl 1312.68010 |
| [4] | Anderson, K. D.; Acuff, M. E.; Arp, B. G., United States (US) multi-center study to assess the validity and reliability of the spinal cord independence measure (SCIM III), Spinal Cord, 49, 8, 880-885 (2011) |
| [5] | Arvaneh, M.; Guan, C.; Ang, K. K.; Quek, C., Optimizing the channel selection and classification accuracy in EEG-based BCI, IEEE Transactions on Biomedical Engineering, 58, 6, 1865-1873 (2011) |
| [6] | Athanasiou, A.; Klados, M. A.; Pandria, N., A systematic review of investigations into functional brain connectivity following spinal cord injury, Frontiers in Human Neuroscience, 11, 517-526 (2017) |
| [7] | Azab, A. M.; Mihaylova, L.; Ang, K. K.; Arvaneh, M., Weighted transfer learning for improving motor imagery-based brain-computer interface, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27, 1352-1359 (2019) |
| [8] | Bogaarts, J. G.; Hilkman, D. M.W.; Gommer, E. D., Improved epileptic seizure detection combining dynamic feature normalization with EEG novelty detection, Medical & Biological Engineering & Computing, 54, 1883-1892 (2016) |
| [9] | Cajigas, I.; Davis, K. C.; Meschede-Krasa, B., Implantable brain-computer interface for neuroprosthetic-enabled volitional hand grasp restoration in spinal cord injury, Brain Communications, 3, 4, fcab248 (2021) |
| [10] | Chen, X.; Zheng, Y.; Niu, Y.; Li, C., Epilepsy classification for mining deeper relationships between EEG channels based on GCN, (2020 international conference on computer vision, image and deep learning. 2020 international conference on computer vision, image and deep learning, CVIDL (2020)), 701-706 |
| [11] | Duan, Suolin; Xu, Tingting; Zhuang, Wei; Mao, Dan, The feature extraction of ERD/ERS signals based on the wavelet package and ICA, (Proceeding of the 11th world congress on intelligent control and automation (2014)), 5621-5625 |
| [12] | Duan, L., Zero-shot learning for EEG classification in motor imagery-based BCI system, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28, 11, 2411-2419 (2020) |
| [13] | Fallani, F. D.V.; Astolfi, L.; Cincotti, F., Cortical functional connectivity networks in normal and spinal cord injured patients: evaluation by graph analysis, Human Brain Mapping, 28, 1334-1346 (2007) |
| [14] | Garrett, D.; Peterson, D. A.; Anderson, C. W.; Thaut, M. H., Comparison of linear, nonlinear, and feature selection methods for EEG signal classification, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11, 2, 141-144 (2003) |
| [15] | Giri, E. P.; Fanany, M. I.; Arymurthy, A. M.; Wijaya, S. K., Ischemic stroke identification based on EEG and EOG using ID convolutional neural network and batch normalization, (2016 international conference on advanced computer science and information systems. 2016 international conference on advanced computer science and information systems, ICACSIS (2016)), 484-491 |
| [16] | Goel, M. K., An overview of brain computer interface, (2015 recent and emerging trends in computer and computational sciences. 2015 recent and emerging trends in computer and computational sciences, RETCOMP (2015)), 10-17 |
| [17] | Gong, A.; Liu, J.; Chen, S.; Fu, Y., Time-frequency cross mutual information analysis of the brain functional networks underlying multiclass motor imagery, Journal of Motor Behavior, 50, 254-267 (2018) |
| [18] | Guanghui, C.; Xiaoping, Z., Multi-modal emotion recognition by fusing correlation features of speech-visual, IEEE Signal Processing Letters, 28, 533-537 (2021) |
| [19] | Guger, C.; Edlinger, G.; Harkam, W.; Niedermayer, I.; Pfurtscheller, G., How many people are able to operate an EEG-based brain-computer interface (BCI), IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11, 2, 145-147 (2003) |
| [20] | Hu, Xinmei; Yuan, Shasha; Xu, Fangzhou; Leng, Yan; Yuan, Kejiang; Yuan, Qi, Scalp EEG classification using deep Bi-LSTM network for seizure detection, Computers in Biology and Medicine, 124, 10-19 (2020) |
| [21] | Huang, Shih-Feng; Lin, Yen-Wen, A feature fusion approach for multiple signal classification, (Computer symposium (ICS) 2020 international (2020)), 37-42 |
| [22] | Huang, H.; Niu, Z.; Liu, G., Early consciousness disorder in acute large hemispheric infarction: An analysis based on quantitative EEG and brain network characteristics, Neurocritical Care, 33, 376-388 (2020) |
| [23] | Ieracitano, C.; Morabito, F. C.; Hussain, A., A hybrid-domain deep learning-based BCI for discriminating hand motion planning from EEG sources, International Journal of Neural Systems, 31, 09, Article 2150038 pp. (2021) |
| [24] | Jahan, M.; Khan, Y. U.; Sharma, B. B., Classification of EEG signals based on imaginary movement of right and left hand wrist, (2014 international conference on me-dical imaging, m-health and emerging communication systems (MedCom) (2014)), 193-196 |
| [25] | Jin, M.; Chen, H.; Li, Z.; Li, J., EEG-based emotion recognition using graph convolutional network with learnable electrode relations, (2021 43rd annual international conference of the IEEE engineering in medicine & biology society (2021)), 5953-5957 |
| [26] | Jin, J.; Miao, Y.; Daly, I.; Zuo, C.; Hu, D.; Cichocki, A., Correlation-based channel selection and regularized feature optimization for MI-based BCI, Neural Networks, 262-270 (2019) |
| [27] | Kevric, J.; Subasi, A., Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system, Biomedical Signal Processing and Control, 31, 398-406 (2017) |
| [28] | Kübler, A.; Rupp, R.; Kleih, S., P300 BCI for persons with spinal cord injury: A BCI in search of an application?, (Neuroprosthetics and brain-computer interfaces in spinal cord injury (2021), Springer: Springer Cham), 193-216 |
| [29] | Kumar, S.; Sharma, A.; Mamun, K.; Tsunoda, T., A deep learning approach for motor imagery EEG signal classification, (2016 3rd Asia-Pacific world congress on computer science and engineering (APWC on CSE) (Nadi) (2016)), 34-39 |
| [30] | Leocani, L.; Toro, C.; Manganotti, P.; Zhuang, P.; Hallett, M., Event-related coherence and event-related desynchronization/synchronization in the 10 Hz and 20 Hz EEG during self-paced movements, Electroencephalography and Clinical Neurophysiology/Evoked Potentials Section, 104, 3, 199-206 (1997) |
| [31] | Li, M.-A., Decoding of motor imagery EEG based on brain source estimation, Neurocomputing, 339, 182-193 (2019) |
| [32] | Lin, Xiaohan; Zou, Xiaolong; Ji, Zilong; Huang, Tiejun; Wu, Si; Mi, Yuanyuan, A brain-inspired computational model for spatio-temporal information processing, Neural Networks, 143, 74-87 (2021) · Zbl 1521.68058 |
| [33] | Liu, Y. H., Analysis of electroencephalography event-related desynchronisation and synchronisation induced by lower-limb stepping motor imagery, Journal of Medical and Biological Engineering, 39, 1, 54-69 (2019) |
| [34] | Lopez-Larraz, E.; Antelis, J. M.; Montesano, L.; Gil-Agudo, A.; Minguez, J., Continuous decoding of motor attempt and motor imagery from EEG activity in spinal cord injury patients, (2012 annual international conference of the IEEE engineering in medicine and biology society (2012)), 1798-1801 |
| [35] | Mammone, N.; Ieracitano, C.; Morabito, F. C., A deep CNN approach to decode motor preparation of upper limbs from time-frequency maps of EEG signals at source level, Neural Networks, 124, 357-372 (2020) |
| [36] | Martín-Clemente, R.; Olias, J.; Thiyam, D. B.; Cichocki, A.; Cruces, S., Information theoretic approaches for motor-imagery BCI systems: Review and experimental comparison, Entropy, 20, 1, 7-13 (2018) |
| [37] | Mathur, P.; Chakka, V. K., Graph signal processing of EEG signals for detection of epilepsy, (2020 7th international conference on signal processing and integrated networks. 2020 7th international conference on signal processing and integrated networks, SPIN (2020)), 839-843 |
| [38] | Matsuoka, S.; Amano, H.; Nakajima, K., From flops to bytes: disruptive change in high-performance computing towards the post-moore era, (Proceedings of the ACM international conference on computing frontiers (2016)), 274-281 |
| [39] | Merante, A.; Zhang, Y.; Kumar, S., Brain-Computer interfaces for spinal cord injury rehabilitation, (Neuroergonomics (2020), Springer: Springer Cham), 315-328 |
| [40] | Miao, M., Hu, W., & Zhang, W. (2021). A spatial-frequency-temporal 3D convolutional neural network for motor imagery EEG signal classification. 15, 1797-1804. |
| [41] | Mirzabagherian, H.; Sardari, M. A.; Menhaj, M. B., Classification of raw spinal cord injury EEG data based on the temporal-spatial inception deep convolutional neural network, (2021 9th RSI international conference on robotics and mechatronics (ICRoM) (2021), IEEE), 43-50 |
| [42] | Nagarajan, A.; Robinson, N.; Guan, C., Investigation on robustness of EEG-based brain-computer interfaces, (2021 43rd annual international conference of the IEEE engineering in medicine & biology society. 2021 43rd annual international conference of the IEEE engineering in medicine & biology society, EMBC (2021)), 6334-6340 |
| [43] | Nam, C. S., Movement imagery-related lateralization of event-related (de) synchronization (ERD/ERS): motor-imagery duration effects, Clinical Neurophysiology, 122, 3, 567-577 (2011) |
| [44] | Netzer, Eitan; Frid, Alex; Feldman, Dan, Real-time EEG classification via coresets for BCI applications, Engineering Applications of Artificial Intelligence, 89, 33-35 (2020) |
| [45] | Nicolas-Alonso, L. F.; Gomez-Gil, J., Brain computer interfaces, a review, Sensors, 12, 2, 1211-1279 (2012) |
| [46] | Olias, J.; Martín-Clemente, R.; Sarmiento-Vega, M. A.; Cruces, S., EEG signal processing in MI-BCI applications with improved covariance matrix estimators, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27, 5, 895-904 (2019) |
| [47] | Opsommer, E.; Chevalley, O.; Korogod, N., Motor imagery for pain and motor function after spinal cord injury: a systematic review, Spinal Cord, 58, 262-274 (2020) |
| [48] | Pfurtscheller, G., Functional brain imaging based on ERD/ERS, Vision Research, 41, 10-11, 1257-1260 (2001) |
| [49] | Pfurtscheller, G.; Linortner, P.; Winkler, R.; Korisek, G.; Müller-Putz, G., Discrimination of motor imagery-induced EEG patterns in patients with complete spinal cord injury, Computational Intelligence and Neuroscience, 9, 6-12 (2009) |
| [50] | Pfurtscheller, G.; Lopes da Silva, F. H., Event-related EEG/MEG synchronization and desynchronization: basic principles, Clinical Neurophysiology, 110, 184-185 (1999) |
| [51] | Pfurtscheller, G.; Lopes da Silva, F. N., Event-related EEG/MEG synchronization and desynchronization: basic principles, Journal of Clinical Neurophysiology, 110, 11, 1842-1845 (1999) |
| [52] | Praveena, H. D.; Subhas, C.; Naidu, K. R., Automatic epileptic seizure recognition using reliefF feature selection and long short-term memory classifier, Journal of Ambient Intelligence and Humanized Computing, 12, 6151-6167 (2021) |
| [53] | Ren, W.; Han, M.; Wang, J.; Wang, D.; Li, T., Efficient feature extraction framework for EEG signals classification, (2016 seventh international conference on intelligent control and information processing. 2016 seventh international conference on intelligent control and information processing, ICICIP (2016)), 167-172 |
| [54] | Ren, Y.; Shi, Y.; Zhang, K.; Chen, Z.; Yan, Z., Medical treatment migration prediction based on GCN via medical insurance data, IEEE Journal of Biomedical and Health Informatics, 24, 9, 2516-2522 (2020) |
| [55] | Shajil, N.; Mohan, S.; Srinivasan, P., Multiclass classification of spatially filtered motor imagery EEG signals using convolutional neural network for BCI based applications, Journal of Medical and Biological Engineering, 40, 663-672 (2020) |
| [56] | Shi, Jh.; Shen, Jz.; Ji, Y., A submatrix-based P300 brain-computer interface stimulus presentation paradigm, JoBCI system can drivurnal of Zhejiang University Science C, 13, 452-459 (2012) |
| [57] | Silvoni, S.; Ramos-Murguialday, A.; Cavinato, M., Brain-computer interface in stroke: A review of progress, Clinical EEG and Neuroscience, 42, 4, 245-252 (2011) |
| [58] | Song, T.; Liu, S.; Zheng, W.; Zong, Y.; Cui, Z., Instance-adaptive graph for EEG emotion recognition, AAAI, 34, 2701-2708 (2020) |
| [59] | Song, T.; Zheng, W.; Song, P.; Cui, Z., EEG emotion recognition using dynamical graph convolutional neural networks, IEEE Transactions on Affective Computing, 11, 3, 532-541 (2020) |
| [60] | Sreeja, S. R.; Himanshu; Samanta, D., Distance-based weighted sparse representation to classify motor imagery EEG signals for BCI applications, Multimedia Tools and Applications, 79, 13775-13793 (2020) |
| [61] | Sreeja, S. R.; Sahay, R. R.; Samanta, D.; Mitra, P., Removal of eye blink artifacts from EEG signals using sparsity, IEEE Journal of Biomedical and Health Informatics, 22, 5, 1362-1372 (2018) |
| [62] | Stockwell, R. G.; Mansinha, L.; Lowe, R. P., Localization of the complex spectrum: the S transform, IEEE Transactions on Signal Processing, 44, 998-1001 (1996) |
| [63] | Wang, H.; Xu, L.; Bezerianos, A.; Chen, C.; Zhang, Z., Linking attention-based multiscale CNN with dynamical GCN for driving fatigue detection, IEEE Transactions on Instrumentation and Measurement, 70, 1-11 (2021) |
| [64] | Wang, G., Seizure prediction using directed transfer function and convolution neural network on intracranial EEG, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28, 12, 2711-2720 (2020) |
| [65] | Wang, J., A weighted overlook graph representation of EEG data for absence epilepsy detection, (2020 IEEE international conference on data mining. 2020 IEEE international conference on data mining, ICDM (2020)), 581-590 |
| [66] | Xu, Fangzhou; Miao, Yunjing; Sun, Yanan; Guo, Dongju; Xu, Jiali; Wang, Yuandong, A transfer learning framework based on motor imagery rehabilitation for stroke, Scientific Reports, 11, 1, 1-9 (2021) |
| [67] | Xu, Fangzhou; Rong, Fenqi; Leng, Jiancai; Sun, Tao; Zhang, Yang; Siddharth, Siddharth, Classification of left-versus right-hand motor imagery in stroke patients using supplementary data generated by cyclegan, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29, 2417-2424 (2021) |
| [68] | Xu, Fangzhou; Rong, Fenqi; Miao, Yunjing; Sun, Yanan; Dong, Gege; Li, Han, Representation learning for motor imagery recognition with deep neural network, Electronics, 10, 2, 112 (2021) |
| [69] | Xu, Fangzhou; Zheng, Wenfeng; Shan, Dongri; Yuan, Qi; Zhou, Weidong, Decoding spectral-temporal representation for motor imagery recognition using electrocorticogram, Journal of Integrative Neuroscience, 19, 2, 259-272 (2020) |
| [70] | Xu, F.; Zhou, W.; Zhen, Y.; Yuan, Q., Classification of ECoG with modified S-transform for brain-computer interface, Computer Information System, 10, 8029-8041 (2014) |
| [71] | Xu, Fangzhou; Zhou, Weidong; Zhen, Yilin; Yuan, Qi; Wu, Qi, Using fractal and local binary pattern features for classification of ecog motor imagery tasks obtained from the right brain hemisphere, International Journal of Neural Systems, 26, 6, 1-13 (2016) |
| [72] | Yu, Zuyi; Zhou, Weidong; Zhang, Fan; Xu, Fangzhou; Yuan, Shasha; Leng, Yan, Automatic seizure detection based on kernel robust probabilistic collaborative representation, Medical & Biological Engineering & Computing, 57, 1, 205-219 (2019) |
| [73] | Zhang, Y.; Ji, X.; Liu, B., Combined feature extraction method for classification of EEG signals, Neural Computing and Applications, 28, 3153-3161 (2017) |
| [74] | Zhang, Y.; Nam, C. S.; Zhou, G.; Jin, J.; Wang, X.; Cichocki, A., Temporally constrained sparse group spatial patterns for motor imagery BCI, IEEE Transactions on Cybernetics, 49, 9, 3322-3332 (2018) |
| [75] | Zhao, Q.; Zhang, L.; Cichocki, A., EEG-based asynchronous BCI control of a car in 3D virtual reality environments, Chinese Science Bulletin, 54, 78-87 (2009) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
