Abstract
With the emergence of more and more lightweight, convenient and cheap surface electromyography signal (sEMG) snsors, gesture recognition based on sEMG sensors has attracted much attention of researchers. In this study, combined with the sEMG sensor, a novel dynamic hand gesture recognition approach is proposed for effective and accurate dynamic gesture prediction. Here, a portable sEMG sensor (Myo wristband) is adopted to acquire the multi-channel sEMG signals of dynamic hand gestures and the continuous wavelet transformation (CWT) is proposed for data preprocessing to acquire the time-frequency maps. Due to the success of powerful contextual feature representation capability of deep convolutional neural networks (DCNNs), a deep residual attention network is proposed for accurate prediction of time-frequency maps. To effectively extract the key spatial and channel features from multi-channel sEMG signals, a residual attention network is proposed to act as the backbone network for effective feature representation. Besides, In the proposed recognition network, a multi-scale feature enhancement (MFE) module and an attention fusion block (AFB) are proposed, which respectively improve the multi-scale expression ability of the network and effectively realize multi-scale feature enhancement, respectively. Experimental results show that the proposed recognition network could achieve a superior detection ability compared with other state-of-the-art recognition models. The source code and dataset are available at https://github.com/lyangucas92/Ges_Net.
Similar content being viewed by others
Data Availibility
The data that support the findings of this study are available upon reasonable request.
Notes
PaddlePaddle: https://www.paddlepaddle.org.cn/
References
Allard UC, Nougarou F, Fall CL, Giguère P, Gosselin C, Laviolette F, Gosselin B (2016) A convolutional neural network for robotic arm guidance using semg based frequency-features. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp 2464–2470
Al-Timemy AH, Khushaba RN, Bugmann G, Escudero J (2015) Improving the performance against force variation of emg controlled multifunctional upper-limb prostheses for transradial amputees. IEEE Trans Neural Syst Rehabil Eng 24(6):650–661
Chen L, Fu J, Wu Y, Li H, Zheng B (2020) Hand gesture recognition using compact cnn via surface electromyography signals. Sensors 20(3):672
Cheng H, Yang L, Liu Z (2015) Survey on 3d hand gesture recognition. IEEE Transactions on Circuits and Systems for Video Technology 26(9):1659–1673
Côté-Allard U, Fall CL, Campeau-Lecours A, Gosselin C, Laviolette F, Gosselin B (2017) Transfer learning for semg hand gestures recognition using convolutional neural networks. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, pp 1663–1668
Dong Y, Liu J, Yan W (2021) Dynamic hand gesture recognition based on signals from specialized data glove and deep learning algorithms. IEEE Trans Instrum Meas 70:1–14
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929
Duan F, Ren X, Yang Y (2018) A gesture recognition system based on time domain features and linear discriminant analysis. IEEE Transactions on Cognitive and Developmental Systems 13(1):200–208
Fatimah B, Singh P, Singhal A, Pachori RB (2021) Hand movement recognition from semg signals using fourier decomposition method. Biocybernetics Biomed Eng 41(2):690–703
Feng Q, Yang C, Wu X, Li Z (2013) A smart tv interaction system based on hand gesture recognition by using rgb-d sensor. In Proceedings of International Conference on Mechatronic Sciences, Electric Engineering and Computer (MEC). IEEE, pp 1319–1322
Gao Q, Chen Y, Ju Z, Liang Y (2021) Dynamic hand gesture recognition based on 3d hand pose estimation for human-robot interaction. IEEE Sensors J
Ghotkar AS, Khatal R, Khupase S, Asati S, Hadap M (2012) Hand gesture recognition for indian sign language. In Proceedings of International Conference on Computer Communication and Informatics. IEEE, pp 1–4
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 770–778
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 4700–4708
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 7132–7141
Jiang S, Gao Q, Liu H, Shull PB (2020) A novel, co-located emgfmg-sensing wearable armband for hand gesture recognition. Sensors Actuators A Phys 301:111738
Joseph K, Khan S, Khan FS, Balasubramanian VN (2021) Towards open world object detection. In Proc IEEE/CVF Conf Comput Vis Pattern Recognit (CVPR), pp 5830–5840
Kang P, Li J, Fan B, Jiang S, Shull PB (2021) Wrist-worn hand gesture recognition while walking via transfer learning. IEEE J Biomed Health Inform
Khodabandelou G, Jung P-G, Amirat Y, Mohammed S (2020) Attention-based gated recurrent unit for gesture recognition. IEEE Trans Autom Sci Eng 18(2):495–507
Lee S, Hara S, Yamada Y (2009) A safety measure for control mode switching of skill-assist for effective automotive manufacturing. IEEE Trans Autom Sci Eng 7(4):817–825
Liu K, Chen C, Jafari R, Kehtarnavaz N (2014) Fusion of inertial and depth sensor data for robust hand gesture recognition. IEEE Sensors J 14(6):1898–1903
Liu M-K, Lin Y-T, Qiu Z-W, Kuo C-K, Wu C-K (2020) Hand gesture recognition by a mmg-based wearable device. IEEE Sensors J 20(24):14 703--14 712
Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp 10 012–10 022
Luong M-T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025
Mehta S, Rastegari M (2021) Mobilevit: light-weight, generalpurpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178
Nahid N, Rahman A, Ahad M (2020) Deep learning based surface emg hand gesture classification for low-cost myoelectric prosthetic hand. In Proceedings of Joint 9th International Conference on Informatics, Electronics & Vision (ICIEV) and 4th International Conference on Imaging, Vision & Pattern Recognition (icIVPR). IEEE, pp 1–8
Altilio R, Panella M (2018) A smartphone-based application using machine learning for gesture recognition: Using feature extraction and template matching via hu image moments to recognize gestures. IEEE Consum Electron Mag 8(1):25–29
Phinyomark A, Phukpattaranont P, Limsakul C (2012) Feature reduction and selection for emg signal classification. Expert Syst Appl 39(8):7420–7431
Phinyomark A, Scheme E (2018) An investigation of temporally inspired time domain features for electromyographic pattern recognition. In Proceedings of 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp 5236–5240
Plouffe G, Cretu A-M (2015) Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Trans Instrum Meas 65(2):305–316
Qureshi SA, Raza SEA, Hussain L, Malibari AA, Nour MK, Rehman Au, Al-Wesabi FN, Hilal AM (2022) Intelligent ultralight deep learning model for multi-class brain tumor detection. Appl Sci 12(8):3715
Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimedia 15(5):1110–1120
Rioul O, Duhamel P (1992) Fast algorithms for discrete and continuous wavelet transforms. IEEE Trans Inf Theory 38(2):569–586
Sharma S, Singh S (2021) Vision-based hand gesture recognition using deep learning for the interpretation of sign language. Expert Syst Appl 182:115657
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Song W, Han Q, Lin Z, Yan N, Luo D, Liao Y, Zhang M, Wang Z, Xie X, Wang A et al (2019) Design of a flexible wearable smart semg recorder integrated gradient boosting decision tree based hand gesture recognition. IEEE Trans Biomed Circ Syst 13(6):1563–1574
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 1–9
Tan M, Pang R, Le QV (2020) Efficientdet: Scalable and efficient object detection. In Proc IEEE/CVF Conf Comput Vis Pattern Recognit (CVPR), pp 10 781–10 790
Wei W, Dai Q, Wong Y, Hu Y, Kankanhalli M, Geng W (2019) Surface-electromyography-based gesture recognition by multi-view deep learning. IEEE Trans Biomed Eng 66(10):2964–2973
Wen R, Wang Q, Li Z (2021) Human hand movement recognition using infinite hidden markov model based semg classification. Biomed Signal Process Control 68:102592
Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pp 3–19
Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 1492–1500
Yang Y, Duan F, Ren J, Xue J, Lv Y, Zhu C, Yokoi H (2020) Performance comparison of gesture recognition system based on different classifiers. IEEE Trans Cogn Dev Syst 13(1):141–150
Yao Y, Fu Y (2014) Contour model-based hand-gesture recognition using the kinect sensor. IEEE Transactions on Circuits and Systems for Video Technology 24(11):1935–1944
Yuan G, Liu X, Yan Q, Qiao S, Wang Z, Yuan L (2020) Hand gesture recognition using deep feature fusion network based on wearable sensors. IEEE Sensors J 21(1):539–547
Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122
Zhang X, Chen X, Li Y, Lantz V, Wang K, Yang J (2011) A framework for hand gesture recognition based on accelerometer and emg sensors. IEEE Trans Syst Man Cybern Syst Hum 41(6):1064–1076
Zhang Y, Cao C, Cheng J, Lu H (2018) Egogesture: A new dataset and benchmark for egocentric hand gesture recognition. IEEE Trans Multimedia 20(5):1038–1050
Acknowledgements
All authors would like to thank the anonymous referees for their valuable suggestions and comments.
Funding
This work was supported by the National Key Research & Development Project of China (2020-YFB1313701), the National Natural Science Foundation of China (No. 62003309) and Outstanding Foreign Scientist Support Project in Henan Province of China (No. GZS2019008).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare that they have no conflicts of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Liu, Y., Li, X., Yu, H. et al. A novel sEMG-based dynamic hand gesture recognition approach via residual attention network. Multimed Tools Appl 83, 9329–9349 (2024). https://doi.org/10.1007/s11042-023-15748-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-15748-5