Decoding motor imagery based on dipole feature imaging and a hybrid CNN with embedded squeeze-and-excitation block

• Autorzy: Wang Linlin , Li Mingai

Identyfikatory

DOI

10.1016/j.bbe.2023.10.004

• Języki publikacji: EN

Abstrakty

EN

Motor imagery (MI) decoding is the core of an intelligent rehabilitation system in brain computer interface, and it has a potential advantage by using source signals, which have higher spatial resolution and the same time resolution compared to scalp electroencephalography (EEG). However, how to delve and utilize the personalized frequency characteristic of dipoles for improving decoding performance has not been paid sufficient attention. In this paper, a novel dipole feature imaging (DFI) and a hybrid convolutional neural network (HCNN) with an embedded squeeze-and-excitation block (SEB), denoted as DFI-HCNN, are proposed for decoding MI tasks. EEG source imaging technique is used for brain source estimation, and each sub-band spectrum powers of all dipoles are calculated through frequency analysis and band division. Then, the 3D space information of dipoles is retrieved, and by using azimuthal equidistant projection algorithm it is transformed to a 2D plane, which is combined with nearest neighbor interpolation to generate multi sub-band dipole feature images. Furthermore, a HCNN is designed and applied to the ensemble of sub-band dipole feature images, from which the importance of sub-bands is acquired to adjust the corresponding attentions adaptively by SEB. Ten-fold cross-validation experiments on two public datasets achieve the comparatively higher decoding accuracies of 84.23% and 92.62%, respectively. The experiment results show that DFI is an effective feature representation, and HCNN with an embedded SEB can enhance the useful frequency information of dipoles for improving MI decoding.

Słowa kluczowe

EN

brain computer interface; motor imagery; EEG source imaging; hybrid convolutional neural network; squeeze and excitation block

PL

interfejs mózg-komputer; obrazowanie motoryczne; obrazowanie EEG; sieć neuronowa konwolucyjna

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2023
• Tom: Vol. 43, no. 4
• Strony: 751--762
• Opis fizyczny: Bibliogr. 55 poz., rys., tab., wykr.

Twórcy

autor

Wang Linlin

• Faculty of Information Technology, Beijing University of Technology, Beijing, China

autor

Li Mingai

• Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
• Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing 100124, China
• Engineering Research Center of Digital Community, Ministry of Education, Beijing 100124, China

Bibliografia

• [1] Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. Brain-computer interfaces for communication and control. Clin Neurophysiol 2002;113 (6):767-91. https://doi.org/10.1016/s1388-2457(02)00057-3.
• [2] Chaudhary U, Chander BS, Ohry A, Jaramillo-Gonzalez A, Lulé D, Birbaumer N. Brain computer interfaces for assisted communication in paralysis and quality of life. Int J Neural Syst 2021;31(11):2130003. https://doi.org/10.1142/ S0129065721300035.
• [3] Pichiorri F, Morone G, Petti M, Toppi J, Pisotta I, Molinari M, Paolucci S, Inghilleri M, Astolfi L, Cincotti F, Mattia D. Brain-computer interface boosts motor imagery practice during stroke recovery. Ann Neurol 2015; 77(5): 851-865. https://doi. org/10.1002/ana.24390.
• [4] Kim KT, Suk HI, Lee SW. Commanding a brain-controlled wheelchair using steadystate somatosensory evoked potentials. IEEE Trans Neural Syst Rehabil Eng 2018; 26(3):654-65. https://doi.org/10.1109/TNSRE.2016.2597854.
• [5] Edelman BJ, Meng J, Suma D, Zurn C, Nagarajan E, Baxter BS, et al. Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Sci Rob 2019;4(31):6844. https://doi.org/10.1126/scirobotics.aaw684.
• [6] He S, Tan H, Li Y, Zhou Y, Yu T, Zhang R, et al. EEG-and EOG-based asynchronous hybrid BCI: A system integrating a speller, a web browser, an E-mail client, and a file explorer. IEEE Trans Neural Syst Rehabil Eng 2020;28(2):519-30. https://doi. org/10.1109/TNSRE.2019.2961309.
• [7] Cantillo-Negrete J, Carino-Escobar RI, Carrillo-Mora P, Barraza-Madrigal JA, Arias-Carrión O. Robotic orthosis compared to virtual hand for brain-computer interface feedback. Biocybern Biomed Eng 2019;39(2):263-72. https://doi.org/10.1016/j. Bbe.2018.12.002.
• [8] Lee P-L, Chen S-H, Chang T-C, Lee W-K, Hsu H-T, Chang H-H. Continual learning of a transformer-based deep learning classifier using an initial model from action observation EEG data to online motor imagery classification. Bioengineering 2023; 10(2):186. https://doi.org/10.3390/bioengineering10020186.
• [9] Rashid M, Sulaiman N, Majeed APPA, Musa RM, Ab Nasir AF, Bari BS, et al. Current status, challenges, and possible solutions of eeg-based brain-computer interface: A comprehensive review. Front Neurorobotics 2020;14:25. https://doi.org/10.3389/ fnbot.2020.00025.
• [10] Brusini L, Stival F, Setti F, Menegatti E, Menegaz G, Storti SF. A systematic review on motor-imagery brain-connectivity-based computer interfaces. IEEE T HumMach Syst 2021;51(6):725-33. https://doi.org/10.1109/THMS.2021.3115094.
• [11] Vavoulis A, Figueiredo P, Vourvopoulos A. A review of online classification performance in motor imagery-based brain-computer interfaces for stroke neurorehabilitation. Signals 2023;4(1):73-86. https://doi.org/10.3390/signals4010004.
• [12] Selim S, Tantawi MM, Shedeed HA, Badr A. A CSP\AM-BA-SVM approach for motor imagery BCI system. IEEE Access 2018;6:49192-208. https://doi.org/ 10.1109/ACCESS.2018.2868178.
• [13] Thenmozhi T, Helen R. Feature selection using extreme gradient boosting bayesian optimization to upgrade the classification performance of motor imagery signals for BCI. J Neurosci Methods 2022;366:109425. https://doi.org/10.1016/j. Jneumeth.2021.109425.
• [14] Miao Y, Jin J, Daly I, Zuo C, Wang X, Cichocki A, et al. Learning common time-frequency-spatial patterns for motor imagery classification. IEEE Trans Neural Syst Rehabil Eng 2021;29:699-707. https://doi.org/10.1109/TNSRE.2021.3071140.
• [15] Pourali H, Omranpour H. CSP-Ph-PS: Learning CSP-phase space and Poincare sections based on evolutionary algorithm for EEG signals recognition. Expert Syst Appl 2023;211:118621. https://doi.org/10.1016/j.eswa.2022.118621.
• [16] Ma Y, Wu X, Zheng L, Lian P, Xiao Y, Yi Z. Iterative outlier removal clustering based time-frequency-spatial feature selection for binary EEG motor imagery decoding. IEEE Trans Instrum Meas 2022;71:1-14. https://doi.org/10.1109/ TIM.2022.3193407.
• [17] Huang S, Chen Y, Wang T, Ma T. Spectrum-weighted tensor discriminant analysis for motor imagery-based BCI. IEEE Access 2020;8:93749-59. https://doi.org/ 10.1109/ACCESS.2020.2995302.
• [18] Hwang J, Park S, Chi J. Improving multi-class motor imagery EEG classification using overlapping sliding window and deep learning model. Electronics 2023;12 (5):1186. https://doi.org/10.3390/electronics12051186.
• [19] An Y, Lam HK, Ling SH. Multi-classification for EEG motor imagery signals using data evaluation-based auto-selected regularized FBCSP and convolutional neural network. Neural Comput & Applic 2023;35:12001-27. https://doi.org/10.1007/ s00521-023-08336-z.
• [20] Liu K, Yang M, Yu Z, Wang G, Wu W. FBMSNet: A filter-bank multi-scale convolutional neural network for EEG-based motor imagery decoding. IEEE Trans Biomed Eng 2023;70(2):436-45. https://doi.org/10.1109/TBME.2022.3193277.
• [21] Jia H, Yu S, Yin S, Liu L, Yi C, Xue K, et al. A model combining multi branch spectral-temporal CNN, efficient channel attention, and lightGBM for MI-BCI classification. IEEE Trans Neural Syst Rehabil Eng 2023;31:1311-20. https://doi. org/10.1109/TNSRE.2023.3243992.
• [22] Taia B, Nms B, Mao B. Compact convolutional neural network (CNN) based on SincNet for end-to-end motor imagery decoding and analysis. Biocybern Biomed Eng 2021;41:1629-45. https://doi.org/10.1016/j.bbe.2021.10.001.
• [23] Wang H, Yu H, Wang H. EEG_GENet: A feature-level graph embedding method for motor imagery classification based on EEG signals. Biocybern Biomed Eng 2022; 42:1023-40. https://doi.org/10.1016/j.bbe.2022.08.003.
• [24] Wang L, Li M. The quantitative application of channel importance in movement intention decoding. Biocybern Biomed Eng 2022;42(2):630-45. https://doi.org/ 10.1016/j.bbe.2022.05.002.
• [25] Hsu WY, Cheng YW. EEG-channel-temporal-spectral-attention correlation for motor imagery EEG classification. IEEE Trans Neural Syst Rehab Eng 2023;31: 1659-69. https://doi.org/10.1109/TNSRE.2023.3255233.
• [26] Ortiz-Echeverri CJ, Salazar-Colores S, Rodríguez-Reséndiz J, Gómez-Loenzo RA. A new approach for motor imagery classification based on sorted blind source separation, continuous wavelet transform, and convolutional neural network. Sensors 2019;19(20):4541. https://doi.org/10.3390/s19204541.
• [27] Wang J, Yao L, Wang Y. IFNet: An interactive frequency convolutional neural network for enhancing motor imagery decoding from EEG. IEEE Trans Neural Syst Rehab Eng 2023;31:1900-11. https://doi.org/10.1109/TNSRE.2023.3257319.
• [28] Rammy SA, Abbas W, Mahmood SS, Riaz H, Rehman HU, Abideen RZU, et al. Sequence-to-sequence deep neural network with spatio-spectro and temporal features for motor imagery classification. Biocybern Biomed Eng 2021;41:97-110. https://doi.org/10.1016/j.bbe.2020.12.004.
• [29] Li Y, Guo L, Liu Y, Liu J, Meng F. A temporal-spectral-based dqueeze-andexcitation feature fusion network for motor imagery EEG decoding. IEEE Trans Neural Syst Rehab Eng 2021;29:1534-45. https://doi.org/10.1109/TNSRE.2021.3099908.
• [30] Altaheri H, Muhammad G, Alsulaiman M. Physics-informed attention temporal convolutional network for EEG-based motor imagery classification. IEEE Trans Ind Inform 2023;19(2):2249-58. https://doi.org/10.1109/TII.2022.3197419.
• [31] Shenoy Handiru V, Vinod AP, Guan C. EEG source imaging of movement decoding: The state of the art and future directions. IEEE Syst Man Cybern Mag 2018;4(2): 14-23. https://doi.org/10.1109/MSMC.2017.2778458.
• [32] Edelman B J, Meng J, Suma D, Zurn C, Nagarajan E, Baxter B S, Cline C C, He B. Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Science Robotics 2019; 4(31):eaaw6844. https://doi.org/10.1126/ scirobotics.aaw6844.
• [33] Sohrabpour A, He B. Exploring the extent of source imaging: Recent advances in noninvasive electromagnetic brain imaging. Curr Opin Biomed Eng 2021;18(1): 100277. https://doi.org/10.1016/j.cobme.2021.100277.
• [34] Sosnik R, Ben ZO. Reconstruction of hand, elbow and shoulder actual and imagined trajectories in 3D space using EEG slow cortical potentials. J Neural Eng 2020;17 (1):016065. https://doi.org/10.1088/1741-2552/ab59a7.
• [35] Xygonakis I, Athanasiou A, Pandria N, Kugiumtzis D, Bamidis PD. Decoding motor imagery through common spatial pattern filters at the EEG source space. Comput Intell Neurosci 2018;2018:7957408. https://doi.org/10.1155/2018/7957408.
• [36] Li MA, Wang YF, Jia SM, Sun YJ, Yang JF. Decoding of motor imagery EEG based on brain source estimation. Neurocomputing 2019;339:182-93. https://doi.org/ 10.1016/j.neucom.2019.02.006.
• [37] Li MA, Wang YF, Zhu XQ, Yang JF. A wrapped time-frequency combined selection in the source domain. Biomed Signal Process Control 2020;57(4):101748. https:// doi.org/10.1016/j.bspc.2019.101748.
• [38] Li MA, Dong YX, Sun YJ, Yang JF, Duan LJ. Subject-based dipole selection for decoding motor imagery tasks. Neurocomputing 2020;402:195-208. https://doi. org/10.1016/j.neucom.2020.03.055.
• [39] Dong Y, Wang L, Li M. Applying correlation analysis to electrode optimization in source domain. Med Biol Eng Compu 2023;61:1225-38. https://doi.org/10.1007/s11517-023-02770-w.
• [40] Rajabioun M. Motor imagery classification by active source dynamics. Biomed Signal Process Control 2020;61:102028. https://doi.org/10.1016/j. Bspc.2020.102028.
• [41] Hou Y, Zhou L, Jia S, Lun X. A novel approach of decoding EEG four-class motor imagery tasks via scout ESI and CNN. J Neural Eng 2020;17(1):016048. https://doi.org/10.1088/1741-2552/ab4af6.
• [42] Mammone N, Ieracitano C, Morabito FC. A deep cnn approach to decode motor preparation of upper limbs from time-frequency maps of eeg signals at source level. Neural Netw 2020;124:357-72. https://doi.org/10.1016/j.neunet.2020.01.027.
• [43] Fang T, Song Z, Zhan G, Zhang X, Mu W, Wang P, et al. Decoding motor imagery tasks using ESI and hybrid feature CNN. J Neural Eng 2022;19(1):016022. https://doi.org/10.1088/1741-2552/ac4ed0.
• [44] Li J, Zheng N. An EEG source imaging-based feature extraction method for motor imagery classification. Prague, Czech Republic: IEEE Syst Man Cybern Mag; 2022. p. 1648-52. https://doi.org/10.1109/SMC53654.2022.9945567.
• [45] Li MA, Ruan ZW. A novel decoding method for motor imagery tasks with 4D data representation and 3D convolutional neural networks. J Neural Eng 2021;18(4): 046029. https://doi.org/10.1088/1741-2552/abf68b.
• [46] Li MA, Ruan ZW. Decoding motor imagery with a simplified distributed dipoles model at source level. Cogn Neurodyn 2022;17:445-57. https://doi.org/10.1007/s11571-022-09826-x.
• [47] Hu Y, Liu Y, Zhang S, Zhang T, Dai B, Peng B, et al. A cross-space CNN with customized characteristics for motor imagery EEG classification. IEEE Trans Neural Syst Rehab Eng 2023;31:1554-65. https://doi.org/10.1109/TNSRE.2023.3249831.
• [48] Tangermann M, Müller KR, Aertsen A, Birbaumer N, Braun C, Brunner C, et al. Review of the BCI competition IV. Front Neurosci 2012;13(6):55. https://doi.org/ 10.3389/fnins.2012.00055.
• [49] Blankertz B, Müller K R, Krusienski D, Schalk G, Wolpaw J R, Schlogl A, Pfurtscheller G, Millan J R, Schroder M, Birbaumer N. Bci competition iii. Fraunhofer First. Ida 2005. http://ida.first. fraunhofer.de/projects/bci/competition_iii.
• [50] Pascual-Marqui RD. Standardized low-resolution brain electromagnetic tomography (sLORETA): technical details. Methods Find Exp Clin Pharmacol 2002; 24:5-12.
• [51] Gramfort A, Papadopoulo T, Olivi E, Maureen C. OpenMEEG: opensource software for quasistatic bioelectromagnetics. Biomed Eng Online 2010;9:45. https://doi. org/10.1186/1475-925X-9-45.
• [52] Hu J, Shen L, Albanie S, Sun G, Wu E. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 2020;42(8):2011-23. https://doi.org/10.1109/ TPAMI.2019.2913372.
• [53] Thio BJ, Aberra AS, Dessert GE, Grill WM. Ideal current dipoles are appropriate source representations for simulating neurons for intracranial recordings. Clin Neurophysiol 2023;145:26-35. https://doi.org/10.1016/j.clinph.2022.11.002.
• [54] Zhang R, Chen Y, Xu Z, Zhang L, Hu Y, Chen M. Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network. Front Neurosci 2023;17:1129049. https://doi.org/10.3389/ fnins.2023.1129049.
• [55] Xie Y, Oniga S. Classification of motor imagery EEG signals based on data augmentation and convolutional neural networks. Sensors 2023;23(4):1932. https://doi.org/10.3390/s23041932.