Sequence-to-sequence deep neural network with spatio-spectro and temporal features for motor imagery classification

• Autorzy: Rammy Sadaqat Ali , Abbas Waseem , Mahmood Syed Shahid , Riaz Haider , Rehman Haseeb Ur , Abideen Rao Zain Ul , Aqeel Muhammad , Zhang Wu

Identyfikatory

DOI

10.1016/j.bbe.2020.12.004

• Języki publikacji: EN

Abstrakty

EN

Electroencephalography (EEG) is a method of the brain–computer interface (BCI) that measures brain activities. EEG is a method of (non-)invasive recording ofthe electrical activity ofthe brain. This can be used to build BCIs. From the last decade, EEG has grasped researchers' attention to distinguish human activities. However, temporal information has rarely been retained to incorporate temporal information for multi-class (more than two classes) motor imagery classification. This research proposes a long-short-term-memory-based deep learning model to learn the hidden sequential patterns. Two types of features are used to feed the proposed model, including Fourier Transform Energy Maps (FTEMs) and Common Spatial Patterns (CSPs) filters. Multiple experiments have been conducted on a publicly available dataset. Extraction of spatial and spectro-temporal features using CSP filters and FTEM allow the sequence-tosequence based proposed model to learn the hidden sequential features. The proposed method is trained, evaluated, and optimized for a publicly available benchmark data set and resulted in 0.81 mean kappa value. Obtained results depict the model robustness for the artifacts and suitable for real-life applications with comparable classification accuracy. The code and findings will be available at https://github.com/waseemabbaas/Motor-Imagery-Classification.git.

Słowa kluczowe

EN

deep learning; convolutional neural network; long short term memory; electroencephalography; motor imagery

PL

uczenie głębokie; sieć neuronowa konwolucyjna; elektroencefalografia; wyobrażenie ruchu

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2021
• Tom: Vol. 41, no. 1
• Strony: 97--110
• Opis fizyczny: Bibliogr. 50 poz., rys., tab.

Twórcy

autor

Rammy Sadaqat Ali

• School of Computer Engineering and Sciences, Shanghai University, Shanghai, China

autor

Abbas Waseem

• Cloud Application Solutions Division, Mentor, A Siemens Business, Lahore, Pakistan

autor

Mahmood Syed Shahid

• Neurosurgery Department, Nishtar Medical University, Multan, Pakistan

autor

Riaz Haider

• Department of Computer & Software Engineering, National University of Sciences & Technology, Islamabad, Pakistan

autor

Rehman Haseeb Ur

• Department of Computer Sciences, Government College University Faisalabad, Pakistan

autor

Abideen Rao Zain Ul

• Department of Information & Technology, Bahauddin Zakariya University, Multan, Pakistan

autor

Aqeel Muhammad

• Institute of Research & Advanced Studies, Multan, Pakistan

autor

Zhang Wu

• School of Computer Engineering and Sciences, Shanghai University, Shanghai, China; Shanghai Institute of Applied Mathematics and Mechanics, Shanghai, China

Bibliografia

• [1] Li Y, Long J, Yu T, Yu Z, Wang C, Zhang H, et al. An EEGbased BCI system for 2-D cursor control by combining mu/ beta rhythm and p300 potential. IEEE Trans Biomed Eng 2010;57(10):2495–505.
• [2] Long J, Li Y, Wang H, Yu T, Pan J, Li F. A hybrid brain computer interface to control the direction and speed of a simulated or real wheelchair. IEEE Trans Neural Syst Rehabil Eng 2012;20(5):720–9.
• [3] LaFleur K, Cassady K, Doud A, Shades K, Rogin E, He B. Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface. J Neural Eng 2013;10(4):046003.
• [4] Lotte F, Congedo M, Lécuyer A, Lamarche F, Arnaldi B. A review of classification algorithms for EEG-based brain– computer interfaces. J Neural Eng 2007;4(2):R1.
• [5] Suk H-I, Lee S-W. A novel Bayesian framework for discriminative feature extraction in brain–computer interfaces. IEEE Trans Pattern Anal Mach Intell 2012;35 (2):286–99.
• [6] Lemm S, Blankertz B, Dickhaus T, M“uller K-R. Introduction to machine learning for brain imaging. Neuroimage 2011;56(2):387–99.
• [7] Asensio-Cubero J, Gan JQ, Palaniappan R. Extracting optimal tempo-spatial features using local discriminant bases and common spatial patterns for brain computer interfacing. Biomed Signal Process Control 2013;8(6):772–8.
• [8] Kam T-E, Suk H-I, Lee S-W. Non-homogeneous spatial filter optimization for electroencephalogram (EEG)-based motor imagery classification. Neurocomputing 2013;108:58–68.
• [9] Ghaheri H, Ahmadyfard A. Extracting common spatial patterns from EEG time segments for classifying motor imaginery classes in a brain computer interface (BCI) , Scientia Iranica. Trans D Comput Sci Eng Electr 2013;20(6);2061.
• [10] Metlická M. EEG signal pattern matching on GPU. Technical University Ostrava; 2013.
• [11] Stewart AX, Nuthmann A, Sanguinetti G. Single-trial classification of EEG in a visual object task using ICA and machine learning. J Neurosci Methods 2014;228:1–14.
• [12] Nicolas-Alonso LF, Corralejo R, Gomez-Pilar J, Álvarez D, Hornero R. Adaptive stacked generalization for multiclass motor imagery-based brain computer interfaces. IEEE Trans Neural Syst Rehabil Eng 2015;23(4):702–12.
• [13] Aghaei AS, Mahanta MS, Plataniotis KN. Separable common spatio-spectral patterns for motor imagery BCI systems. IEEE Trans Biomed Eng 2015;63(1):15–29.
• [14] Raza H, Cecotti H, Prasad G. A combination of transductive and inductive learning for handling non-stationarities in motor imagery classification. International Joint Conference on Neural Networks (IJCNN) 2016;763–70.
• [15] Zanini P, Congedo M, Jutten C, Said S, Berthoumieu Y. Transfer learning: a Riemannian geometry framework with applications to brain–computer interfaces. IEEE Trans Biomed Eng 2017;65(5):1107–16.
• [16] Hadsund JT, Leerskov KK. Feasibility of using error-related potentials as an appropriate method for adaptation in a brain–computer interface.
• [17] Abbas W, Khan NA. FBCSP-based multi-class motor imagery classification using BP and TDP features. 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2018;215–8.
• [18] Abbas W, Khan NA. DeepMI: deep learning for multiclass motor imagery classification. 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2018;219–22.
• [19] Sakhavi S, Guan C, Yan S. Learning temporal information for brain–computer interface using convolutional neural networks. IEEE Trans Neural Netw Learn Syst 2018;29 (11):5619–29.
• [20] Zhang R, Zong Q, Dou L, Zhao X. A novel hybrid deep learning scheme for four-class motor imagery classification. J Neural Eng 2019;16(6):066004.
• [21] Vaughan TM, Heetderks WJ, Trejo LJ, Rymer WZ, Weinrich M, Moore MM, et al. Brain–computer interface technology: a review of the second international meeting. IEEE Trans Neural Syst Rehabil Eng 2003;11(2):94–109.
• [22] Polat K, G“unes S. Classification of epileptiform EEG using a hybrid system based on decision tree classifier and fast Fourier transform. Appl Math Comput 2007;187 (2):1017–26.
• [23] Pfurtscheller G, Neuper C. Motor imagery and direct brain– computer communication. Proc IEEE 2001;89(7):1123–34.
• [24] Miao M, Zeng H, Wang A, Zhao C, Liu F. Discriminative spatial-frequency-temporal feature extraction and classification of motor imagery EEG: an sparse regression and weighted na”ive Bayesian classifier-based approach. J Neurosci Methods 2017;278:13–24.
• [25] Ang KK, Chin ZY, Wang C, Guan C, Zhang H. Filter bank common spatial pattern algorithm on BCI competition iv datasets 2A and 2B. Front Neurosci 2012;6:39.
• [26] Cecotti H, Graeser A. Convolutional neural network with embedded Fourier transform for EEG classification. 19th International Conference on Pattern Recognition 2008;1–4.
• [27] Bevilacqua V, Tattoli G, Buongiorno D, Loconsole C, Leonardis D, Barsotti M, et al. A novel BCI-SSVEP based approach for control of walking in virtual environment using a convolutional neural network. International Joint Conference on Neural Networks (IJCNN) 2014;4121–8.
• [28] Cecotti H, Graser A. Convolutional neural networks for p300 detection with application to brain–computer interfaces. IEEE Trans Pattern Anal Mach Intell 2010;33(3):433–45.
• [29] Helal MA, Eldawlatly S, Taher M. Using autoencoders for feature enhancement in motor imagery brain–computer interfaces. 13th IASTED International Conference on Biomedical Engineering (BioMed) 2017;89–93.
• [30] Zheng W-L, Zhu J-Y, Peng Y, Lu B-L. EEG-based emotion classification using deep belief networks. IEEE International Conference on Multimedia and Expo (ICME); 2014. pp. 1–6.
• [31] An X, Kuang D, Guo X, Zhao Y, He L. A deep learning method for classification of EEG data based on motor imagery. International Conference on Intelligent Computing. Springer; 2014. p. 203–10.
• [32] Lu N, Li T, Ren X, Miao H. A deep learning scheme for motor imagery classification based on restricted Boltzmann machines. IEEE Trans Neural Syst Rehabil Eng 2016;25 (6):566–76.
• [33] Sutskever I, Vinyals O, Le QV. Sequence to sequence learning with neural networks. Advances in neural information processing systems. 2014;3104–12.
• [34] Dai AM, Le QV. Semi-supervised sequence learning. Advances in neural information processing systems. 2015;3079–87.
• [35] Zhou J, Meng M, Gao Y, Ma Y, Zhang Q. Classification of motor imagery EEG using wavelet envelope analysis and LSTM networks. Chinese Control And Decision Conference (CCDC); 2018. pp. 5600–5.
• [36] Tabar YR, Halici U. A novel deep learning approach for classification of EEG motor imagery signals. J Neural Eng 2016;14(1):016003.
• [37] Schirrmeister RT, Springenberg JT, Fiederer LDJ, Glasstetter M, Eggensperger K, Tangermann M, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp 2017;38 (11):5391–420.
• [38] Xu B, Zhang L, Song A, Wu C, Li W, Zhang D, et al. Wavelet transform time-frequency image and convolutional network-based motor imagery EEG classification. IEEE Access 2018;7:6084–93.
• [39] Ma X, Qiu S, Du C, Xing J, He H. Improving EEG-based motor imagery classification via spatial and temporal recurrent neural networks. 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); 2018. pp. 1903–6.
• [40] Antelis JM, Falcón LE, et al. Spiking neural networks applied to the classification of motor tasks in EEG signals. Neural Netw 2020;122:130–43.
• [41] Rammy SA, Abrar M, Anwar SJ, Zhang W. Recurrent deep learning for EEG-based motor imagination recognition. 3rd International Conference on Advancements in Computational Sciences (ICACS); 2020. pp. 1–6.
• [42] Brunner C, Leeb R, M“uller-Putz G, Schl”ogl A, Pfurtscheller G. BCI competition 2008 – Graz data set A, 16. Institute for Knowledge Discovery (Laboratory of Brain– Computer Interfaces), Graz University of Technology; 2008. p. 1–6.
• [43] Tangermann M, Müller K-R, Aertsen A, Birbaumer N, Braun C, Brunner C, et al. Review of the BCI competition IV. Front Neurosci 2012;6:55.
• [44] Yang B, He M, Liu Y, Han Z. Multi-class feature extraction based on common spatial patterns of multi-band cross filter in bcis. International Computer Science Conference. Springer; 2012. p. 399–408.
• [45] Bengio Y, Simard P, Frasconi P, et al. Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 1994;5(2):157–66.
• [46] Bashivan P, Rish I, Yeasin M, Codella N. Learning representations from EEG with deep recurrentconvolutional neural networks. arXiv preprint arXiv:1511.06448.
• [47] Wang D, Miao D, Blohm G. Multi-class motor imagery EEG decoding for brain–computer interfaces. Front Neurosci 2012;6:151.
• [48] Abbas W, Khan NA. A discriminative spectral-temporal feature set for motor imagery classification. IEEE International Workshop on Signal Processing Systems (SiPS); 2017. pp. 1–6.
• [49] Asensio-Cubero J, Gan J, Palaniappan R. Multiresolution analysis over simple graphs for brain computer interfaces. J Neural Eng 2013;10(4):046014.
• [50] Abdelfattah SM, Abdelrahman GM, Wang M. Augmenting the size of EEG datasets using generative adversarial networks. International Joint Conference on Neural Networks (IJCNN); 2018. pp. 1–6.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2021).