Classification of pilots' mental states using a multimodal deep learning network

• Autorzy: Han Soo-Yeon , Kwak No-Sang , Oh Taegeun , Lee Seong-Whan

Identyfikatory

DOI

10.1016/j.bbe.2019.12.002

• Języki publikacji: EN

Abstrakty

EN

An automation system for detecting the pilot's diversified mental states is an extremely important and essential technology, as it could prevent catastrophic accidents caused by the deteriorated cognitive state of pilots. Various types of biosignals have been employed to develop the system, since they accompany neurophysiological changes corresponding to the mental state transitions. In this study, we aimed to investigate the feasibility of a robust detection system of the pilot's mental states (i.e., distraction, workload, fatigue, and normal) based on multimodal biosignals (i.e., electroencephalogram, electrocardiogram, respiration, and electrodermal activity) and a multimodal deep learning (MDL) network. To do this, first, we constructed an experimental environment using a flight simulator in order to induce the different mental states and to collect the biosignals. Second, we designed the MDL architecture – which consists of a convolutional neural network and long short-term memory models – to efficiently combine the information of the different biosignals. Our experimental results successfully show that utilizing multimodal biosignals with the proposed MDL could significantly enhance the detection accuracy of the pilot's mental states.

Słowa kluczowe

EN

multimodal deep learning; electroencephalography; electrocardiography; EDA; pilot's mental states

PL

uczenie głębokie; elektroencefalografia; elektrokardiografia; EDA; zdrowie psychiczne

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2020
• Tom: Vol. 40, no. 1
• Strony: 324--336
• Opis fizyczny: Bibliogr. 67 poz., rys., tab., wykr.

Twórcy

autor

Han Soo-Yeon

• Department of Brain and Cognitive Engineering, Korea University, Seoul, Korea

autor

Kwak No-Sang

• Department of Brain and Cognitive Engineering, Korea University, Seoul, Korea

autor

Oh Taegeun

• Agency for Defense Development, Daejeon, Korea

autor

Lee Seong-Whan

• Department of Brain and Cognitive Engineering, Korea University, Seoul, Korea; Department of Artificial Intelligence, Korea University, Seoul, Korea

Bibliografia

• [1] Hankins TC, Wilson GF. A comparison of heart rate, eye activity, EEG and subjective measures of pilot mental workload during flight. Aviat Space Environ Med 1998;69 (4):360–7.
• [2] Boksem MA, Tops M. Mental fatigue: costs and benefits. Brain Res Rev 2008;59(1):125–39.
• [3] Yen J-R, Hsu C-C, Yang H, Ho H. An investigation of fatigue issues on different flight operations. J Air Transp Manag 2009;15(5):236–40.
• [4] Wiegmann DA, Shappell SA. A human error approach to aviation accident analysis: the human factors analysis and classification system. Routledge; 2003.
• [5] Sonnleitner A, Treder MS, Simon M, Willmann S, Ewald A, Buchner A, et al. EEG alpha spindles and prolonged brake reaction times during auditory distraction in an on-road driving study. Accid Anal Prev 2014;62:110–8.
• [6] Borghini G, Astolfi L, Vecchiato G, Mattia D, Babiloni F. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci Biobehav Rev 2014;44:58–75.
• [7] Chaudhuri A, Routray A. Driver fatigue detection through chaotic entropy analysis of cortical sources obtained from scalp EEG signals. IEEE Trans Intell Transp Syst 2019.
• [8] Dehais F, Duprès A, Blum S, Drougard N, Scannella S, Roy RN, et al. Monitoring pilot's mental workload using ERPs and spectral power with a six-dry-electrode EEG system in real flight conditions. Sensors 2019;19(6):13–24.
• [9] Bashivan P, Rish I, Yeasin M, Codella N. Learning representations from EEG with deep recurrent- convolutional neural networks; 2015;1–15, arXiv:1511.06448.
• [10] Jiao Z, Gao X, Wang Y, Li J, Xu H. Deep convolutional neural networks for mental load classification based on EEG data. Pattern Recogn 2018;76:582–95.
• [11] Zhang P, Wang X, Zhang W, Chen J. Learning spatial-spectral-temporal EEG features with recurrent 3D convolutional neural networks for cross-task mental workload assessment. IEEE Trans Neural Syst Rehabil Eng 2018;27(1):31–42.
• [12] Wu EQ, Peng X, Zhang CZ, Lin J, Sheng RS. Pilots' fatigue status recognition using deep contractive autoencoder network. IEEE Trans Instrum Meas 2019.
• [13] Gao Z, Wang X, Yang Y, Mu C, Cai Q, Dang W, et al. EEG-based spatio-temporal convolutional neural network for driver fatigue evaluation. IEEE Trans Neural Netw Learn Syst 2019.
• [14] Patel M, Lal SK, Kavanagh D, Rossiter P. Applying neural network analysis on heart rate variability data to assess driver fatigue. Expert Syst Appl 2011;38(6):7235–42.
• [15] Hogervorst MA, Brouwer A-M, Van Erp JB. Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload. Front Neurosci 2014;8(322):1–14.
• [16] Ahn S, Nguyen T, Jang H, Kim JG, Jun SC. Exploring neuro-physiological correlates of drivers' mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and fNIRS data. Front Human Neurosci 2016;10(219):1–14.
• [17] Liu Y, Ayaz H, Shewokis PA. Multisubject ‘‘learning’’ for mental workload classification using concurrent EEG, fNIRS, and physiological measures. Front Human Neurosci 2017;11(389):1–14.
• [18] Zhang P, Wang X, Chen J, You W. Feature weight driven interactive mutual information modeling for heterogeneous bio-signal fusion to estimate mental workload. Sensors 2017;17(10):1–24.
• [19] Lee M-H, Fazli S, Mehnert J, Lee S-W. Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI. Pattern Recogn 2015;48(8):2725–37.
• [20] Suk H-I, Lee S-W. A novel Bayesian framework for discriminative feature extraction in brain–computer interfaces. IEEE Trans Pattern Anal Mach Intell 2012;35 (2):286–99.
• [21] Ding X, Lee S-W. Changes of functional and effective connectivity in smoking replenishment on deprived heavy smokers: a resting-state FMRI study. PLOS ONE 2013;8 (3):1–12.
• [22] Zhu X, Suk H-I, Lee S-W, Shen D. Canonical feature selection for joint regression and multi-class identification in Alzheimer's disease diagnosis. Brain Imaging Behav 2016;10(3):818–28.
• [23] Chen X, Zhang H, Zhang L, Shen C, Lee S-W, Shen D. Extraction of dynamic functional connectivity from brain grey matter and white matter for MCI classification. Human Brain Mapp 2017;38(10):5019–34.
• [24] Stehlin SA, Nguyen XP, Niemz MH. EEG with a reduced number of electrodes: where to detect and how to improve visually, auditory and somatosensory evoked potentials. Biocybern Biomed Eng 2018;38(3):700–7.
• [25] Kwak N-S, Lee S-W. Error correction regression framework for enhancing the decoding accuracies of ear-EEG brain– computer interfaces. IEEE Trans Cybern 2019;1–14.
• [26] Kam T-E, Suk H-I, Lee S-W. Non-homogeneous spatial filter optimization for electroencephalogram EEG-based motor imagery classification. Neurocomputing 2013;108:58–68.
• [27] Kim I-H, Kim J-W, Haufe S, Lee S-W. Detection of braking intention in diverse situations during simulated driving based on EEG feature combination. J Neural Eng 2014;12 (1):1–12.
• [28] Yeom S-K, Fazli S, Müller K-R, Lee S-W. An efficient ERP-based brain-computer interface using random set presentation and face familiarity. PLOS ONE 2014;9(11):1–13.
• [29] Suk H-I, Lee S-W, Shen D, Initiative ADN. Deep sparse multi-task learning for feature selection in Alzheimer's disease diagnosis. Brain Struct Funct 2016;221(5):2569–87.
• [30] Kwak N-S, Müller K-R, Lee S-W. A lower limb exoskeleton control system based on steady state visual evoked potentials. J Neural Eng 2015;12(5):1–14.
• [31] Górecka J, Walerjan P. Artifacts extraction from EEG data using the infomax approach. Biocybern Biomed Eng 2011;31 (4):59–74.
• [32] Lee M-H, Kwon O-Y, Kim Y-J, Kim H-K, Lee Y-E, Williamson J, et al. EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy. GigaScience 2019;8(5):1–16.
• [33] Lotfan S, Shahyad S, Khosrowabadi R, Mohammadi A, Hatef B. Support vector machine classification of brain states exposed to social stress test using EEG-based brain network measures. Biocybern Biomed Eng 2019;39(1):199–213.
• [34] Lal SK, Craig A, Boord P, Kirkup L, Nguyen H. Development of an algorithm for an EEG-based driver fatigue countermeasure. J Saf Res 2003;34(3):321–8.
• [35] Jap BT, Lal S, Fischer P, Bekiaris E. Using EEG spectral components to assess algorithms for detecting fatigue. Expert Syst Appl 2009;36(2):2352–9.
• [36] Kar S, Bhagat M, Routray A. EEG signal analysis for the assessment and quantification of driver's fatigue. Transp Res Part F: Traff Psychol Behav 2010;13(5):297–306.
• [37] Trejo LJ, Kubitz K, Rosipal R, Kochavi RL, Montgomery LD. EEG-based estimation and classification of mental fatigue. Psychology 2015;6(5):572–89.
• [38] Rogado E, Garcia JL, Barea R, Bergasa LM, López E. Driver fatigue detection system. 2008 IEEE International Conference on Robotics and Biomimetics 2009;1105–10.
• [39] Jung S-J, Shin H-S, Chung W-Y. Driver fatigue and drowsiness monitoring system with embedded electrocardiogram sensor on steering wheel. IET Intell Transp Syst 2014;8(1):43–50.
• [40] Hajinoroozi M, Mao Z, Jung T-P, Lin C-T, Huang Y. EEG-based prediction of driver's cognitive performance by deep convolutional neural network. Signal Process: Image Commun 2016;47:549–55.
• [41] Yang G, Lin Y, Bhattacharya P. A driver fatigue recognition model based on information fusion and dynamic Bayesian network. Inform Sci 2010;180(10):1942–54.
• [42] Shin H-S, Jung S-J, Kim J-J, Chung W-Y. Real time car driver's condition monitoring system. SENSORS 2010 IEEE 2010;951–4.
• [43] Khushaba RN, Kodagoda S, Lal S, Dissanayake G. Driver drowsiness classification using fuzzy wavelet-packet-based feature-extraction algorithm. IEEE Trans Biomed Eng 2010;58(1):121–31.
• [44] Rigas G, Goletsis Y, Bougia P, Fotiadis DI. Towards driver's state recognition on real driving conditions. Int J Vehic Technol 2011;2011:1–14.
• [45] Lee B-G, Lee B-L, Chung W-Y. Mobile healthcare for automatic driving sleep-onset detection using wavelet-based EEG and respiration signals. Sensors 2014;14(10):17915–36.
• [46] Fu R, Wang H, Zhao W. Dynamic driver fatigue detection using hidden Markov model in real driving condition. Expert Syst Appl 2016;63:397–411.
• [47] Sahayadhas A, Sundaraj K, Murugappan M, Palaniappan R. A physiological measures-based method for detecting inattention in drivers using machine learning approach. Biocybern Biomed Eng 2015;35(3):198–205.
• [48] Malik M. Heart rate variability: standards of measurement, physiological interpretation, and clinical use: Task Force of the European Society of Cardiology and the North American Society for Pacing and Electrophysiology. Ann Noninvas Electrocardiol 1996;1(2):151–81.
• [49] Skibniewski FW, Dziuda L, Baran PM, Krej MK, Guzowski S, Piotrowski MA, et al. Preliminary results of the LF/HF ratio as an indicator for estimating difficulty level of flight tasks. Aerosp Med Human Perform 2015;86(6):518–23.
• [50] Greco A, Valenza G, Bicchi A, Bianchi M, Scilingo EP. Assessment of muscle fatigue during isometric contraction using autonomic nervous system correlates. Biomed Signal Process Control 2019;51:42–9.
• [51] Bulling A, Ward JA, Gellersen H, Troster G. Eye movement analysis for activity recognition using electrooculography. IEEE Trans Pattern Anal Mach Intell 2010;33(4):741–53.
• [52] Çakir MP, Vural M, Koç SÖ, Toktas A. Real-time monitoring of cognitive workload of airline pilots in a flight simulator with fNIR optical brain imaging technology. International Conference on Augmented Cognition 2016;147–58.
• [53] Choi M, Koo G, Seo M, Kim SW. Wearable device-based system to monitor a driver's stress, fatigue, and drowsiness. IEEE Trans Instrum Meas 2017;67(3):634–45.
• [54] Hyvärinen A, Oja E. Independent component analysis: algorithms and applications. Neural Netw 2000;13(4–5):411–30.
• [55] Cochran WT, Cooley JW, Favin DL, Helms HD, Kaenel RA, Lang WW, et al. What is the fast Fourier transform? Proc IEEE 1967;55(10):1664–74.
• [56] Bulthoff HH, Lee S-W, Poggio T, Wallraven C. Biologically motivated computer vision. Springer-Verlag; 2003.
• [57] Hu D, Wang C, Nie F, Li X. Dense multimodal fusion for hierarchically joint representation. Proceedings of the International Conference on Acoustics Speech and Signal Processing (ICASSP) 2019;3941–5.
• [58] Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation 1997;9(8):1735–80.
• [59] Lemm S, Blankertz B, Dickhaus T, Müller K-R. Introduction to machine learning for brain imaging. NeuroImage 2011;56 (2):387–99.
• [60] Kwak N-S, Müller K-R, Lee S-W. A convolutional neural network for steady state visual evoked potential classification under ambulatory environment. PLOS ONE 2017;12(2):1–20.
• [61] Cover TM, Hart PE. Nearest neighbor pattern classification. IEEE Trans Inform Theory 1967;13(1):21–7.
• [62] Breiman L. Random forests. Mach Learn 2001;45(1):5–32.
• [63] Cox DR. The regression analysis of binary sequences. J R Stat Soc: Ser B Methodol 1958;20(2):215–32.
• [64] Defazio A, Bach F, Lacoste-Julien S. SAGA: a fast incremental gradient method with support for non- strongly convex composite objectives. Advances in Neural Information Processing Systems 2014;1646–54.
• [65] Cortes C, Vapnik V. Support-vector networks. Mach Learn 1995;20(3):273–97.
• [66] Chang C-C, Lin C-J. Libsvm: A library for support vector machines. ACM Trans Intell Syst Technol 2011;2 (3):1–27.
• [67] Mika S, Ratsch G, Weston J, Schölkopf B, Müller K-R. Fisher discriminant analysis with kernels. Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop 1999;41–8.

Uwagi

PL

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2020).