Gray-level co-occurrence matrix of Smooth Pseudo Wigner-Ville distribution for cognitive workload estimation

• Autorzy: Mirzaeian Rezvan , Ghaderyan Peyvand

Identyfikatory

DOI

10.1016/j.bbe.2023.01.001

• Języki publikacji: EN

Abstrakty

EN

Automatic, cost-effective, and reliable cognitive workload estimation (CWE) is one of the important issues in diagnosis and treatment of neurocognitive diseases, cognitive performance improvement and error preventive strategies. To address this issue, this paper has proposed a novel and robust CWE method by detecting the time–frequency (TF) changes of electrodermal activities (EDA). Firstly, the local and global properties of the time-variant characteristics of EDA have been presented using Smooth Pseudo WignerVille distribution with enhanced TF resolution. Then, the transient changes in TF images of EDA signals have been quantified using a set of textural features based on Gray Level Co-occurrence Matrix descriptor (GLCM). Several static and dynamic classifiers, such as support vector machine, K- k-nearest neighbor, cascade forward neural network, and recurrent neural network have been explored. A real EDA data experiment recorded during arithmetic task with different workload levels have been used to evaluate the performance of the proposed method. The obtained results have confirmed that it can achieve a high estimation performance of 97.71% using contrast feature for discrimination of three workload levels. Further analysis has also suggested that the model is robust to GLCM parameters and classifiers and can provide a better tradeoff between computational complexity and high performance using minimum number of textural features in comparison with previous studies.

Słowa kluczowe

EN

electrodermal activity; textural features; cascade forward neural network; recurrent neural network

PL

aktywność elektrodermalna; cecha tekstury; sieć neuronowa rekurencyjna

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2023
• Tom: Vol. 43, no. 1
• Strony: 261--278
• Opis fizyczny: Bibliogr. 80 poz., rys., tab., wykr.

Twórcy

autor

Mirzaeian Rezvan

• Faculty of Biomedical Engineering, Sahand University of Technology, Tabriz, Iran

autor

Ghaderyan Peyvand

• Computational Neuroscience Laboratory, Faculty of Biomedical Engineering, Sahand University of Technology, Tabriz, Iran

Bibliografia

• [1] Ghaderyan P, Abbasi A. Dynamic Hilbert warping, a new measure of RR-interval signals evaluated in the cognitive load estimation. Med Eng Phys 2017;40:103-9.
• [2] Ghaderyan P, Abbasi A. Sparse coding classification and cepstral singular value for cognitive workload estimation. Comput Electr Eng 2021;91 107031.
• [3] Reiner M, Gelfeld TM. Estimating mental workload through event-related fluctuations of pupil area during a task in a virtual world. Int J Psychophysiol 2014;93:38-44.
• [4] Paas F, Tuovinen JE, Tabbers H, Van Gerven PW. Cognitive load measurement as a means to advance cognitive load theory. Educational psychologist: Routledge; 2016. p. 63-71.
• [5] Borghini G, Astolfi L, Vecchiato G, Mattia D, Babiloni F. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci Biobehav Rev 2014;44:58-75.
• [6] Chen S, Epps J. Automatic classification of eye activity for cognitive load measurement with emotion interference. Comput Methods Programs Biomed 2013;110:111-24.
• [7] Ghaderyan P, Abbasi A. A novel cepstral-based technique for automatic cognitive load estimation. Biomed Signal Process Control 2018;39:396-404.
• [8] Galy E, Cariou M, Mélan C. What is the relationship between mental workload factors and cognitive load types? Int J Psychophysiol 2012;83:269-75.
• [9] Ghaderyan P, Abbasi A. An efficient automatic workload estimation method based on electrodermal activity using pattern classifier combinations. Int J Psychophysiol 2016;110:91-101.
• [10] Boucsein W. Electrodermal activity. Springer Science & Business Media; 2012.
• [11] Horinouchi T, Sakurai K, Munekata N, Kurita T, Takeda Y, Kusumi I. Decreased electrodermal activity in patients with epilepsy. Epilepsy Behav 2019;100 106517.
• [12] Alvarez V, Reinsberger C, Scirica B, O’Brien MH, Avery KR, Henderson G, et al. Continuous electrodermal activity as a potential novel neurophysiological biomarker of prognosis after cardiac arrest – A pilot study. Resuscitation 2015;93:128-35.
• [13] Bach DR, Friston KJ. Model-based analysis of skin conductance responses: Towards causal models in psychophysiology. Psychophysiology 2013;50:15-22.
• [14] Ghaderyan P, Abbasi A, Ebrahimi A. Time-varying singular value decomposition analysis of electrodermal activity: a novel method of cognitive load estimation. Measurement 2018;126:102-9.
• [15] FiĬo P, Janoušek O. The relation between physical and mental load, and the course of physiological functions and cognitive performance. Theor Issues Ergon Sci 2022;23:38-59.
• [16] Neukirchen T, Stork M, Hoppe MW, Vorstius C. Spirometry has added value over electrodermal activity as a physiological marker of mental load in male subjects. Sci Rep 2022;12:1-8.
• [17] Lagomarsino M, Lorenzini M, De Momi E, Ajoudani A. An Online Framework for Cognitive Load Assessment in Industrial Tasks. Rob Comput Integr Manuf 2022;78 102380.
• [18] Widyanti A, Muslim K, Sutalaksana IZ. The sensitivity of Galvanic Skin Response for assessing mental workload in Indonesia. Work 2017;56:111-7.
• [19] Posada-Quintero HF, Florian JP, Orjuela-Cañón AD, Aljama-Corrales T, Charleston-Villalobos S, Chon KH. Power spectral density analysis of electrodermal activity for sympathetic function assessment. Ann Biomed Eng 2016;44:3124-35.
• [20] Salah J, Abdelrahman Y, Abdrabou Y, Kassem K, Abdennadher S. Exploring the usage of commercial biosensors for multitasking detection. Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia; 2018. p. 265-77.
• [21] Aricò P, Reynal M, Di Flumeri G, Borghini G, Sciaraffa N, Imbert J-P, et al. How neurophysiological measures can be used to enhance the evaluation of remote tower solutions. Front Hum Neurosci 2019;13:303.
• [22] Ding Y, Cao Y, Wang Y. Physiological indicators of mental workload in visual display terminal work. In: International Conference on Applied Human Factors and Ergonomics. Springer; 2019. p. 86-94.
• [23] Wulvik AS, Dybvik H, Steinert M. Investigating the relationship between mental state (workload and affect) and physiology in a control room setting (ship bridge simulator). Cogn Tech Work 2020;22:95-108.
• [24] Buchwald M, Kupiński S, Bykowski A, Marcinkowska J, Ratajczyk D, Jukiewicz M. Electrodermal activity as a measure of cognitive load: a methodological approach. 2019 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA): IEEE; 2019. p. 175-9.
• [25] Hossain D, Salimullah SM, Mahmudi R, Hasan SMN, Kabir E, Chowdhury AN, et al. Cognitive load measurement using galvanic skin response for listening tasks. 2019 4th International Conference on Electrical Information and Communication Technology (EICT): IEEE; 2019. p. 1-4.
• [26] Marucci M, Di Flumeri G, Borghini G, Sciaraffa N, Scandola M, Pavone EF, et al. The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence. Sci Rep 2021;11:1-15.
• [27] Loeches De La Fuente H, Berthelon C, Fort A, Etienne V, De Weser M, Ambeck J, et al. Electrophysiological and performance variations following driving events involving an increase in mental workload. Eur Transp Res Rev 2019;11:1-9.
• [28] Armougum A, Gaston-Bellegarde A, Joie-La Marle C, Piolino P. Physiological investigation of cognitive load in real-life train travelers during information processing. Appl Ergon 2020;89 103180.
• [29] Nourbakhsh N, Wang Y, Chen F. GSR and blink features for cognitive load classification. IFIP conference on human-computer interaction: Springer; 2013. p. 159-66.
• [30] Solovey ET, Zec M, Garcia Perez EA, Reimer B, Mehler B. Classifying driver workload using physiological and driving performance data: two field studies. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems2014. p. 4057-66.
• [31] Arroyo-Gómez N, Laparra-Hernández J, Soler-Valero A, Medina E, Rosario Hd. Physiological Model to Classify Physical and Cognitive Workload During Gaming Activities. International Conference on Applied Human Factors and Ergonomics: Springer; 2017. p. 246-54.
• [32] van Gent P, Farah H, Nes NV, van Arem B, van Gent P, Farah H, et al. Towards Real-Time, Nonintrusive Estimation of Driver Workload in a Simulated Environment. Proceedings of the 6th International Conference on Road Safety & Simulation (RSS)2017.
• [33] Zhao G, Liu Y-J, Shi Y. Real-time assessment of the cross-task mental workload using physiological measures during anomaly detection. IEEE Trans Hum-Mach Syst 2018;48:149-60.
• [34] Manawadu UE, Kawano T, Murata S, Kamezaki M, Muramatsu J, Sugano S. Multiclass classification of driver perceived workload using long short-term memory based recurrent neural network. In: 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE; 2018. p. 1-6.
• [35] Elahi MTE, Islam IB. Galvanic skin response signal based cognitive load classification using machine learning classifier. In: 2019 3rd International Conference on Electrical, Computer & Telecommunication Engineering (ICECTE). IEEE; 2019. p. 33-6.
• [36] Luong T, Martin N, Raison A, Argelaguet F, Diverrez J-M, Lécuyer A. Towards real-time recognition of users mental workload using integrated physiological sensors into a VR HMD. In: 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE; 2020. p. 425-37.
• [37] Rajavenkatanarayanan A, Nambiappan HR, Kyrarini M, Makedon F. Towards a real-time cognitive load assessment system for industrial human-robot cooperation. 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN): IEEE; 2020. p. 698-705.
• [38] Feradov F, Ganchev T, Markova V. Automated Detection of Cognitive Load from Peripheral Physiological Signals based on Hjorth’s Parameters. 2020 International Conference on Biomedical Innovations and Applications (BIA): IEEE; 2020. p. 85-8.
• [39] Tervonen J, Pettersson K, Mäntyjärvi J. Ultra-short window length and feature importance analysis for cognitive load detection from wearable sensors. Electronics 2021;10:613.
• [40] Posada-Quintero HF, Bolkhovsky JB. Machine learning models for the identification of cognitive tasks using autonomic reactions from heart rate variability and electrodermal activity. Behavioral Sciences 2019;9:45.
• [41] Han S-Y, Kim J-W, Lee S-W. Recognition of pilot’s cognitive states based on combination of physiological signals. 2019 7th International Winter Conference on Brain-Computer Interface (BCI): IEEE; 2019. p. 1-5.
• [42] Ding Y, Cao Y, Duffy VG, Wang Y, Zhang X. Measurement and identification of mental workload during simulated computer tasks with multimodal methods and machine learning. Ergonomics 2020;63:896-908.
• [43] Jarvis J, Putze F, Heger D, Schultz T. Multimodal person independent recognition of workload related biosignal patterns. Proceedings of the 13th international conference on multimodal interfaces ;2011. p. 205-8.
• [44] Zhang H, Zhu Y, Maniyeri J, Guan C. Detection of variations in cognitive workload using multi-modality physiological sensors and a large margin unbiased regression machine. 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society: IEEE; 2014. p. 2985-8.
• [45] Jimenez-Molina A, Retamal C, Lira H. Using psychophysiological sensors to assess mental workload during web browsing. Sensors 2018;18:458.
• [46] Siriya S, Lochner M, Duenser A, Taib R. Exploring Novel Methodology for Classifying Cognitive Workload. In: International Conference on Intelligent Technologies for Interactive Entertainment. Springer; 2018. p. 105-14.
• [47] Radhakrishnan V, Merat N, Louw T, Gonçalves RC, Torrao G, Lyu W, et al. Physiological indicators of driver workload during car-following scenarios and takeovers in highly automated driving. Transport Res F: Traffic Psychol Behav 2022;87:149-63.
• [48] Krishnan S, Athavale Y. Trends in biomedical signal feature extraction. Biomed Signal Process Control 2018;43:41-63.
• [49] Saha S, Jindal K, Shakti D, Tewary S, Sardana V. Chirplet transform-based machine-learning approach towards classification of cognitive state change using galvanic skin response and photoplethysmography signals. Expert Syst 2022:e12958.
• [50] Aricò P, Reynal M, Di Flumeri G, Borghini G, Sciaraffa N, Imbert J-P, et al. How neurophysiological measures can be used to enhance the evaluation of remote tower solutions. Front Hum Neurosci 2019;303.
• [51] Zihisire Muke P, Telec Z, Trawiński B. Cognitive Load Measurement Using Arithmetic and Graphical Tasks and Galvanic Skin Response. In: Conference on Computational Collective Intelligence Technologies and Applications. Springer; 2022. p. 836-50.
• [52] Malagoli A, Corradini M, Corradini P, Shuett T, Fonda S. Towards a method for the objective assessment of cognitive workload: A pilot study in vessel traffic service (VTS) of maritime domain. 2017 IEEE 3rd International Forum on Research and Technologies for Society and Industry (RTSI): IEEE; 2017. p. 1-6.
• [53] Romine WL, Schroeder NL, Graft J, Yang F, Sadeghi R, Zabihimayvan M, et al. Using machine learning to train a wearable device for measuring students’ cognitive load during problem-solving activities based on electrodermal activity, body temperature, and heart rate: development of a cognitive load tracker for both personal and classroom use. Sensors 2020;20:4833.
• [54] Hirachan N, Mathews A, Romero J, Rojas RF. Measuring Cognitive Workload Using Multimodal Sensors. arXiv preprint arXiv:220504235. 2022.
• [55] Romine W, Schroeder N, Banerjee T, Graft J. Toward Mental Effort Measurement Using Electrodermal Activity Features. Sensors 2022;22:7363.
• [56] Jaiswal D, Moulick M, Chatterjee D, Ranjan R, Ramakrishnan RK, Pal A, et al. Assessment of cognitive load from biopotentials measured using wearable endosomatic device. Proceedings of the 6th ACM Workshop on Wearable Systems and Applications2020. p. 13-8.
• [57] Hlawatsch F, Auger F. Time-frequency analysis. John Wiley & Sons; 2013.
• [58] Meignen S, Oberlin T, Wu H-T. Time-frequency reassignment and synchrosqueezing. IEEE Signal Process Mag 2013;30:32-41.
• [59] Li X, Xi L, Yuan X, Zhang X. Application of the joint algorithm of smooth pseudo Wigner-Ville distribution and fourparameter particle swarm optimization to BOTDR. Results Phys 2021;25 104215.
• [60] Li Y-F, Zhang L-H, Geng X-C, Jin Z, Wang H-X, Lin R, et al. The Comparison and Study of WVD, PWVD, SPWVD Time Frequency Analysis Technologies Based on Observed Dynamic EEG Data. Proceedings of 2017 2nd International Conference on Information Technology and Management Engineering (ITME 2017) Shanghai: IEEE 2017. p. 1-7.
• [61] Veeranki YR, Ganapathy N, Swaminathan R. Electrodermal Activity Based Emotion Recognition using Time-Frequency Methods and Machine Learning Algorithms. Current Directions in Biomedical Engineering 2021;7:863-6.
• [62] Wang P, Qiao H, Zhang Y, Li Y, Feng Q, Chen K. Meso-damage evolution analysis of magnesium oxychloride cement concrete based on X-CT and grey-level co-occurrence matrix. Constr Build Mater 2020;255 119373.
• [63] Maktabdar Oghaz M, Maarof MA, Rohani MF, Zainal A, Shaid SZM. An optimized skin texture model using gray-level cooccurrence matrix. Neural Comput & Applic 2019;31:1835-53.
• [64] Mamli S, Kalbkhani H. Gray-level co-occurrence matrix of Fourier synchro-squeezed transform for epileptic seizure detection. Biocybernetics and Biomedical Engineering 2019;39:87-99.
• [65] Warsito B, Santoso R, Yasin H. Cascade forward neural network for time series prediction. Journal of Physics: Conference Series: IOP Publishing 2018 012097.
• [66] Wiebe EN, Roberts E, Behrend TS. An examination of two mental workload measurement approaches to understanding multimedia learning. Comput Hum Behav 2010;26:474-81.
• [67] Zarjam P, Epps J, Chen F, Lovell NH. Estimating cognitive workload using wavelet entropy-based features during an arithmetic task. Comput Biol Med 2013;43:2186-95.
• [68] Imbo I, Vandierendonck A, De Rammelaere S. The role of working memory in the carry operation of mental arithmetic: Number and value of the carry. Q J Exp Psychol 2007;60:708-31.
• [69] Ezazi Y, Ghaderyan P. Textural feature of EEG signals as a new biomarker of reward processing in Parkinson’s disease detection. Biocybernetics and Biomedical Engineering 2022;42:950-62.
• [70] Gao S. Gray level co-occurrence matrix and extreme learning machine for Alzheimer’s disease diagnosis. International Journal of Cognitive Computing in Engineering 2021;2:116-29.
• [71] Sun W, Zeng N, He Y. Morphological arrhythmia automated diagnosis method using gray-level co-occurrence matrix enhanced convolutional neural network. IEEE Access 2019;7:67123-9.
• [72] Osowski S, Siwek K, Markiewicz T. In: MLP and SVM networks-a comparative study. IEEE; 2004. p. 37-40.
• [73] Ray S. A quick review of machine learning algorithms. In: 2019 International conference on machine learning, big data, cloud and parallel computing (COMITCon): IEEE. p. 35-9.
• [74] Ray S. An analysis of computational complexity and accuracy of two supervised machine learning algorithms-K-nearest neighbor and support vector machine. Data Management, Analytics and Innovation: Springer; 2021. p. 335-47.
• [75] Ghaderyan P, Abbasi A, Sedaaghi MH. An efficient seizure prediction method using KNN-based undersampling and linear frequency measures. J Neurosci Methods 2014;232:134-42.
• [76] Binkhonain M, Zhao L. A review of machine learning algorithms for identification and classification of non-functional requirements. Expert Systems with Applications: X 2019;1 100001.
• [77] Krichene E, Masmoudi Y, Alimi AM, Abraham A, Chabchoub H. Forecasting using Elman recurrent neural network. International Conference on Intelligent Systems Design and Applications: Springer; 2016. p. 488-97.
• [78] González JJ, Méndez LD, Mañas S, Duque MR, Pereda E, De Vera L. Performance analysis of univariate and multivariate EEG measurements in the diagnosis of ADHD. Clin Neurophysiol 2013;124:1139-50.
• [79] Saljuqi M, Ghaderyan P. A novel method based on matching pursuit decomposition of gait signals for Parkinson’s disease, Amyotrophic lateral sclerosis and Huntington’s disease detection. Neurosci Lett 2021;761 136107.
• [80] Zhou J, Jung JY, Chen F. Dynamic workload adjustments in human-machine systems based on GSR features. IFIP Conference on Human-Computer Interaction: Springer 2015:550-8.

Uwagi

PL

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2022-2023)