Study on deep learning models for VR sickness levels classification

• Autorzy: Na Hae Chan , Kim Yoon Sang

Identyfikatory

DOI

10.35784/acs-2024-37

• Języki publikacji: EN

Abstrakty

EN

Virtual Reality (VR) sickness is often accompanied by symptoms such as nausea and dizziness, and a prominent theory explaining this phenomenon is the sensory conflict theory. Recently, studies have used Deep Learning to classify VR sickness levels; however, there is a paucity of research on Deep Learning models that utilize both visual information and motion data based on sensory conflict theory. In this paper, the authors propose a parallel merging of a Deep Learning model (4bay) to classify the level of VR sickness by utilizing the user's motion data (HMD, controller data) and visual data (rendered image, depth image) based on sensory conflict theory. The proposed model consists of a visual processing module, a motion processing module, and an FC-based VR sickness level classification module. The performance of the proposed model was compared with that of the developed models at the time of design. As a result of the comparison, it was confirmed that the proposed model performed better than the single model and the merged (2bay) model in classifying the user's VR sickness level.

Słowa kluczowe

EN

VR sickness; cybersickness; Deep Learning; LSTM; ResNet; sensory conflict theory

• Wydawca: Polskie Towarzystwo Promocji Wiedzy ; Lublin University of Technology
• Czasopismo: Applied Computer Science
• Rocznik: 2024
• Tom: Vol. 20, no. 4
• Strony: 1--13
• Opis fizyczny: Bibliogr. 20 poz., fig., tab.

Twórcy

autor

Na Hae Chan

• Korea University of Technology and Education, Institute for Bioengineering Application Technology, Department of Computer Science and Engineering, BioComputing Lab, Republic of Korea

• https://orcid.org/0009-0002-5027-2832

autor

Kim Yoon Sang

• Korea University of Technology and Education, Institute for Bioengineering Application Technology, Department of Computer Science and Engineering, BioComputing Lab, Republic of Korea

• https://orcid.org/0000-0002-0416-7938

Bibliografia

• [1] Du, M., Cui, H., Wang, Y., & Duh, H. B. L. (2021). Learning from deep stereoscopic attention for simulator sickness prediction. IEEE Transactions on Visualization and Computer Graphics, 29(2), 1415-1423. https://doi.org/10.1109/TVCG.2021.3115901
• [2] Falkowicz, K., & Kulisz, M. (2024). Prediction of buckling behaviour of composite plate element using artificial neural networks. Advances in Science and Technology. Research Journal, 18(1). https://doi.org/10.12913/22998624/177399
• [3] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual Learning for image recognition. IEEE conference on computer vision and pattern recognition (CVPR) (pp. 770-778). IEEE. https://doi.org/10.1109/CVPR.2016.90
• [4] Jeong, D., Paik, S., Noh, Y., & Han, K. (2023). MAC: multimodal, attention-based cybersickness prediction modeling in virtual reality. Virtual Reality, 27(3), 2315-2330. https://doi.org/10.1007/s10055-023-00804-0
• [5] Karpiński, R., Krakowski, P., Jonak, J., Machrowska, A., & Maciejewski, M. (2023). Comparison of selected classification methods based on Machine Learning as a diagnostic tool for knee joint cartilage damage based on generated vibroacoustic processes. Applied Computer Science, 19(4), 136-150. https://doi.org/10.35784/acs-2023-40
• [6] Keshavarz, B., Peck, K., Rezaei, S., & Taati, B. (2022). Detecting and predicting visually induced motion sickness with physiological measures in combination with Machine Learning techniques. International Journal of Psychophysiology, 176, 14-26. https://doi.org/10.1016/j.ijpsycho.2022.03.006
• [7] Kulisz, M., Kujawska, J., Cioch, M., Cel, W., & Pizoń, J. (2024). Comparative analysis of Machine Learning methods for predicting energy recovery from waste. Applied Sciences, 14(7), 2997. https://doi.org/10.3390/app14072997
• [8] Kundu, R. K., Islam, R., Quarles, J., & Hoque, K. A. (2023). LiteVR: Interpretable and lightweight cybersickness detection using explainable AI. 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR) (pp. 609-619). IEEE. https://doi.org/10.1109/VR55154.2023.00076
• [9] LaViola, Jr, J. J. (2000). A discussion of cybersickness in virtual environments. ACM Sigchi Bulletin, 32(1), 47-56. https://doi.org/10.1145/333329.333344
• [10] Lim, H. K., Ji, K., Woo, Y. S., Han, D. U., Lee, D. H., Nam, S. G., & Jang, K. M. (2021). Test-retest reliability of the virtual reality sickness evaluation using electroencephalography (EEG). Neuroscience Letters, 743, 135589. https://doi.org/10.1016/j.neulet.2020.135589
• [11] Monteiro, D., Liang, H. N., Tang, X., & Irani, P. (2021). Using trajectory compression rate to predict changes in cybersickness in virtual reality games. 2021 IEEE international symposium on mixed and augmented reality (ISMAR), (pp. 138-146). IEEE. https://doi.org/10.1109/ISMAR52148.2021.00028
• [12] Ng, A. K. T., Chan, L. K. Y., & Lau, H. Y. K. (2020). A study of cybersickness and sensory conflict theory using a motion-coupled virtual reality system. Displays, 61, 101922. https://doi.org/10.1016/j.displa.2019.08.004
• [13] Shimada, S., Ikei, Y., Nishiuchi, N., & Yem, V. (2023a). Study of cybersickness prediction in real time using eye tracking data. 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 871-872). IEEE. https://doi.org/10.1109/VRW58643.2023.00278
• [14] Shimada, S., Pannattee, P., Ikei, Y., Nishiuchi, N., & Yem, V. (2023b). High-frequency cybersickness prediction using Deep Learning techniques with eye-related indices. IEEE Access, 11, 95825-95839. https://doi.org/10.1109/ACCESS.2023.3312216
• [15] Shodipe, O. E., & Allison, R. S. (2023). Modelling the relationship between the objective measures of car sickness. 2023 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE) (pp. 570-575). IEEE. https://doi.org/10.1109/CCECE58730.2023.10289000
• [16] Wang, J., Liang, H. N., Monteiro, D. V., Xu, W., Chen, H., & Chen, Q. (2020). Real-time detection of simulator sickness in virtual reality games based on players' psychophysiological data during gameplay. 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (pp. 247-248). IEEE. https://doi.org/10.1109/ISMAR-Adjunct51615.2020.00071
• [17] Wen, E., Gupta, C., Sasikumar, P., Billinghurst, M., Wilmott, J., Skow, E., Dey. A., & Nanayakkara, S. (2024). VR. net: A real-world dataset for virtual reality motion sickness research. IEEE Transactions on Visualization and Computer Graphics, 30(5), 2330-2336. https://doi.org/10.1109/TVCG.2024.3372044
• [18] Yang, A. H. X., Kasabov, N., & Cakmak, Y. O. (2022). Machine Learning methods for the study of cybersickness: A systematic review. Brain Informatics, 9(1), 24. https://doi.org/10.1186/s40708-022-00172-6
• [19] Younis, M. C. (2024). Prediction of patient’s willingness for treatment of mental illness using Machine Learning approaches. Applied Computer Science, 20(2), 175-193. https://doi.org/10.35784/acs-2024-23
• [20] Zhao, J., Tran, K. T., Chalmers, A., Hoh, W. K., Yao, R., Dey, A., Wilmott, J., Lin, J., Billinghurst, M., Lindeman, & Rhee, T. (2023). Deep Learning-based simulator sickness estimation from 3D motion. 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 39-48). IEEE. https://doi.org/10.1109/ISMAR59233.2023.00018

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr POPUL/SP/0154/2024/02 w ramach programu "Społeczna odpowiedzialność nauki II" - moduł: Popularyzacja nauki (2025).