Bridging the gap between AI and human emotion: a multimodal recognition system

• Autorzy: Neeraja Ganta , Teja Jakkula Sai Surya , Ravi Kumar M. , Prasanna Lakshmi J. , Parvez Muzammil M. , Santhosh Chella

Identyfikatory

DOI

10.2478/candc-2024-0024

• Języki publikacji: EN

Abstrakty

EN

This study introduces a novel system that integrates voice and facial recognition technologies to enhance human-computer interaction by accurately interpreting and responding to user emotions. Unlike conventional approaches that analyze either voice or facial expressions in isolation, this system combines both modalities, offering a more comprehensive understanding of emotional states. By evaluating facial expressions, vocal tones, and contextual conversation history, the system generates personalized, context-aware responses, fostering more natural and empathetic AI interactions. This advancement significantly improves user engagement and satisfaction, paving the way for emotionally intelligent AI applications across diverse fields.

Słowa kluczowe

EN

emotion recognition; multimodal AI; human-computer interaction; facial recognition; voice analysis; empathetic AI

• Wydawca: Systems Research Institute, Polish Academy of Sciences
• Czasopismo: Control and Cybernetics
• Rocznik: 2024
• Tom: Vol. 53, No. 4
• Strony: 619--638
• Opis fizyczny: Bibliogr. 25 poz., rys.

Twórcy

autor

Neeraja Ganta

• Department of ECE, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Guntur (Dist), Andhra Pradesh, India 522302

autor

Teja Jakkula Sai Surya

• Department of ECE, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Guntur (Dist), Andhra Pradesh, India 522302

autor

Ravi Kumar M.

• Department of ECE, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Guntur (Dist), Andhra Pradesh, India 522302

autor

Prasanna Lakshmi J.

• Department of Electronics and Communication Engineering, KLH Deemed to Be University, Bowrampet, Hyderabad, Telangana, India 500090

autor

Parvez Muzammil M.

• Department of ECE, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Guntur (Dist), Andhra Pradesh, India 522302

autor

Santhosh Chella

• Department of ECE, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Guntur (Dist), Andhra Pradesh, India 522302

Bibliografia

• Alnuaim, A. A., Zakariah, M., Alhadlaq, A., Shashidhar, C., Hatamleh, W. A., Tarazi, H., Shukla, P. K. and Ratna, R. (2022) Human-Computer Interaction with Detection of Speaker Emotions Using Convolution Neural Networks. Computational Intelligence and Neuroscience, 2022 (1): 7463091.
• Baevski, A., Zhou, Y., Mohamed, A. and Auli, M. (2020) wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449-12460.
• Chandolikar, N., Joshi, C., Roy, P., Gawas, A. and Vishwakarma, M. (2022, March) Voice recognition: A comprehensive survey. In: 2022 International Mobile and Embedded Technology Conference (MECON). IEEE, 45–51.
• Chowdary, M. K., Nguyen, T.N. and Hemanth, D. J. (2023) Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Computing and Applications, 35 (32): 23311–28.
• Ekman, P. and Friesen, W. V. (1971) Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17 (2), 124.
• Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., André, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S. and Truong, K. P. (2015) The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7(2), 190–202.
• Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D. H. and Zhou, Y. (2013) Challenges in representation learning: A report on three machine learning contests. In: Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3-7, 2013. Proceedings, Part III 20. Springer Berlin Heidelberg, 117–124.
• Kanna, R. K., Surendhar, P. A., Rubi, J., Jyothi, G., Ambikapathy, A. and Vasuki, R. (2022) Human Computer Interface Application for Emotion Detection Using Facial Recognition. In: 2022 IEEE International Conference on Current Development in Engineering and Technology (CCET). IEEE, 1–7.
• Khare, S. K., Blanes-Vidal, V., Nadimi, E. S. and Acharya, U. R. (2024) Emotion recognition and artificial intelligence: A systematic review (2014–2023) and research recommendations. Information Fusion, 102, 102019.
• Khattak, A., Asghar, M. Z., Ali, M. and Batool, U. (2022) An efficient deep learning technique for facial emotion recognition. Multimedia Tools and Applications, January, 81 (2): 1649–1683.
• Kumar, H. and Martin, A. (2023) Artificial Emotional Intelligence: Conventional and deep learning approach. Expert Systems with Applications, February, 1, 212: 118651.
• Lim, Y., Ng, K. W., Naveen, P. and Haw, S. C. (2022) Emotion recognition by facial expression and voice: review and analysis. Journal of Informatics and Web Engineering, 1(2), 45–54.
• Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A. and Cambria, E. (2019) Dialoguernn: An attentive rnn for emotion detection in conversations. In: Proceedings of the AAAI conference on artificial intelligence, 33, 01. AAAI, 6818–6825.
• Mannar Mannan, J., Srinivasan, L., Maithili, K. and Ramya, C. (2023) Human emotion recognize using convolutional neural network (CNN) and Mel frequency cepstral coefficient (MFCC). Seybold Report Journal, 18 (4): 49–61.
• Mansouri, A., Affendey, L. S. and Mamat, A. (2008) Named entity recognition approaches. International Journal of Computer Science and Network Security, 8(2), 339–344.
• Poria, S., Cambria, E., Bajpai, R. and Hussain, A. (2017) A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98–125.
• Raji, I. D. and Buolamwini, J. (2019) Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. ACM, 429–435.
• Samek, W., Montavon, G., Vedaldi, A., Hansen, L. K. and Müller, K. R., eds. (2019) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. LNCS 11700 Springer Nature.
• Sarvakar, K., Senkamalavalli, R., Raghavendra, S., Kumar, J. S., Manjunath, R. and Jaiswal, S. (2023) Facial emotion recognition using convolutional neural networks. Materials Today: Proceedings, January, 1, 80: 3560-3564.
• Scherer, K. R. (2003) Vocal communication of emotion: A review of research paradigms. Speech Communication, 40 (1-2), 227–256.
• Shahzad, H. M., Bhatti, S. M., Jaffar, A., Akram, S., Alhajlah, M. and Mahmood, A. (2023) Hybrid facial emotion recognition using CNN-based features. Applied Sciences, 13 (9), 5572.
• Trinh, V. L, Dao, Th. L. T., Le Xuan, T. and Castelli, E. (2022) Emotional speech recognition using deep neural networks. Sensors, February, 12, 22 (4), 1414.
• Venkatesan, R., Shirly, S., Selvarathi, M. and Jebaseeli, T. J. (2023) Human Emotion Detection Using DeepFace and Artificial Intelligence. Engineering Proceedings, 59(1), 37.
• Zafar, A., Aamir, M., Mohd Nawi, N., Arshad, A., Riaz, S., Alruban, A., Almotairi, S. and Dutta, A. K. (2022) A comparison of pooling methods for convolutional neural networks. Applied Sciences, 12 (17), 8643.
• Zeng, Z., Pantic, M., Roisman, G. I. and Huang, T. S. (2007) A survey of affect recognition methods: audio, visual and spontaneous expressions. In: Proceedings of the 9th International Conference on Multimodal Interfaces. ACM, 126–133.