Spatial sound and emotions : a literature survey on the relationship between spatially rendered audio and listeners’ affective responses

• Autorzy: Stefanowska, Antonina , Zieliński, Sławomir K.
• Języki publikacji: EN

Abstrakty

EN

With the development of the entertainment industry, the need for immersive and emotionally impactful sound design has emerged. Utilization of spatial sound is potentially the next step to improve the audio experiences for listeners in terms of their emotional engagement. Hence, the relationship between spatial audio characteristics and emotional responses of the listeners has been the main focus of several recent studies. This paper provides a systematic overview of the above reports, including the analysis of commonly utilized methodology and technology. The survey was undertaken using four literature repositories, namely, Google Scholar, Scopus, IEEE Xplore, and AES E-Library. The overviewed papers were selected according to the empirical validity and quality of the reported studies. According to the survey outcomes, there is growing evidence of a positive influence of the selected spatial audio characteristics on the listeners’ affective responses. However, more data is required to build reliable, universal, and useful models explaining the above relationship. Furthermore, the two research trends on this topic were identified. Namely, the studies undertaken so far can be classified as either technology-oriented or technology-agnostic, depending on the research questions or experimental factors examined. Prospective future research directions regarding this topic are identified and discussed. They include better utilization of scene-based paradigms, affective computing techniques, and exploring the emotional effects of dynamic changes in spatial audio scenes.

Słowa kluczowe

EN

spatial sound; emotions; affective responses

• Czasopismo: International Journal of Electronics and Telecommunications
• Rocznik: 2024
• Tom: Vol. 70, No. 2
• Strony: 293--300
• Opis fizyczny: Bibliogr. 57 poz., rys.

Twórcy

autor

Stefanowska, Antonina

• Faculty of Computer Science, Białystok University of Technology, Poland,

autor

Zieliński, Sławomir K.

• Faculty of Computer Science, Białystok University of Technology, Poland,

Bibliografia

• [1] B. Wu, A. Horner, and C. Lee “The Correspondence of Music Emotion and Timbre in Sustained Musical Instrument Sounds,” Journal of the Audio Engineering Society, vol. 62, no. 10, pp. 663-675, 2014. https://doi.org/10.17743/jaes.2014.0037.
• [2] C. Chau, B. Wu, and A. Horner, “The Emotional Characteristics and Timbre of Nonsustaining Instrument Sounds,” Journal of the Audio Engineering Society, vol. 63, no. 4, pp. 228-244, 2015. https://doi.org/10.17743/jaes.2015.0016.
• [3] J. Guo, J. Liu, Z. Li, J. Zhu, and W. Jiang, “A Study on the Relationship Between Timbre Perception Features and Emotion in Musical Sounds,” in 2020 International Conference on Culture-oriented Science & Technology (ICCST), Beijing, pp. 22-27, 2020. https://doi.org/10.1109/ICCST50977.2020.00010.
• [4] C. Chau, R. Mo, and A. Horner, “The Emotional Characteristics of Piano Sounds with Different Pitch and Dynamics,” Journal of Audio Engineering Society, vol. 64, no. 11, pp. 918-932, 2016. https://doi.org/10.17743/jaes.2016.0049.
• [5] C. Chau, S. J. M. Gilburt, R. Mo, and A. Horner, “The Emotional Characteristics of Bowed String Instruments with Different Pitch and Dynamics,” Journal of Audio Engineering Society, vol. 65, no. 7/8, pp. 573-588, 2017. https://doi.org/10.17743/jaes.2017.0020.
• [6] R. Mo, B. Wu, and A. Horner, “The Effects of Reverberation on the Emotional Characteristics of Musical Instruments,” Journal of Audio Engineering Society, vol. 63, no. 12, pp. 966-979, 2015. https://doi.org/10.17743/jaes.2015.0082.
• [7] R. Mo, R. H. Y. So, and A. Horner, “An Investigation into How Reverberation Effects the Space of Instrument Emotional Characteristics,” Journal of Audio Engineering Society, vol. 64, no. 12, pp. 988-1002, 2016. https://doi.org/10.17743/jaes.2016.0054.
• [8] R. Mo, G. L. Choi, C. Lee, and A. Horner, “The Effects of MP3 Compression on Perceived Emotional Characteristics in Musical Instruments,” Journal of Audio Engineering Society, vol. 64, no. 11, pp. 858-867, 2016. https://doi.org/10.17743/jaes.2016.0031.
• [9] Y. Hong, C. Chau, and A. Horner, “An Analysis of Low-Arousal Piano Music Ratings to Uncover What Makes Calm and Sad Music So Difficult to Distinguish in Music Emotion Recognition,” Journal of Audio Engineering Society, vol. 65, no. 4, pp. 304-320, 2017. https://doi.org/10.17743/jaes.2017.0001.
• [10] W. L. Sin, X. Ma, and A. Horner, “The emotional characteristics of rain sound effects,” in 2018 International Computer Music Conference (ICMC), Daegu, pp. 344-349, 2018. https://hdl.handle.net/1783.1/95682.
• [11] W. L. Sin, B. Y. Chang, X. Ma, and A. Horner, “The Acoustic Features and Their Relationship to the Emotional Characteristics of Rain Sound Effects,” in 45th International Computer Music Conference (ICMC) and International Computer Music Conference New York City Electroacoustic Music Festival (NYCEMF), New York, pp. 84-89, 2019. https://hdl.handle.net/1783.1/98625.
• [12] C. Gafni and R. Tsur, “Some experimental evidence for sound-emotion interaction,” Scientific Study of Literature, vol. 9, no. 1, pp. 53-71, 2019. https://doi.org/10.1075/ssol.19002.gaf.
• [13] T. A. Alam and N. Dibben, “A Comparison of Presence and Emotion Between Immersive Virtual Reality and Desktop Displays for Musical Multimedia,” in Future Directions of Music Cognition, Virtual, pp. 97-101, 2021. https://doi.org/10.18061/FDMC.2021.0017.
• [14] A. Algargoosh, B. Soleimani, S. O’Modhrain, and M. Navvab, “The impact of the acoustic environment on human emotion and experience: A case study of worship spaces,” Building Acoustics, vol. 29, no. 1, pp. 85-106, 2021. https://doi.org/10.1177/1351010X211068850.
• [15] F. Cuadrado, I. Lopez-Cobo, T. Mateos-Blanco, and A. Tajadura-Jiménez, “Arousing the Sound: A Field Study on the Emotional Impact on Children of Arousing Sound Design and 3D Audio Spatialization in an Audio Story,” Frontiers in Psychology, vol. 11, pp. 737, 2020. https://doi.org/10.3389/fpsyg.2020.00737.
• [16] K. Drossos, A. Floros, A. Giannakoulopoulos, and N. Kanellopoulos, “Investigating the Impact of Sound Angular Position on the Listener Affective State,” IEEE Transactions on Affective Computing, vol. 6, no. 1, pp. 27-42, 2015. https://doi.org/10.1109/TAFFC.2015.2392768.
• [17] I. Ekman, and R. Kajastila, “Localization Cues Affect Emotional Judgments - Results from a User Study on Scary Sound,” in 35th Audio Engineering Society International Conference: Audio for Games, London, pp. 166-171, 2009. http://www.aes.org/e-lib/browse.cfm?elib=15177.
• [18] E. D. Filippi, T. Schmele, A. Nandi, A. G. Torres, and A. Pereda-Baños, “Emotional Impact of Source Localization in Music Using Machine Learning and EEG: a proof-of-concept study,” TechRxiv, 2022. https://doi.org/10.36227/techrxiv.21789866.v1.
• [19] J. Gong, Y. Shi, J. Wang, D. Shi, and Y. Xu, “Escape from the Dark Jungle: A 3D Audio Game for Emotion Regulation,” in Virtual, Augmented and Mixed Reality: Applications in Health, Cultural Heritage, and Industry (VAMR 2018), Las Vegas, pp. 57-76, 2018, https://doi.org/10.1007/978-3-319-91584-5_5.
• [20] E. Hahn, “Musical Emotions Evoked by 3D Audio,” in Audio Engineering Society International Conference on Spatial Reproduction - Aesthetics and Science, Tokyo, 2018. http://www.aes.org/e-lib/browse.cfm?elib=19640.
• [21] Y. Hyodo, C. Sugai, J. Suzuki, M. Takahashi, M. Koizumi, A. Tomura, Y. Mitsufuji, and Y. Komoriya, “Psychophysiological Effect of Immersive Spatial Audio Experience Enhanced Using Sound Field Synthesis,” in 9th International Conference on Affective Computing and Intelligent Interaction (ACII), Nara, 2021. https://doi.org/10.1109/ACII52823.2021.9597435.
• [22] G. Kailas and N. Tiwari, “An Empirical Measurement Tool for Overall Listening Experience of Immersive Audio,” in IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, 2021. https://doi.org/10.1109/ICCE50685.2021.9427770.
• [23] P. Larsson and D. Västfjäll, “Emotional and behavioural responses to auditory interfaces in commercial vehicles,” International Journal of Vehicle Noise and Vibration, vol. 9, no. 1-2, pp. 75-95, 2013. https://doi.org/10.1504/IJVNV.2013.053818
• [24] P. Larsson, A. Opperud, K. Fredriksson, and D. Västfjäll, “Emotional and Behavioral Response to Auditory Icons and Earcons in Driver-Vehicle Interfaces,” in 21st International Technical Conference on the Enhanced Safety of Vehicles (ESV), Stuttgart, 2009.
• [25] S. Lepa, S. Weinzierl, H.-J. Maempel, and E. Ungeheuer, “Emotional Impact of Different Forms of Spatialization in Everyday Mediatized Music Listening: Placebo or Technology Effects?” in 136th Audio Engineering Society Convention, Berlin, 2014. http://www.aes.org/e-lib/browse.cfm?elib=17171.
• [26] J. Li, L. Maffei, A. Pascale, and M. Masullo, “Effects of the Spatialisation of Water-Sounds Sequences on the Perception of Traffic Noise,” Vibrations in Physical Systems, vol. 33, no. 1, 2022. https://doi.org/10.21008/j.0860-6897.2022.1.10.
• [27] G. Moiragias and J. Mourjopoulos, “A listener preference model for spatial sound reproduction, incorporating affective response,” PLoS One, vol. 18, no. 6, 2023. https://doi.org/10.1371/journal.pone.0285135.
• [28] S. Oode and A. Ando, “Estimation of Kandoh Degree with Emphasis on Spatial Sound Impressions,” in 13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, Kyoto, 2012. https://doi.org/10.1109/SNPD.2012.98.
• [29] Y. Ooishi, M. Kobayashi, M. Kashino, and K. Ueno, “Presence of Three-Dimensional Sound Field Facilitates Listeners’ Mood, Felt Emotion, and Respiration Rate When Listening to Music,” Frontiers in Psychology, vol. 12, pp. 650777, 2021. https://doi.org/10.3389/fpsyg.2021.650777.
• [30] J. Pätynen, and T. Lokki, “Concert halls with strong and lateral sound increase the emotional impact of orchestra music,” The Journal of the Acoustical Society of America, vol. 139, no. 3, pp. 1214-24, 2016. https://doi.org/10.1121/1.4944038.
• [31] A. P. Pinheiro, D. Lima, P. B. Albuquerque, A. Anikin, and C. F. Lima, “Spatial location and emotion modulate voice perception,” Cognition and Emotion, vol. 33, no. 8, pp. 1577-1586, 2019. https://doi.org/10.1080/02699931.2019.1586647
• [32] J. Ramalho and T. Chambel, “Immersive 360º Mobile Video with an Emotional Perspective,” in ACM International workshop on Immersive media experiences (ImmersiveMe ’13), Barcelona, pp. 35-40, 2013. https://doi.org/10.1145/2512142.2512144
• [33] M. Shin, S. W. Song, S. J. Kim, and F. Biocca, “The effects of 3D sound in a 360-degree live concert video on social presence, parasocial interaction, enjoyment, and intent of financial supportive action,” International Journal of Human-Computer Studies, vol. 126, pp. 81-93, 2019. https://doi.org/10.1016/j.ijhcs.2019.02.001.
• [34] A. Tajadura-Jiménez, P. Larsson, A. Väljamäe, D. Västfjäll, and M. Kleiner, “When room size matters: acoustic influences on emotional responses to sounds,” Emotion, vol. 10, no. 3, pp. 416-422, 2010. https://doi.org/10.1037/a0018423.
• [35] D. Västfjäll, “The Subjective Sense of Presence, Emotion Recognition, and Experienced Emotions in Auditory Virtual Environments,” CyberPsychology & Behavior, vol. 6, no. 2, pp. 181-188, 2003. https://doi.org/10.1089/109493103321640374.
• [36] R. Warp, M. Zhu, I. Kiprijanovska, J. Wiesler, S. Stafford, and I. Mavridou, “Moved By Sound: How head-tracked spatial audio affects autonomic emotional state and immersion-driven auditory orienting response in VR Environments,” in 152nd Audio Engineering Society Convention, The Hague, 2022. http://www.aes.org/e-lib/browse.cfm?elib=21703.
• [37] R. Warp, M. Zhu, I. Kiprijanovska, J. Wiesler, S. Stafford, and I. Mavridou, “Validating the effects of immersion and spatial audio using novel continuous biometric sensor measures for Virtual Reality,” in IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Singapore, 2022. https://doi.org/10.1109/ISMAR-Adjunct57072.2022.00058.
• [38] Y. Wycisk, K. Sander, R. Kopiez, F. Platz, S. Preihs, and J. Peissig, “Development of the Immersive Music Experience Inventory (IMEI),” Frontiers in Psychology, vol. 13, pp. 951161, 2022. https://doi.org/10.3389/fpsyg.2022.951161.
• [39] D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, The PRISMA Group, “Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement,” PLoS Medicine, vol. 6, no. 7, pp. e1000097, 2009. https://doi.org/10.1371/journal.pmed.1000097.
• [40] ITU-R. BS. 775-4 Recommendation. “Multichannel stereophonic sound system with and without accompanying picture,” International Telecommunication Union, Geneva, 2022.
• [41] F. Rumsey, “Spatial Audio,” 1st Edition, Routledge, London, 2001.
• [42] J. Blauert ed., “The Technology of Binaural Listening,” Springer, Berlin, 2013. https://doi.org/10.1007/978-3-642-37762-4.
• [43] J. Paterson and H. Lee ed., “3D Audio,” Routledge, New York, 2021.
• [44] J. A. Russell, “Affective space is bipolar,” Journal of Personality and Social Psychology, vol. 37, no. 3, pp. 345-356, 1979. https://doi.org/10.1037/0022-3514.37.3.345.
• [45] A. Mehrabian, “Relations among personality scales of aggression, violence, and empathy: Validational evidence bearing on the Risk of Eruptive Violence Scale,” Aggressive Behavior, vol. 23, no. 6, pp. 433-445, 1997. https://doi.org/10.1002/(SICI)1098-2337(1997)23:6<433::AID-AB3>3.0.CO;2-H.
• [46] P. Ekman, “An argument for basic emotions,” Cognition and Emotion, vol. 6, no. 3-4, pp. 169-200, (1992). https://doi.org/10.1080/02699939208411068.
• [47] R. Plutchik, “A psychoevolutionary theory of emotions,” Social Science Information, vol. 21, no. 4-5, pp. 529-553, (1982). https://doi.org/10.1177/053901882021004003.
• [48] F. Rumsey, “Spatial Quality Evaluation for Reproduced Sound: Terminology, Meaning, and a Scene-Based Paradigm,” Journal of Audio Engineering Society, vol. 50, no. 9, pp. 651-666, 2002. https://www.aes.org/e-lib/browse.cfm?elib=11067.
• [49] S. K. Zieliński, F. Rumsey, and S. Bech, “Effects of Down-Mix Algorithms on Quality of Surround Sound,” Journal of Audio Engineering Society, vol. 51, no. 9, pp. 780-798, 2003. http://www.aes.org/e-lib/browse.cfm?elib=12208.
• [50] S. K. Zieliński, F. Rumsey, and S. Bech, “Effects of Bandwidth Limitation on Audio Quality in Consumer Multichannel Audiovisual Delivery Systems,” Journal of Audio Engineering Society, vol. 51, no. 6, pp. 475. 501, 2003. http://www.aes.org/e-lib/browse.cfm?elib=12222.
• [51] M. M. Bradley, and P. J. Lang, “Measuring emotion: The self-assessment manikin and the semantic differential,” Journal of Behavior Therapy and Experimental Psychiatry, vol. 25, no. 1, pp. 49-59, 1994. https://doi.org/10.1016/0005-7916(94)90063-9.
• [52] L. F. Barrett and J. A. Russell, “Independence and Bipolarity in the Structure of Current Affect,” Journal of Personality and Social Psychology, vol. 74, no. 4, pp. 967-984, 1998. https://doi.org/10.1037/0022-3514.74.4.967.
• [53] A. Betella and P. F. M. J. Verschure, “The Affective Slider: A Digital Self-Assessment Scale for the Measurement of Human Emotions,” PLoS One, vol 11, no. 2, pp. e0148037, 2016. https://doi.org/10.1371/journal.pone.0148037.
• [54] Y. Wang, W. Song, W. Tao, A. Liotta, D. Yang, X. Li, S. Gao, Y. Sun, W. Ge, W. Zhang, and W. Zhang, “A systemic review on affective computing: emotion models, databases, and recent advances”, Information Fusion, vol. 83-84, pp. 19-52, 2022. https://doi.org/10.1016/j.inffus.2022.03.009.
• [55] F. L. Wightman & D. J. Kistler, “Resolution of front-back ambiguity in spatial hearing by listener and source movement”, The Journal of the Acoustical Society of America, vol. 105, no. 5, pp. 2841-2853, 1999. https://doi.org/10.1121/1.426899.
• [56] A. J. Berkhout, D. de Vries, and P. Vogel, “Acoustic control by wave field synthesis,” The Journal of the Acoustical Society of America, vol. 93, no. 5, pp. 2764-2778, 1993. https://doi.org/10.1121/1.405852.
• [57] Y. Mitsufuji, A. Tomura, and K. Ohkuri, “Creating a highly-realistic “Acoustic Vessel Odyssey” using Sound Field Synthesis with 576 Loudspeakers,” in Audio Engineering Society International Conference on Spatial Reproduction - Aesthetics and Science, Tokyo, 2018. http://www.aes.org/e-lib/browse.cfm?elib=19648.

Identyfikatory

DOI

10.24425/ijet.2024.149544