Objectivization of Audio-Visual Correlation Analysis

• Autorzy: Kunka B. , Kostek B.
• Języki publikacji: EN

Abstrakty

EN

Simultaneous perception of audio and visual stimuli often causes concealment or misrepresentation of information actually contained in these stimuli. Such effects are called the "image proximity effect" or the "ventriloquism effect" in the literature. Until recently, most research carried out to understand their nature was based on subjective assessments. The authors of this paper propose a methodology based on both subjective and objectively retrieved data. In this methodology, objective data reflect the screen areas that attract most attention. The data were collected and processed by an eye-gaze tracking system. To support the proposed methodology, two series of experiments were conducted – one with a commercial eye-gaze tracking system Tobii T60, and another with the Cyber-Eye system developed at the Multimedia Systems Department of the Gdańsk University of Technology. In most cases, the visual-auditory stimuli were presented using a 3D video. It was found that the eye-gaze tracking system did objectivize the results of experiments. Moreover, the tests revealed a strong correlation between the localization of a visual stimulus on which a participant’s gaze focused and the value of the "image proximity effect". It was also proved that gaze tracking may be useful in experiments which aim at evaluation of the proximity effect when presented visual stimuli are stereoscopic.

Słowa kluczowe

EN

sound perception; sound source localization; virtual sound source shifting; bi-modal perception; cross-modal perception; image proximity effect; ventriloquism effect; visual attention; perceptual illusion; eye-gaze tracking

• Wydawca: Instytut Podstawowych Problemów Techniki PAN ; Komitet Akustyki PAN ; Polskie Towarzystwo Akustyczne
• Czasopismo: Archives of Acoustics
• Rocznik: 2012
• Tom: Vol. 37, No. 1
• Strony: 63--72
• Opis fizyczny: Bibliogr. 38 poz., fot., tab., wykr.

Twórcy

autor

Kunka B.

autor

Kostek B.

• Multimedia Systems Department, Gdańsk University of Technology Narutowicza 11/12, 80-233 Gdańsk, Poland,

Bibliografia

• 1. Abel A., Hussain A., Nguyen Q.-D., Ringeval F., Chetouani M., Milgram M. (2009), Maximising Audiovisual Correlation with Automatic Lip Tracking and Vowel Based Segmentation, Biometric ID Management and Multimodal Communication, Madrid, Spain, 16- 18.
• 2. Bech S., Hansen V., Woszczyk W. (1995), Interaction Between Audio-Visual Factors in a Home Theater System: Experimental Results, 99th Audio Eng. Soc. Conv., New York, Preprint No. 4096.
• 3. Beerends J.G., De Caluwe F.E. (1999), The Influence of Video Quality on Perceived Audio Quality and Vice Versa, Journal of the Audio Engineering Society, 47, 5, 355-362.
• 4. Beerends J.G., Stemerdink J.A. (1992), A perceptual audio quality measure based on a psychoacoustic sound representation, J. Audio Eng. Soc., 40, 12, 963-978.
• 5. Bertelson P. (1998), Starting from the ventriloquist: The perception of multimodal event, M. Sabourin, F.I.M. Craik, M. Robert [Eds.], Advances in psychological science, 1. Biological and cognitive aspects, Hove, U.K.: Psychology Press, 419-439.
• 6. Bertelson P., Radeau M. (1981), Cross-modal bias and perceptual fusion with auditory-visual spatial discordance, Perception and Psychophysics, 29, 6, 578-584.
• 7. Bogdanowicz M. (2000), Perceptual-motor integration, theory - diagnosis - therapy [in Polish], Methodological Centre for Psychological and Pedagogical, issue III, Warsaw.
• 8. Brook M., Danilenko L., Strasser W. (1984), Wie bewertet der Zuschauer das stereofone Fernsehes [in German], 13 Tonemeistertagung; Internationaler Kongres, 367-377.
• 9. Chen T., Rao R.R. (1998), Audio-visual integration in multimodal communication, Proceedings of the IEEE, 86, 5, 837-852.
• 10. Davis E.T., Scott K., Pair J., Hodges L.F., Oliverio J. (1999), Can audio enhance visual perception and performance in a virtual environment?, 43rd Human Factors and Ergonomics Society Annual Meeting, Houston.
• 11. Gardner M.B. (1968), Proximity image effect in sound localization, J. Acoust. Soc. Amer., 43, 163.
• 12. Hollier M.P., Voelcker R. (1997), Objective performance assessment: video quality as an influence on audio perception, 103rd Eng. Soc. Conv., New York, Preprint No. 4590.
• 13. ITU-R BS.1116-1 (1994-1997), Methods for the subjective assessment of small impairments in audio systems including multichannel sound systems.
• 14. ITU-R BS.1286 (1997), Methods for the subjective assessment of audio systems with accompanying picture.
• 15. Kin M.J., Plaskota P. (2011), Comparison of sound attributes of multichannel and mixed-down stereo recordings, Archives of Acoustics, 36, 2, 333-345.
• 16. Klonari D., Pastiadis K., Papadelis G., Papanikolaou G. (2011), Loudness assessment of musical tones equalized in A-weighted level, Archives of Acoustics, 36, 2, 239-250.
• 17. Komiyama S. (1989), Subjective evaluation of angular displacement between picture and sound directions for HDTV sound systems, J. Audio Eng. Soc., 37, 4, 210-214.
• 18. Kostek B. (2005), Perception-based data processing in acoustics. Applications to music information retrieval and psychophysiology of hearing, Springer Verlag, Berlin, 389-400.
• 19. Kostek B., Sankiewicz M. (2011), Retrospecting Polish Audio Engineering Society membership on 20th anniversary of the Polish Section of the Audio Engineering Society, Archives of Acoustics, 36, 2, 187-197.
• 20. Kunka B., Kostek B. (2009), A new method of audiovisual correlation analysis, Proc. of 2nd International Symposium on Multimedia - Applications and Processing MMAP'09, 4, 497-502, Mrągowo, Poland.
• 21. Kunka B., Kostek B. (2010a), Exploiting audiovisual correlation by means of gaze tracking, International Journal of Computer Science and Applications, 7, 3, 104-123.
• 22. Kunka B., Kostek B. (2010b), Objectivization of audio-video correlation assessment experiments, 128th Audio Engineering of Society Convention, Paper No. 8148, London.
• 23. Kunka B., Kostek B., Kulesza M., Szczuko P., Czyżewski A. (2010), Gaze-tracking based audiovisual correlation analysis employing quality of experience methodology, Intelligent Decision Technologies Journal, 4, 3, 217-227.
• 24. Lee J.S., De Simone F., Ebrahimi T. (2010), Efficient video coding based on audio-visual focus of attention, Journal of Visual Communication and Image Representation, Elsevier.
• 25. Lewald J. (1997), Eye-position effects in directional hearing, Behavioural Brain Research, 87, 35-48.
• 26. Lewald J., Ehrenstein W.H. (1998), Auditory visual spatial integration: a new psychophysical approach using laser pointing to acoustic targets, J. Acoust. Soc. Am., 104, 3, 1586-1597.
• 27. Liu Y., Sato Y. (2008), Recovering audio-to-video synchronization by audio-visual correlation analysis, 19th International Conference on Pattern Recognition (ICPR 2008), Tampa, Florida, USA.
• 28. McGurk H., MacDonald J. (1976), Hearing lips and seeing voices, Nature, 264, 746-748.
• 29. Mujal M., Kirlin R.L. (2002), Compression enhancement of video motion of mouth region using joint audio and video coding, 5th IEEE Southwest Symposium on Image Analysis and Interpretation.
• 30. Nakayama Y., Watanabe K., Komiyama S., Okano F., Izumi Y. (2003), A method of 3-D sound image localization using loudspeaker arrays, 114 Audio Eng. Soc. Convention, Paper No. 5793.
• 31. Rao R.R., Chen T. (1998), Exploiting audio-visual correlation in coding of talking head sequences, IEEE Trans. on Industrial Electronics, 45, 1, 15-22.
• 32. Rakowski A., Rogowski P. (2010), Pitch strength of residual sounds estimated through chroma recognition by absolute-pitch possessors, Archives of Acoustics, 35, 3, 331-347.
• 33. Rakowski A., Rogowski P. (2011), Absolute pitch and its frequency range, Archives of Acoustics, 36, 2, 251-266.
• 34. Rorden C., Driver J. (1999), Does auditory attention shift in the direction of an upcoming saccade?, Pergamon, Neuropsychologia, 37, 357-377.
• 35. Rutkowska L., Socha J. (2005), Statistical data analysis employing STATISTICA program [in Polish], Lecture Notes, Forestry Faculty, University of Agriculture, Cracow.
• 36. Sabin A.T., Rafii Z., Pardo B. (2011), Weighting Function-Based Rapid Mapping of Descriptors to Audio Processing Parameters, J. Audio Eng. Soc., 59, 6, 419-430.
• 37. Sitek A., Kostek B. (2011), Study of preference for surround microphone techniques used in the recording of choir and instrumental ensemble, Archives of Acoustics, 36, 2, 365-378.
• 38. Storms R.L., Zyda M.J. (2000), Interactions in perceived quality of auditory-visual displays, Presence:Teleoperators and Virtual Environment, 9, 6, 557-580