Methods, Tools and Techniques for Multimodal Analysis of Accommodation in Intercultural Communication

• Autorzy: Karpiński M. , Klessa K.

Identyfikatory

DOI

10.12921/cmst.2018.0000006

• Języki publikacji: EN

Abstrakty

EN

The holistic approach to interpersonal communication in dialogue, involving the analysis of multiple sensory modalities and channels, poses a serious challenge not only in terms of research techniques and methods but also from the viewpoint of data infrastructure. In the Borderland project, the process of communicative accommodation is studied in young people in the intercultural context of the Polish-German boundary region. In order to collect, annotate, and analyse research material, a new technical and analytic infrastructure has been developed. Centred around a data management system, it incorporates well-known annotation and transcription tools along with custom-designed transcription and annotation tagsets and procedures. Data and metadata formats are designed to enable further corpus-based analyses of accommodation-related processes, mostly in the paralinguistic domain of prosody and gestures. Data formats used in the project ensure wide interchangeability and usage of almost any analytic software. Previously tested methods of quantitative accommodation analysis are adjusted, supplemented with new custom procedures, and applied to each channel under study as well as to the cross-modal (e.g., prosody - gesture) accommodation processes.

Słowa kluczowe

EN

communicative accommodation; intercultural communication; multimodal linguistic resources; multimodal data; metadata management

• Wydawca: Institute of Bioorganic Chemistry Scientific Publishers OWN, Polish Academy of Sciences
• Czasopismo: Computational Methods in Science and Technology
• Rocznik: 2018
• Tom: Vol. 24, No. 1
• Strony: 29--41
• Opis fizyczny: Bibliogr. 88 poz., rys.

Twórcy

autor

Karpiński M.

autor

Klessa K.

Bibliografia

• [1] M. Karpiński, New challenges in psycholinguistics: Interactivity and alignment in interpersonal communication, Lingua Posnaniensis, vol. 54, issue 1, pp. 97-106 (2014).
• [2] S.E. Brennan, H.H. Clark, Conceptual pacts and lexical choice in conversation, Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, pp. 1482–93 (1996).
• [3] H.P. Branigan, M.P. Pickering, A. Cleland, Syntactic coordination in dialogue, Cognition, 75, pp. 13-25 (2000).
• [4] R.L. Street Speech convergence and evaluation in fact-finding interviews, Human Communication Research, 11(2), pp. 139–169 (1984).
• [5] S. Kousidis, A Study of Accomodation of Prosodic and Temporal Features in Spoken Dialogues in View of Speech Technology Applications, Doctoral Thesis. Dublin Institute of Technology (2010).
• [6] M. Karpiński, K. Klessa, A. Czoska, Local and global convergence in the temporal domain in Polish task-oriented dialogue, Proceedings of the 7th Speech Prosody Conference, 20-23 May 2014, Dublin, Ireland. ISSN: 2333-2042, pp. 743-747 (2014).
• [7] F. Hahn, H. Rieser, Explaining Speech Gesture Alignment in MM Dialogue Using Gesture Typology, [In:] P.Lupowski, M. Purver (Eds.), Aspects of Semantics and Pragmatics of Dialogue. SemDial 2010, pp. 99–111 (2010).
• [8] K. Bergmann, S. Kopp, Gestural alignment in natural dialogue, [In:] R.P. Cooper, D. Peebles, N. Miyake, (Eds.) Proceedings of the 34th Annual Conference of the Cognitive Science Society (CogSci 2012), pp. 1326–1331. Austin, T X: Cognitive Science Society (2012).
• [9] T.L. Chartrand, J.A. Bargh The chameleon effect: The perception-behavior link and social interaction, J. Pers. Soc. Psychol. 76, pp. 893–910, (1999).
• [10] B. Oben, G. Brône, Explaining interactive alignment: A multimodal and multifactorial account, Journal of Pragmatics 104, pp. 32-51 (2016).
• [11] A. Czoska, K. Klessa, M. Karpiński, E. Nowikow-Jarmołowicz, Prosody and gesture in dialogue: Cross-modal interactions, Proceedings of 4th Gesture and Speech in Interaction (GESPIN) Conference, Nantes, France, pp. 83-88 (2015).
• [12] D.A. Cai, J.I. Rodriguez Adjusting to Cultural Differences: The Intercultural Adaptation Model. Intercultural Communication Studies VI: 2 (1996).
• [13] R. Levitan, J. Hirschberg, Measuring acoustic-prosodic entrainment with respect to multiple levels and dimensions, Proceedings of Interspeech 2011, Florence, Italy, August 2011, pp. 3081–3084 (2011).
• [14] R.M. Krauss, J.S. Pardo, Is alignment always the result of automatic priming?. Behavioral and Brain Sciences, 27(02), 203-204 (2004).
• [15] D.G. Hewett, B.M.Watson, C. Gallois, M.Ward, B.A. Leggett, Intergroup communication between hospital doctors: implications for quality of patient care. Social science medicine, 69(12), pp. 1732-1740 (2009).
• [16] R. Porzel, A. Scheffler, R. Malaka, How entrainment increases dialogical efficiency, Proceedings of Workshop on Effective Multimodal Dialogue Interfaces, Sydney 2006.
• [17] M.J. Pickering, S. Garrod, Toward a mechanistic psychology of dialogue, Behavioral and Brain Sciences 27, pp. 169–226 (2004).
• [18] D. Reitter, J.D. Moore, Predicting success in dialogue, Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 808–815 (2007).
• [19] F. Ramseyer, W. Tschaecher, Nonverbal synchrony or random coincidence? How to tell the difference, [In:] A. Esposito, ed., COST 2102 International Training School, Heidelberg: Springer Verlag, 2009, pp. 182–196 (2009).
• [20] H. Giles, D.M. Taylor, R.Y. Bourhis, Towards a theory of interpersonal accommodation through language: Some Canadian data, Language in Society 2, pp. 177–192 (1973).
• [21] H. Giles, P. Smith, Accommodation theory: Optimal levels of convergence, [In:] H. Giles, R.N. St. Clair, eds., Language and Social Psychology. Baltimore: University Park Press, pp. 45-65 (1979).
• [22] H. Giles, N. Coupland, J. Coupland, Accommodation theory: Communication, context, and consequence. Contexts of accommodation, Developments in applied sociolinguistics, pp. -68 (1991).
• [23] E.A. Schegloff, Discourse as an interactional achievement: Some uses of ‘uh huh’ and other things that come between sentences, [In:] D. Tannen (Ed.) Analyzing Discourse: Text and Talk, pp. 71–93. Georgetown: Georgetown University Press (1982).
• [24] H.H Clark, Using language. Cambridge: CUP (1996).
• [25] S. Garrod, A. Anderson, Saying what you mean in dialogue: A study in conceptual and semantic co-ordination, Cognition 27(2), 1987, pp. 181-218 (1987).
• [26] F.J. Bernieri, R. Rosenthal, Interpersonal coordination: Behaviour matching and interactional synchrony, [In:] R. Feldman, B. Rimé, eds., Fundamentals of non-verbal behaviour, pp. 401–432. New York: Cambridge University Press (1991).
• [27] M.J. Pickering, S. Garrod, Alignment as the basis for successful communication, Research on Language, Computation 4(2), 203-228 (2006).
• [28] M. Kaschak, A. Glenberg, Interactive alignment: Priming or memory retrieval?, Behavioral and Brain Sciences 27(02), 201-202 (2004).
• [29] N. Ide, J. Pustejovsky, What does interoperability mean, anyway? Toward an operational definition of interoperability for language technology. Proceedings of the Second International Conference on Global Interoperability for Language Resources. Hong Kong, China (2010).
• [30] S. Bonacchi, M. Mela, Practical remarks about the interoperability of the computer programmes Folker, ELAN and Praat for transcription and multimodal linguistic annotation from the user’s point of view, Journal of Multimodal Communication Studies 2/2014, pp. 18-29 (2014).
• [31] P. Caponetto, Red Hen Lab: A further step towards multimodality [report]. Journal of Multimodal Communication Studies 3(1-2) (2016).
• [32] D.A. López, MOCA-A (Multimodal Oral Corpora Administration), on-line: http://www.hpsl.uni-freiburg.de/
• [33] K. Peshkov, L. Prévot, Segmentation evaluation metrics, a comparison grounded on prosodic and discourse units, [In:] Proceedings of the 9th Language Resources and Evaluation Conference, Reykyavik, Iceland, pp. 321-325 (2014).
• [34] J. Cohen, A coefficient of agreement for nominal scales, Educational and psychological measurement 20(1), pp. 37-46 (1960).
• [35] P.H.Wittenburg, A. Brugman, A. Russel, Klassmann, H. Sloetjes, ELAN: a professional framework for multimodality research. Proceedings of the 5th Language Resources and Evaluation Conference, Genoa, Italy, pp. 1556-1559 (2006).
• [36] K. Klessa, Karpiński, M., A. Wagner, Annotation Pro – a new software tool for annotation of linguistic and paralinguistic features, [In:] B. Bigi, D. Hirst, eds.. Proceedings of TRASP (Tools and Resources for the Analysis of Speech Prosody), Aix-en-Provence, s. 51-54. Aix-Marseille Université. ISBN 978-2-7466-6443-2 (2013).
• [37] B. Bigi, SPPAS-Multi-lingual approaches to the automatic annotation of speech, [In:] K. Klessa, B. Bigi, eds., The Phonetician - International Society of Phonetic Sciences, Vol. 111-112, pp. 55-69 (2015).
• [38] M. Karpiński, J. Klesta, E. Baranowska, K. Francuzik (Klessa), Interphrase Pause Realization Rules for the Purpose of High Quality Polish Speech Synthesis, Speech Analysis,Synthesis and Recognition (SASR), Szczyrk, AGH Kraków,pp. 85-89 (2005).
• [39] A. Kendon, Some relationships between body motion and speech, Studies in dyadic communication 7(177), p. 90 (1972).
• [40] S. Kita, The temporal relationship between gesture and speech: A study of Japanese-English bilinguals. MS, Department of Psychology, University of Chicago, 90, 91-94 (1990).
• [41] D. Mc Neill, Gesture and Thought. Chicago: University of Chicago Press 2007.
• [42] J. Bressem, Transcription systems for gestures, speech, prosody, postures, gaze, [In:] C. Müller, A. Cienki, E. Fricke, S.H. Ladewig, D. Mc Neill, S. Teßendorf, eds., Body-Language-Communication: An international Handbook on Multimodality in Human interaction, Berlin, Boston: De Gruyter: Mouton (2013).
• [43] M. Karpiński, E. Jarmołowicz-Nowikow, A. Czoska, Gesture annotation scheme development and application for entrainment analysis in task-oriented dialogues in diverse cultures. Proceedings of GESPIN 2015 Conference, Nantes, France, pp. 161-166 (2015).
• [44] S.Z. Kousidis, P. Malisz,Wagner, D. Schlangen, Exploring annotation of head gesture forms in spontaneous human interaction, [In:] Proceedings of the Tilburg Gesture Meeting (TiGeR 2013) (2013).
• [45] Z. Malisz, M. Karpiński, Multimodal aspects of positive and negative responses in Polish task-oriented dialogues, [In:] Proceedings of Speech Prosody (2010).
• [46] G. Demenko, M. Wypych, E. Baranowska, Implementation of grapheme-to-phoneme rules and extended SAMPA alphabet in Polish text-to-speech synthesis, Speech and Language Technology 7(17) (2003).
• [47] M. Szymański, S. Grocholewski, Transcription-based automatic segmentation of speech, Proceedings of 2nd Language, Technology Conference, Poznań, pp. 11-15 (2005).
• [48] T. Kisler, U.D. Reichel, F. Schiel, C. Draxler, B. Jackl, N. Pörner, BAS Speech Science Web Services-an update of current developments. Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, Paper ID 668 (2016).
• [49] K. Klessa, Speech annotation mining with Annotation Pro plugins, Wydawnictwo Rys, Poznań. ISBN 978-83-65483-20-1 (2016).
• [50] S. Kousidis, A Study of Accomodation of Prosodic and Temporal Features in Spoken Dialogues in View of Speech Technology pplications, Doctoral Thesis. Dublin Institute of Technology (2010).
• [51] S. Kousidis, D. Dorran, Y. Wang, B. Vaughan, C. Cullen, D. Campbell, C. McDonnell, E. Coyle, Towards measuring continuous acoustic feature convergence in unconstrained spoken dialogues, Proceedings of Interspeech 2008, pp. 1692–1695 (2008).
• [52] S.D. Farley, S.M. Hughes, J.N. LaFayette, People Will Know We Are in Love: Evidence of Differences Between Vocal Samples Directed Toward Lovers and Friends, Journal of Nonverbal Behavior, Ausgabe 3/2013 (2013).
• [53] A. Cutler, Linguistic rhythm and speech segmentation, [In:] J. Sundberg, L. Nord, R. Carlson, eds., Music, language, speech and brain, Macmillan Education UK, pp. 157-166 (1991).
• [54] K.L. Pike, The Intonation of American English, University of Michigan Press, Ann Arbor (1945).
• [55] W. Jassem, D.R. Hill, I.H. Witten, Isochrony in English speech: its statistical validity and linguistic relevance, [In:] D. Gibbon, H. Richter, eds., Intonation, accent and rhythm. Studies in Discourse Phonology 8, pp. 203–225 (1984).
• [56] F. Ramus, M. Nespor, J. Mehler, Correlates of linguistic rhythm in the speech signal, Cognition 73(3), 265-292 (1999).
• [57] P.A. Barbosa, Explaining cross-linguistic rhythmic variability via a coupled-oscillator model of rhythm production, Speech Prosody 2002, International Conference (2002).
• [58] E. Grabe, E.L. Low, Durational variability in speech and the rhythm class hypothesis, Papers in laboratory phonology 7, pp. 515-546 (2002).
• [59] V. A. Dellwo, Fourcin, E. Abberton, Rhythmical classification of languages based on voice parameters. Proceedings of the XVIth ICPhS, Saarbrücken, pp. 1129-1132 (2007).
• [60] P. Wagner, The rhythm of language and speech: Constraining factors, models, metrics and applications, Germany: Habilitationsschrift, University of Bonn (2008).
• [61] V. Dellwo, Rhythm and speech rate: A variation coefficient for ΔC. Language and language-processing, pp. 231-241 (2006).
• [62] B. Vaughan, Prosodic synchrony in co-operative task-based dialogues: A measure of agreement and disagreement, Proceedings of Interspeech 2011, Florence, Italy, August 2011, pp. 1865–1867 (2011).
• [63] K. Klessa, D. Gibbon, Annotation Pro+ TGA: automation of speech timing analysis. Annotation Pro + TGA: automation of speech timing analysis, Proceedings of the 9th Language Resources and Evaluation Conference, Reykjavik, Iceland, pp. 1499-1505 (2014).
• [64] D. Gibbon, TGA: a web tool for Time Group Analysis. Proceedings of the Tools and Resources for the Analysis of Speech Prosody (TRASP)Workshop, Aix-en-Provence, France, pp. 66-69 (2013).
• [65] P. Wagner, Visualizing levels of rhythmic organization, [In:] XVIth International Congress of the Phonetic Sciences, Saarbrücken, 6-10 August 2007 (2007).
• [66] C. Gussenhoven, Phonology of Tone and Intonation, Cambridge: CUP 2004.
• [67] A. Chen, C. Gussenhoven, T. Rietveld, Language-Specificity in the Perception of Paralinguistic Intonational Meaning. Language and Speech 47(4), pp. 311–349 (2004).
• [68] A. Chen, Universal and Language-specic Perception of Paralinguistic Intonational Meaning, PhD Thesis, Radboud Universiteit Nijmegen (2005).
• [69] M. Karpinski, The Boundaries of Language: Dealing with Paralinguistic Features, Lingua Posnaniensis, vol. LIV (2)/2012.The Poznań Society for the Advancement of the Arts and Sciences, pp. 37-54. PL ISSN 0079-4740, ISBN 978-83-7654-252-2 (2012).
• [70] L. Rabiner, On the use of autocorrelation analysis for pitch detection, IEEE transactions on acoustics, speech, and signal processing, 25(1), pp. 24-33 (1977).
• [71] L. Rabiner, M. Cheng, A. Rosenberg, C. McGonegal, A comparative performance study of several pitch detection algorithms. IEEE Transactions on Acoustics, Speech, and Signal Processing, 24(5), pp. 399-418 (1976).
• [72] W. Hess, Pitch determination of speech signals: algorithms and devices. Berlin – Heidelberg – New York: Springer-Verlag, 1983.
• [73] P. Mertens, The Prosogram: Semi-automatic transcription of prosody based on a tonal perception model. In Speech Prosody 2004, Nara, Japan, pp. 549-552 (2004).
• [74] P. Mertens, Polytonia: a system for the automatic transcription of tonal aspects in speech corpora, Journal of Speech Sciences, 4(2), pp. 17-57 (2014).
• [75] A. Kendon, Do gestures communicate? A review. Research on Language and Social Interaction 27(3), pp. 175-200 (1994).
• [76] McD. Neill, Hand and Mind. What gestures reveal about thought, The University of Chicago Press (1995).
• [77] A. Kendon, Gesture. Visible Action as Utterance, Cambridge: CUP 2005.
• [78] R.M. Krauss, R.A. Dushay, Y. Chen, F. Rauscher, The Communicative value of conversational hand gestures, Journal of Experimental Social Psychology, 31, pp. 533-552 (1995).
• [79] R.M. Krauss, The role of speech-related arm/hand gestures in word retrival, [In:] R. Campbell, L. Messing, eds., Gesture, Speech and Sign, 93-116, Oxford University Press (2001).
• [80] J. Bavelas, J. Gerwing, Ch. Sutton, D. Prevost, Gesturing on the telephone: Independent effects of dialogue and visibility, Journal of Memory and Language, 5(2), pp. 495-520 (2007).
• [81] Jarmołowicz-E. Nowikow, Karpiński, M. Communicative intentions behind pointing gestures in task-oriented dialogues. [in:] P. Z. Wagner, C. Malisz, Kirchhof (Eds.) Proceedings of GESPIN 2011: Gesture and Speech in Interaction Conference (2011).
• [82] McD. Neill, Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press (1992).
• [83] J. Gumperz, Contextualization and Understanding, [In:] A. Duranti, Ch. Goodwin, eds. Rethinking Context: Language as an Interactive Phenomenon.Cambridge: CUP, pp. 229-252 (1992).
• [84] M. Silverstein, Shifters, Linguistic Categories, and Cultural Description, [In:] K. Basso, H. Selby, eds. Meaning in Anthropology. Albuquerque: University of New Mexico Press (1976).
• [85] D. Figiel, Lexical alignment in task-oriented dialogue. Unpublished A thesis supervised by M. Karpiński, Institute of Linguistics, AMU in Poznań (2017).
• [86] F. Cummins, R.F. Port, Rhythmic commonalities between hand gestures and speech, [In:] Proceedings of the eighteenth meeting of the Cognitive Science Society, pp. 415-419 (1996).
• [87] A.F. Hayes, Introduction to mediation, moderation, and conditional process analysis: A regression-based approach, Guilford Press (2013).
• [88] K. Bergmann, H.P. Branigan, S. Kopp, Exploring the alignment space – lexical and gestural alignment with real and virtual humans, Frontiers in ICT (2015).

Uwagi

PL

Opracowanie rekordu w ramach umowy 509/P-DUN/2018 ze środków MNiSW przeznaczonych na działalność upowszechniającą naukę (2018).