Aligning speech and co-speech gestu re in a constraint-based grammar

• Autorzy: Alahverdzhieva K. , Lascarides A. , Flickinger D.

Identyfikatory

DOI

10.15398/jlm.v5i3.167

• Języki publikacji: EN

Abstrakty

EN

This paper concerns the form-meaning mapping of communicative action consisting of speech and improvised co-speech gestures. Based on the findings of previous cognitive and computational approaches, we advance a new theory in which this form-meaning mapping is analysed in a constraint-based grammar. Motivated by observations in naturally occurring examples, we propose several construction rules, which use linguistic form, gesture form and their relative timing to constrain the derivation of a single speech-gesture syntax tree, from which a meaning representation can be composed via standard methods for semantic composition. The paper further reports on implementing these speech-gesture construction rules within the English Resource Grammar (Flickinger 2000). Since gestural form of ten underspecifies its meaning, the logical formulae that are composed via syntax are underspecified so that current models of the semantics/pragmatics interface support the range of possible interpretations of the speech-gesture act in its context of use.

Słowa kluczowe

EN

co-speech gesture; constraint-based grammar; compositional semantics; underspecification

• Wydawca: Instytut Podstaw Informatyki PAN
• Czasopismo: Journal of Language Modelling
• Rocznik: 2017
• Tom: Vol. 5, No. 3
• Strony: 421--464
• Opis fizyczny: Bibliogr. 59 poz., rys., tab.

Twórcy

autor

Alahverdzhieva K.

• School of Informatics, University of Edinburgh, UK

autor

Lascarides A.

• School of Informatics, University of Edinburgh, UK

autor

Flickinger D.

• Center for the Study of Language and Information, Stanford University, USA

Bibliografia

• [1] Dorit Abusch (2014), Temporal Succession and Aspectual Type in Visual Narrative, in Luka Crnič and Uli Sauerland, editors, The Art and Craft of Semantics: A Festschrift for Irene Heim, volume 1, pp. 9-29, MIT Working Papers in Linguistics, Cambride, MA.
• [2] Peter Adolphs, Stephan Oepen, Ulrich Callmeier, Berthold Crysmann, Daniel Flickinger, and Bernd Kiefer (2008), Some Fine Points of Hybrid Natural Language Parsing, in Proceedings of the Sixth International Language Resources and Evaluation, ELRA.
• [3] Stergos Afantenos, Eric Kow, Nicholas Asher, and Jeremy Perret (2015), Discourse parsing for multi-party chat dialogues, in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, pp. 928-937, Lisbon.
• [4] Hiyan Alshawi (1992), The Core Language Engine, Cambridge: MIT Press.
• [5] Nicholas Asher and Alex Lascarides (1998), Bridging, Journal of Semantics, 15 (1): 83-113.
• [6] Nicholas Asher and Alex Lascarides (2003), Logics of Conversation, Cambridge University Press.
• [7] Janet Beavin Bavelas and Nicole Chovil (2006), Hand gestures and facia displays as part of language use in face-to-face dialogue, in V. Manusov and M. Patterson, editors, Handbook of Nonverbal Communication, pp. 97-115, Thousand Oaks, CA: Sage.
• [8] Johan Bos (2004), Computational Semantics in Discourse: Underspecification, Resolution, and Inference, J. of Logic, Lang. and Inf., 13 (2): 139-157, ISSN 0925-8531, doi: 10.1023/B:JLLI.0000024731.26883.86, http://dx.doi.org/10.1023/B:JLLI.0000024731.26883.86.
• [9] Jean Carletta (2006), Announcing the AMI Meeting Corpus, The ELRA Newsletter, 11 (1): 3-5.
• [10] Jean Carletta (2007), Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus, Language Resources and Evaluation, 41 (2): 181-190.
• [11] Justine Cassell, David McNeill, and K. E. McCullough (1999), Speech-Gesture Mismatches: Evidence for One Underlying Representation of Linguistic and Non-Linguistic Information, Pragmatics and Cognition, 7 (1): 1-33.
• [12] Ann Copestake (2007), Semantic composition with (robust) minima recursion semantics, in DeepLP ’07: Proceedings of the Workshop on Deep Linguistic Processing, pp. 73-80, Association for Computational Linguistics, Morristown, NJ, USA.
• [13] Ann Copestake and Ted Briscoe (1995), Semi-Productive Polysemy and Sense Extension, Journal of Semantics, 12: 15-67.
• [14] Ann Copestake, Dan Flickinger, Ivan Sag, and Carl Pollard (2005), Minimal Recursion Semantics: An introduction, Journal of Research on Language and Computation, 3 (2-3): 281-332.
• [15] Ann Copestake, Alex Lascarides, and Dan Flickinger (2001), An Algebra for Semantic Construction in Constraint-based Grammars, in Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (ACL/EACL 2001), pp. 132-139, Toulouse.
• [16] Markus Egg, Alexander Koller, and Joachim Niehren (2001), The Constraint Language for Lambda Structures, Journal of Logic, Language and Information, 10: 457-485, ISSN 0925-8531, doi: 10.1023/A:1017964622902, http://portal.acm.org/citation.cfm?id=595849.596040.
• [17] Randi Engle (2000), Toward a Theory of Multimodal Communication: Combining Speech, Gestures, Diagrams and Demonstrations in Structural Explanations, Stanford University, PhD thesis.
• [18] Dan Flickinger (2000), On Building a More Efficient Grammar by Exploiting Types, Natural Language Engineering, 6 (1) (Special Issue on Efficient Processing with HPSG): 15-28.
• [19] Ellen Fricke (2008), Foundations of a Multimodal Grammar for German: Syntactic Structures and Functions (Grundlagen einer multimodalen Grammatik des Deutschen: Syntaktische Strukturen und Funktionen), Europa-Universität Viadrina Frankfurt (Oder), Habilitation, Manuskript. Original document in German.
• [20] Gianluca Giorgolo (2012), Integration of Gesture and Verbal Language: A Formal Semantics Approach, in Eleni Efthimiou, Georgios Kouroupetroglou, and Stavroula-Evita Fotinea, editors, Gesture and Sign Language in Human-Computer Interaction and Embodied Communication, volume 7206 of Lecture Notes in Computer Science, pp. 216-227, Springer Berlin Heidelberg, ISBN 978-3-642-34181-6, doi: 10.1007/978-3-642-34182-3_20, http://dx.doi.org/10.1007/978-3-642-34182-3_20.
• [21] Gianluca Giorgolo and Ash Asudeh (2011), Multimodal Communication in LFG: Gestures and the Correspondence Architecture , in Miriam Butt and Tracy Holloway King, editors, The Proceedings of the LFG 2011 Conference, pp. 257-277, Hong Kong, http://cslipublications.stanford.edu/LFG/16/abstracts/lfg11abs-giorgoloasudeh2.html.
• [22] Gianluca Giorgolo and Frans Verstraten (2008), Perception of speech-and-gesture integration, in Proceedings of the International Conference on Auditory-Visual Speech Processing 2008, pp. 31-36.
• [23] Erving Goffman (1963), Behavior in Public Places: Notes on the Social Organization of Gatherings, The Free Press.
• [24] Alex Grzankowski (2015), Pictures Have Propositional Content, Review of Philosophy and Psychology, 6 (1): 151-163, ISSN 1878-5158, doi: 10.1007/s13164-014-0217-0, http://dx.doi.org/10.1007/s13164-014-0217-0.
• [25] Florian Hahn and Hannes Rieser (2010), Explaining Speech Gesture Alignment in MM Dialogue Using Gesture Typology, in Paweł Łupkowski and Matthew Purver, editors, Aspects of Semantics and Pragmatics of Dialogue. SemDial 2010, 14th Workshop on the Semantics and Pragmatics of Dialogue, pp. 99-109, Polish Society for Cognitive Science, Poznań.
• [26] Jerry R Hobbs (1985), On the Coherence and Structure of Discourse, Technical report, Stanford University, Center for the Study of Language and Information.
• [27] Michael Johnston (1998a), Multimodal Language Processing, in Proceedings of the International Conference on Spoken Language Processing (ICSLP), Sydney, Australia.
• [28] Michael Johnston (1998b), Unification-based Multimodal Parsing, in Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics – Volume 1, ACL 1998, pp. 624-630, Association for Computational Linguistics, Stroudsburg, PA, USA, doi: http://dx.doi.org/10.3115/980845.980949, http://dx.doi.org/10.3115/980845.980949.
• [29] Michael Johnston, Philip R. Cohen, David McGee, Sharon L. Oviatt, James A. Pittman, and Ira Smith (1997), Unification-Based Multimodal Integration, in Philip R. Cohen and Wolfgang Wahlster, editors, Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics, pp. 281-288, Association for Computational Linguistics, Somerset, New Jersey.
• [30] David Kaplan (1989), Demonstratives, in J. Almog, J. Perry, and H. Wettstein, editors, Themes from Kaplan, Oxford.
• [31] Andrew Kehler (2002), Coherence, Reference, and the Theory of Grammar, CSLI] Publications.
• [32] Ruth Kempson, Wilfried Meyer-Viol, and Dov M Gabbay (2000), Dynamic syntax: The flow of language understanding, Wiley-Blackwell.
• [33] Adam Kendon (1972), Some relationships between body motion and speech, in A. Seigman and B. Pope, editors, Studies in Dyadic Communication, pp. 177-216, Pergamon Press, Elmsford, New York.
• [34] Adam Kendon (2004), Gesture. Visible Action as Utterance, Cambridge University Press, Cambridge.
• [35] Ewan Klein (2000), A constraint-based approach to English prosodic constituents, in ACL ’00: Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pp. 217-224, Association for Computational Linguistics, Morristown, NJ, USA, doi: http://dx.doi.org/10.3115/1075218.1075246.
• [36] Alexander Koller, Michaela Regneri, and Stefan Thater (2008), Regular tree grammars as a formalism for scope underspecification, in Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-08: HLT), Columbus, Ohio.
• [37] Stefan Kopp, Paul Tepper, and Justine Cassell (2004), Towards integrated microplanning of language and iconic gesture for multimodal output, in ICMI ’04: Proceedings of the 6th international conference on Multimodal interfaces, pp. 97-104, State College, PA, USA, ACM, New York, NY, USA, ISBN 1-58113-995-0, doi: http://doi.acm.org/10.1145/1027933.1027952.
• [38] Stefan Kopp, Paul A. Tepper, Kimberley Ferriman, Kristina Striegnitz, and Justine Cassell (2007), Trading Spaces: How Humans and Humanoids Use Speech and Gesture to Give Directions, pp. 133-160, John Wiley & Sons, Ltd, ISBN 9780470512470, doi: 10.1002/9780470512470.ch8, http://dx.doi.org/10.1002/9780470512470.ch8.
• [39] Peter Kühnlein, Manja Nimke, and Jens Stegmann (2002), Towards an HPSG-based Formalism for the Integration of Speech and Co-Verbal Pointing, in Proceedings of Gesture – The Living Medium, Austin, Texas.
• [40] Alex Lascarides and Matthew Stone (2006), Formal Semantics for Iconic Gesture, in Proceedings of Brandial’06, the 10th International Workshop on the Semantics and Pragmatics of Dialogue (SemDial10), pp. 125-132, Universitätsverlag Potsdam, Potsdam, Germany.
• [41] Alex Lascarides and Matthew Stone (2009a), Discourse Coherence and Gesture Interpretation, Gesture, 9 (2): 147-180.
• [42] Alex Lascarides and Matthew Stone (2009b), A Formal Semantic Analysis of Gesture, Journal of Semantics, 26 (4): 393-449.
• [43] Stephen C. Levinson (1983), Pragmatics, Cambridge University Press, Cambrdige.
• [44] Daniel Loehr (2004), Gesture and Intonation, Georgetown University, Washington DC, doctoral dissertation.
• [45] Andy Lücking, Hannes Rieser, and Marc Staudacher (2006a), Multi-modal Integration for Gesture and Speech, in David Schlangen and Raquel Fernández, editors, brandial’06 – Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, pp. 106–113, Universitätsverlag Potsdam, Potsdam.
• [46] Andy Lücking, Hannes Rieser, and Marc Staudacher (2006b), SDRT and Multi-modal Situated Communication, in David Schlangen and Raquel Fernández, editors, brandial’06 – Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, pp. 72-79, Universitätsverlag Potsdam, Potsdam.
• [47] David McNeill (1992), Hand and Mind. What Gestures Reveal about Thought, University of Chicago Press, Chicago.
• [48] David McNeill (2005), Gesture and Thought, University of Chicago Press, Chicago.
• [49] Richard Montague (1988), The Proper Treatment of Quantification in Ordinary English, in Jack Kulas, James H. Fetzer, and Terry L. Rankin, editors, Philosophy, Language, and Artificial Intelligence, volume 2 of Studies in Cognitive Systems, pp. 141-162, Springer Netherlands, ISBN 978-94-010-7726-2, doi: 10.1007/978-94-009-2727-8_7, http://dx.doi.org/10.1007/978-94-009-2727-8_7.
• [50] Cornelia Müller, Jana Bressem, and Silva H. Ladewig (2013), Towards a grammar of gesture – a form based view, Body-Language-Communication: An International Handbook on Multimodality in Human Interaction. (Handbooks of Linguistics and Communication Science 38.1), pp. 707-733.
• [51] Stephan Oepen (2001), [incr tsdb()] — Competence and Performance Laboratory. User Manual, Technical report, Computational Linguistics, Saarland University, Saarbrücken, Germany.
• [52] Stephan Oepen, Klaus Netter, and Judith Klein (1997), TSNLP — Test Suites for Natural Language Processing, in John Nerbonne, editor, Linguistic Databases, pp. 13-36, CSLI Publications, Stanford, CA.
• [53] Patrizia Paggio and Costanza Navarretta (2009), Integration and representation issues in the annotation of multimodal data, in Costanza Navarretta, Patrizia Paggio, Jens Allwood, Elisabeth Alsén, and Yasuhiro Katagiri, editors, Proceedings of the NODALIDA 2009 workshop Multimodal Communication — from Human Behaviour to Computational Models, volume 6, pp. 25-31, Northern European Association for Language Technology (NEALT).
• [54] Thies Pfeiffer, Florian Hofmann, Florian Hahn, Hannes Rieser, and Insa Röpke (2013), Gesture Semantics Reconstruction Based on Motion Capturing and Complex Event Processing: a Circular Shape Example, in Proceedings of the SIGDIAL 2013 Conference, pp. 270-279, Association for Computational Linguistics, http://aclweb.org/anthology/W13-4041.
• [55] Livia Polanyi (1985), A Theory of Discourse Structure and Discourse Coherence, in Proceedings of the 21st Meeting of the Chicago Linguistics Society, Chicago, Illinois: Linguistics Department, University of Chicago.
• [56] Uwe Reyle (1993), Dealing with Ambiguities by Underspecification: Construction, Representation and Deduction, Journal of Semantics, 10: 123-179.
• [57] I. A. Sag and T. A. Wasow (1999), Syntactic Theory: A Formal Introduction, Center for the Study of Language and Information, Stanford, California, ISBN 1575861615 (hard cover), 1575861607 (paper).
• [58] Mark Steedman (2000), The Syntactic Process, The MIT Press.
• [59] Francis & Mark Turner Steen (2013), Multimodal Construction Grammar, Language and the Creative Mind, pp. 255-274.

Uwagi

Opracowanie rekordu w ramach umowy 509/P-DUN/2018 ze środków MNiSW przeznaczonych na działalność upowszechniającą naukę (2018).