A dependency-based approach to word contextualization using compositional distributional semantics

• Autorzy: Gamallo Pablo

Identyfikatory

DOI

10.15398/jlm.v7i1.201

• Języki publikacji: EN

Abstrakty

EN

We propose a strategy to build the distributional meaning of sentences mainly based on two types of semantic objects: context vectors associated with content words and compositional operations driven by syntactic dependencies. The compositional operations of a syntactic dependency make use of two input vectors to build two new vectors representing the contextualized sense of the two related words. Given a sentence, the iterative application of dependencies results in as many contextualized vectors as content words the sentence contains. At the end of the contextualization process, we do not obtain a single compositional vector representing the semantic denotation of the whole sentence (or of the root word), but one contextualized vector for each constituent word of the sentence. Our method avoids the troublesome high-order tensor representations of approaches relying on category theory, by defining all words as first-order tensors (i.e. standard vectors). Some corpus-based experiments are performed to both evaluate the quality of the contextualized vectors built with our strategy, and to compare them to other approaches on distributional compositional semantics. The experiments show that our dependency-based method performs as (or even better than) the state-of-the-art.

Słowa kluczowe

EN

distributional semantics; compositionality; dependency-based parsing

• Wydawca: Instytut Podstaw Informatyki PAN
• Czasopismo: Journal of Language Modelling
• Rocznik: 2019
• Tom: Vol. 7, No. 1
• Strony: 99--138
• Opis fizyczny: Bibliogr. 71 poz., tab.

Twórcy

autor

Gamallo Pablo

• Centro de Investigación en Tecnoloxías Intelixentes (CiTIUS), University of Santiago de Compostela, Galiza

Bibliografia

• [1] Marco Baroni (2013), Composition in Distributional Semantics, Language and Linguistics Compass, 7: 511-522.
• [2] Marco Baroni, Raffaella Bernardi, and Roberto Zamparelli (2014), Frege in Space: A Program for Compositional Distributional Semantics, Linguistic Issues in Language Technology (LiLT), 9: 241-346.
• [3] Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta (2009), The WaCky Wide Web: A Collection of Very Large Linguistically Processed Webcrawled Corpora, Language Resources and Evaluation, 43 (3): 209-226.
• [4] Marco Baroni and Roberto Zamparelli (2010), Nouns Are Vectors, Adjectives Are Matrices: Representing Adjective-noun Constructions in Semantic Space, in Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP’10, pp. 1183-1193, Stroudsburg, PA, USA.
• [5] Jon Barwise (1987), Recent Developments in Situation Semantics, in M. Nagao, editor, Language and Artificial Intelligence, pp. 387-399, North Holla.
• [6] Raffaella Bernardi, Georgiana Dinu, Marco Marelli, and Marco Baroni (2013), A Relatedness Benchmark to Test the Role of Determiners in Compositional Distributional Semantics, in The 51st Annual Meeting of the Association for Computational Linguistics ACL-2013, pp. 53-57, The Association for Computational Linguistics.
• [7] Chris Biemann and Martin Riedl (2013), Text: Now in 2D! A Framework for Lexical Expansion with Contextual Similarity, Journal of Language Modelling, 1 (1): 55-95.
• [8] B. Coecke, M. Sadrzadeh, and S. Clark (2010), Mathematical Foundations for a Compositional Distributional Model of Meaning, Linguistic Analysis, 36 (1-4): 345-384.
• [9] Ann Copestake and Aurelie Herbelot (2012), Lexicalised Compositionality, in http://www.cl.cam.ac.uk/˜ah433/lc-semprag.pdf, Unpublished article.
• [10] Fabrizio Costa, Vincenzo Lombardo, Paolo Frasconi, and Giovanni Soda (2001), Wide Coverage Incremental Parsing by Learning Attachment Preferences, in Conference of the Italian Association for Artificial Intelligence (AIIA), pp. 297-307.
• [11] Georgiana Dinu, Nghia Pham, and Marco Baroni (2013a), DISSECT: DIStributional SEmantics Composition Toolkit, in ACL 2013 Workshop on Continuous Vector Space Models and their Compositionality (CVSC 2013), pp. 31-36, East Stroudsburg PA.
• [12] Georgiana Dinu, Nghia Pham, and Marco Baroni (2013b), General Estimation and Evaluation of Compositional Distributional Semantic Models, in ACL 2013 Workshop on Continuous Vector Space Models and their Compositionality (CVSC 2013), pp. 50-58, East Stroudsburg PA.
• [13] Ted Dunning (1993), Accurate Methods for the Statistics of Surprise and Coincidence, Computational Linguistics, 19 (1): 61-74.
• [14] Katrin Erk (2013), Towards a Semantics for Distributional Representations, in 10th International Conference on Computational Semantics (IWCS 2013), pp. 95-106.
• [15] Katrin Erk and Sebastian Padó (2008), A Structured Vector Space Model for Word Meaning in Context, in 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP-2008, pp. 897-906, Honolulu, HI.
• [16] Katrin Erk and Sebastian Padó (2009), Paraphrase Assessment in Structured Vector Space: Exploring Parameters and Datasets, in Proceedings of the EACL Workshop on Geometrical Methods for Natural Language Semantics, pp. 57-65, Athens, Greece.
• [17] John Rupert Firth (1957), A synopsis of linguistic theory 1930-1955, Studies in Linguistic Analysis, pp. 1-32.
• [18] Pablo Gamallo (2008), The Meaning of Syntactic Dependencies, Linguistik OnLine, 35 (3): 33-53.
• [19] Pablo Gamallo (2015), Dependency Parsing with Compression Rules, in Proceedings of the 14th International Workshop on Parsing Technology (IWPT 2015), pp. 107-117, Association for Computational Linguistics, Bilbao, Spain.
• [20] Pablo Gamallo (2017a), Comparing Explicit and Predictive Distributional Semantic Models Endowed with Syntactic Contexts, Language Resources and Evaluation, 51 (3): 727-743.
• [21] Pablo Gamallo (2017b), The Role of Syntactic Dependencies in Compositional Distributional Semantics, Corpus Linguistics and Linguistic Theory, 13 (2): 261-289.
• [22] Pablo Gamallo (2017c), Sense Contextualization in a Dependency-Based Compositional Distributional Model, in Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 1-9, Association for Computational Linguistics, doi: 10.18653/v1/W17-2601, http://aclweb.org/anthology/W17-2601.
• [23] Pablo Gamallo, Alexandre Agustini, and Gabriel Lopes (2005), Clustering Syntactic Positions with Similar Semantic Requirements, Computational Linguistics, 31 (1): 107-146.
• [24] Pablo Gamallo and Stefan Bordag (2011), Is Singular Value Decomposition Useful for Word Simalirity Extraction, Language Resources and Evaluation, 45 (2): 95-119.
• [25] Pablo Gamallo and Marcos Garcia (2018), Dependency Parsing with Finite State Transducers and Compression Rules, Information Processing & Management, 54 (6): 1244-1261.
• [26] Pablo Gamallo and Martín Pereira-Fariña (2017), Compositional Semantics using Feature-Based Models from WordNet, in Proceedings of the 1st Workshop on Sense, Concept and Entity Representations and their Applications, pp. 1-11, Association for Computational Linguistics, http://aclweb.org/anthology/W17-1901.
• [27] Dan Garrette, Katrin Erk, and Raymond Mooney (2014), A Formal Approach to Linking Logical Form and Vector-Space Lexical Semantics, in H. Bunt, J. Bos, and S. Pulman, editors, Text, Speech and Language Technology: Computing Meaning, pp. 27-48, Springer.
• [28] Edward Grefenstette and Mehrnoosh Sadrzadeh (2011a), Experimental Support for a Categorical Compositional Distributional Model of Meaning, in Conference on Empirical Methods in Natural Language Processing (EMNLP 2011), pp. 1394-1404.
• [29] Edward Grefenstette and Mehrnoosh Sadrzadeh (2011b), Experimenting with Transitive Verbs in a DisCoCat, in Workshop on Geometrical Models of Natural Language Semantics (EMNLP 2011).
• [30] Edward Grefenstette, Mehrnoosh Sadrzadeh, Stephen Clark, Bob Coecke, and Stephen Pulman (2011), Concrete Sentence Spaces for Compositional Distributional Models of Meaning, in Proceedings of the Ninth International Conference on Computational Semantics, IWCS’11, pp. 125-134.
• [31] Gregory Grefenstette (1995), Evaluation Techniques for Automatic Semantic Extraction: Comparing Syntatic and Window Based Approaches, in Branimir Boguraev and James Pustejovsky, editors, Corpus processing for Lexical Acquisition, pp. 205-216, The MIT Press.
• [32] Emiliano Guevara (2010), A Regression Model of Adjective-Noun Compositionality in Distributional Semantics, in Proceedings of the 2010 Workshop on GEometrical Models of Natural Language Semantics, GEMS’10, p. 33-37.
• [33] Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Padó (2015), Distributional Vectors Encode Referential Attributes, in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 12-21, Association for Computational Linguistics, Lisbon, Portugal, http://aclweb.org/anthology/D15-1002.
• [34] Kazuma Hashimoto and Yoshimasa Tsuruoka (2015), Learning Embeddings for Transitive Verb Disambiguation by Implicit Tensor Factorization, in Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pp. 1-11, Association for Computational Linguistics, Beijing, China, http://www.aclweb.org/anthology/W15-4001.
• [35] Richard Hudson (2003), The Psychological Reality of Syntactic Dependency Relations, in Proceedings of the First International Conference on Meaning-Text Theory, pp. 181-192, Paris.
• [36] Ozan Irsoy and Claire Cardie (2014), Deep Recursive Neural Networks for Compositionality in Language, in Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2096-2104, http://papers.nips.cc/paper/5551-deep-recursive-neural-networks-for-compositionality-in-language.
• [37] Sylvain Kahane (2003), Meaning-Text Theory, in Dependency and Valency: An International Handbook of Contemporary Reseach, Berlin: De Gruyter.
• [38] Hans Kamp and Uwe Reyle (1993), From Discourse to Logic: Introduction to Model-theoretic Semantics of Natural Languge. Formal Logic and Discourse Representation Theory, Kluwer Academic Publisher.
• [39] Dimitri Kartsaklis (2014), Compositional Operators in Distributional Semantics, Springer Science Reviews, 2 (1-2): 161-177.
• [40] Dimitri Kartsaklis, Nal Kalchbrenner, and Mehrnoosh Sadrzadeh (2014), Resolving Lexical Ambiguity in Tensor Regression Models of Meaning, in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Vol. 2: Short Papers), pp. 212-217, Association for Computational Linguistics, Baltimore, USA.
• [41] Dimitri Kartsaklis and Mehrnoosh Sadrzadeh (2013), Prior Disambiguation of Word Tensors for Constructing Sentence Vectors, in Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), pp. 1590-1601.
• [42] Ruth M. Kempson, Wilfried Meyer-Viol, and Dov Gabbay (1997), Language Understanding: A Procedural Perspective, in C. Retore, editor, First International Conference on Logical Aspects of Computational Linguistics, pp. 228-247, Lecture Notes in Artificial Intelligence Vol. 1328. Springer Verlag.
• [43] Ruth M. Kempson, Wilfried Meyer-Viol, and Dov Gabbay (2001), Dynamic Syntax: The Flow of Language Understanding, Blackwell, Oxford.
• [44] Thomas Kober, Julie Weeds, Jeremy Reffin, and David J. Weir (2016), Improving Sparse Word Representations with Distributional Inference for Semantic Composition, in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1691-1702, http://aclweb.org/anthology/D/D16/D16-1175.pdf.
• [45] Jayant Krishnamurthy and Tom Mitchell (2013), Vector Space Semantic Parsing: A Framework for Compositional Vector Space Models, in Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, pp. 1-10, Association for Computational Linguistics.
• [46] Germán Kruszewski and Marco Baroni (2014), Dead Parrots Make Bad Pets: Exploring Modifier Effects in Noun Phrases, in Proceedings of the Third Joint Conference on Lexical and Computational Semantics, *SEM@COLING 2014, August 23-24, 2014, Dublin, Ireland., pp. 171-181, http://aclweb.org/anthology/S/S14/S14-1021.pdf.
• [47] Ronald W. Langacker (1991), Foundations of Cognitive Grammar: Descriptive Applications, volume 2, Stanford University Press, Stanford.
• [48] Ken McRae, Todd R. Ferreti, and Liane Amyote (1997), Thematic Roles as Verb-specific Concepts, in M. C. MacDonald, editor, Lexical Representations and Sentence Processing, pp. 137-176, Psychology Press.
• [49] Oren Melamud, Ido Dagan, and Jacob Goldberger (2015), Modeling Word Meaning in Context with Substitute Vectors, in NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 – June 5, 2015, pp. 472-482, http://aclweb.org/anthology/N/N15/N15-1050.pdf.
• [50] George A. Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J. Miller (1990), Introduction to Wordnet: an On-Line Lexical Database, International Journal of Lexicography, 3 (4): 235-244.
• [51] David Milward (1992), Dynamics, Dependency Grammar and Incremental Interpretation, in 14th Conference on Computational Linguistics (COLING 1992), pp. 1095-1099, Nantes.
• [52] Jeff Mitchell and Mirella Lapata (2008), Vector-Based Models of Semantic Composition, in Proceedings of the Association for Computational Linguistics: Human Language Technologies (ACL-08: HLT), pp. 236-244, Columbus, Ohio.
• [53] Jeff Mitchell and Mirella Lapata (2009), Language Models Based on Semantic Composition, in Proceedings of Empirical Methods in Natural Language Processing (EMNLP-2009), pp. 430-439.
• [54] Jeff Mitchell and Mirella Lapata (2010), Composition in Distributional Models of Semantics, Cognitive Science, 34 (8): 1388-1439.
• [55] Richard Montague (1970), Universal Grammar, Theoria, 36 (3): 373-398.
• [56] Muntsa Padró, Marco Idiart, Aline Villavicencio, and Carlos Ramisch (2014), Nothing like Good Old Frequency: Studying Context Filters for Distributional Thesauri, in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 419-424.
• [57] Denis Paperno, Nghia The Pham, and Marco Baroni (2014), A Practical and Linguistically-Motivated Approach to Compositional Distributional Semantics, in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 90-99, Association for Computational Linguistics, Baltimore, Maryland, http://www.aclweb.org/anthology/P/P14/P14-1009.
• [58] Nghia The Pham, Germán Kruszewski, Angeliki Lazaridou, and Marco Baroni (2015), Jointly Optimizing Word Representations for Lexical and Sentential Tasks with the C-PHRASE Model, in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pp. 971-981, http://aclweb.org/anthology/P/P15/P15-1094.pdf.
• [59] Tamara Polajnar, Laura Rimell, and Stephen Clark (2015), An Exploration of Discourse-Based Sentence Spaces for Compositional Distributional Semantics, in Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-level Semantics, pp. 1-11, Association for Computational Linguistics, Lisbon, Portugal, http://aclweb.org/anthology/W15-2701.
• [60] James Pustejovsky (1995), The Generative Lexicon, MIT Press, Cambridge.
• [61] Siva Reddy, Ioannis P. Klapaftis, Diana McCarthy, and Suresh Manandhar (2011), Dynamic and Static Prototype Vectors for Semantic Composition, in Fifth International Joint Conference on Natural Language Processing, IJCNLP 2011, Chiang Mai, Thailand, November 8-13, 2011, pp. 705-713, http://aclweb.org/anthology/I/I11/I11-1079.pdf.
• [62] Mehrnoosh Sadrzadeh, Stephen Clark, and Bob Coecke (2013), The Frobenius Anatomy of Word Meanings I: Subject and Object Relative Pronouns, Journal of Logic and Computation, 23 (6): 1293-1317, doi: 10.1093/logcom/ext044, http://dx.doi.org/10.1093/logcom/ext044.
• [63] Matthias Schlesewsky and Ina Bornkessel (2004), On Incremental Interpretation: Degrees of Meaning Accessed During Sentence Comprehension, Lingua, 114: 1213-1234.
• [64] Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng (2012), Semantic Compositionality Through Recursive Matrix-vector Spaces, in Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL’12, pp. 1201-1211, Association for Computational Linguistics, Stroudsburg, PA, USA, http://dl.acm.org/citation.cfm?id=2390948.2391084.
• [65] Mark Steedman (1996), Surface Structure and Interpretation, The MIT Press.
• [66] Michael K. Tanenhaus and Greg Carlson (1989), Lexical Structure and Language Comprehension, in William Marslen-Wilson, editor, Lexical Representation and Process, pp. 530-561, The MIT Press.
• [67] Stefan Thater, Hagen Fürstenau, and Manfred Pinkal (2010), Contextualizing Semantic Representations Using Syntactically Enriched Vector Models, in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 948-957, Stroudsburg, PA, USA.
• [68] John Truswell, Michael K. Tanenhaus, and Susan M. Garnsey (1994), Semantic Influences on Parsing: Use of Thematic Role Information in Syntactic Ambiguity Resolution, Journal of Memory and Language, 33: 285-318.
• [69] Peter D. Turney (2013), Domain and Function: A Dual-Space Model of Semantic Relations and Compositions, Journal of Artificial Intelligence Research (JAIR), 44: 533-585.
• [70] David J. Weir, Julie Weeds, Jeremy Reffin, and Thomas Kober (2016), Aligning Packed Dependency Trees: A Theory of Composition for Distributional Semantics, Computational Linguistics, 42 (4): 727-761.
• [71] Fabio Massimo Zanzotto, Ioannis Korkontzelos, Francesca Fallucchi, and Suresh Manandhar (2010), Estimating Linear Models for Compositional Distributional Semantics, in Proceedings of the 23rd International Conference on Computational Linguistics, COLING’10, pp. 1263-1271.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2020).