Neural heuristics for scaling constructional language processing

• Autorzy: Van Eecke Paul , Nevens Jens , Beuls Katrien

Identyfikatory

DOI

10.15398/jlm.v10i2.318

• Języki publikacji: EN

Abstrakty

EN

Constructionist approaches to language make use of form-meaning pairings, called constructions, to capture all linguistic knowledge that is necessary for comprehending and producing natural language expressions. Language processing consists then in combining the constructions of a grammar in such a way that they solve a given language comprehension or production problem. Finding such an adequate sequence of constructions constitutes a search problem that is combinatorial in nature and becomes intractable as grammars increase in size. In this paper, we introduce a neural methodology for learning heuristics that substantially optimise the search processes involved in constructional language processing. We validate the methodology in a case study for the CLEVR benchmark dataset. We show that our novel methodology outperforms state-of-the-art techniques in terms of size of the search space and time of computation, most markedly in the production direction. The results reported on in this paper have the potential to overcome the major efficiency obstacle that hinders current efforts in learning large-scale construction grammars, thereby contributing to the development of scalable constructional language processing systems.

Słowa kluczowe

EN

neuro-symbolic AI; neural heuristics; language processing; computational construction grammar

• Wydawca: Instytut Podstaw Informatyki PAN
• Czasopismo: Journal of Language Modelling
• Rocznik: 2022
• Tom: Vol. 10, No. 2
• Strony: 287--314.
• Opis fizyczny: Bibliogr. 51 poz., rys., tab., wykr.

Twórcy

autor

Van Eecke Paul

• Artificial Intelligence Laboratory, Vrije Universiteit Brussel

• https://orcid.org/0000-0001-9153-9092

autor

Nevens Jens

• Artificial Intelligence Laboratory, Vrije Universiteit Brussel

• https://orcid.org/0000-0001-5413-2955

autor

Beuls Katrien

• Faculté d’informatique, Université de Namur

• https://orcid.org/+0000-0003-4451-4778+

Bibliografia

• 1. Dzmitry BAHDANAU, Kyunghyun CHO, and Yoshua BENGIO (2015), Neural machine translation by jointly learning to align and translate, in International Conference on Learning Representations (ICLR 2015), pp. 1–15.
• 2. Katrien BEULS (2011), Construction sets and unmarked forms: A case study for Hungarian verbal agreement, in Luc STEELS, editor, Design Patterns in Fluid Construction Grammar, pp. 237–264, John Benjamins, Amsterdam, Netherlands.
• 3. Katrien BEULS and Paul VAN EECKE (2023), Fluid Construction Grammar: State of the art and future outlook, in Proceedings of the First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023), pp. 41–50, Association for Computational Linguistics, Washington, D.C., USA.
• 4. Katrien BEULS, Paul VAN EECKE, and Vanja Sophie CANGALOVIC (2021), A computational construction grammar approach to semantic frame extraction, Linguistics Vanguard, 7(1):20180015.
• 5. Joris BLEYS, Kevin STADLER, and Joachim DE BEULE (2011), Search in linguistic processing, in Luc STEELS, editor, Design Patterns in Fluid Construction Grammar, pp. 149–179, John Benjamins, Amsterdam, Netherlands.
• 6. William CROFT and D. Alan CRUSE (2004), Cognitive linguistics, Cambridge University Press, Cambridge, United Kingdom.
• 7. Holger DIESSEL (2015), Usage-based construction grammar, in Ewa DĄBROWSKA and Dagmar DIVJAK, editors, Handbook of Cognitive Linguistics, pp. 295–321, Mouton de Gruyter, Berlin, Germany.
• 8. Jonas DOUMEN, Katrien BEULS, and Paul VAN EECKE (2023), Modelling language acquisition through syntactico-semantic pattern finding, in Findings of the Association for Computational Linguistics: EACL 2023, forthcoming.
• 9. Timothy DOZAT and Christopher D. MANNING (2017), Deep biaffine attention for neural dependency parsing, in 5th International Conference on Learning Representations, ICLR 2017, pp. 1–8.
• 10. Timothy DOZAT and Christopher D. MANNING (2018), Simpler but more accurate semantic dependency parsing, in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 484–490, Association for Computational Linguistics, Melbourne, Australia, doi:10.18653/v1/P18- 2077.
• 11. Patrick FERBER, Malte HELMERT, and Jörg HOFFMANN (2020), Neural network heuristics for classical planning: A study of hyperparameter space, in 24th European Conference on Artificial Intelligence, pp. 2346–2353.
• 12. Charles J. FILLMORE (1988), The mechanisms of “construction grammar”, in Annual Meeting of the Berkeley Linguistics Society, volume 14, pp. 35–55.
• 13. Jonas GEHRING, Michael AULI, David GRANGIER, Denis YARATS, and Yann N. DAUPHIN (2017), Convolutional sequence to sequence learning, in Proceedings of the 34th International Conference on Machine Learning, pp. 1243–1252.
• 14. Adele E. GOLDBERG (1995), Constructions: A construction grammar approach to argument structure, University of Chicago Press, Chicago, IL, USA.
• 15. Adele E. GOLDBERG (2003), Constructions: A new theoretical approach to language, Trends in Cognitive Sciences, 7(5):219–224.
• 16. Adele E. GOLDBERG (2006), Constructions at work: The nature of generalization in language, Oxford University Press, Oxford, United Kingdom.
• 17. Robin JIA and Percy LIANG (2016), Data recombination for neural semantic parsing, in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12–22, Association for Computational Linguistics, Berlin, Germany, doi:10.18653/v1/P16-1002.
• 18. Justin JOHNSON, Bharath HARIHARAN, Laurens VAN DER MAATEN, Li FEI-FEI, C. LAWRENCE ZITNICK, and Ross GIRSHICK (2017), CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning, in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2901–2910, IEEE Computer Society, Los Alamitos, CA, USA.
• 19. Martin KAY (1996), Chart generation, in 34th Annual Meeting of the Association for Computational Linguistics, pp. 200–204, Association for Computational Linguistics.
• 20. Paul KAY and Charles FILLMORE (1999), Grammatical constructions and linguistic generalizations: The what’s x doing y? construction, Language, 75(1):1–33.
• 21. Eliyahu KIPERWASSER and Yoa GOLDBERG (2016), Simple and accurate dependency parsing using bidirectional LSTM feature representations, Transactions of the Association for Computational Linguistics, 4:313–327, doi:10.1162/tacl_a_00101, https://aclanthology.org/Q16-1023.
• 22. Ioannis KONSTAS, Srinivasan IYER, Mark YATSKAR, Yejin CHOI, and Luke ZETTLEMOYER (2017), Neural AMR: Sequence-to-sequence models for parsing and generation, in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 146–157, Association for Computational Linguistics, Vancouver, Canada, doi:10.18653/v1/P17-1014.
• 23. Volodymyr MNIH, Koray KAVUKCUOGLU, David SILVER, Andrei A. RUSU, Joel VENESS, Marc G. BELLEMARE, Alex GRAVES, Martin RIEDMILLER, Andreas K. FIDJELAND, Georg OSTROVSKI, et al. (2015), Human-level control through deep reinforcement learning, Nature, 518(7540):529–533.
• 24. Stefan MÜLLER (2017), Head-Driven Phrase Structure Grammar, Sign-Based Construction Grammar, and Fluid Construction Grammar: Commonalities and differences, Constructions and Frames, 9(1):139–173.
• 25. Jens NEVENS, Jonas DOUMEN, Paul VAN EECKE, and Katrien BEULS (2022), Language acquisition through intention reading and pattern finding, in Proceedings of the 29th International Conference on Computational Linguistics, pp. 15–25, International Committee on Computational Linguistics, Gyeongju, Republic of Korea, https://aclanthology.org/2022.coling-1.2.
• 26. Jens NEVENS, Paul VAN EECKE, and Katrien BEULS (2019), Computational construction grammar for visual question answering, Linguistics Vanguard, 5(1):20180070.
• 27. Allen NEWELL and Herbert SIMON (1956), The logic theory machine – a complex information processing system, IRE Transactions on Information Theory, 2(3):61–79.
• 28. Nils NILSSON (1971), Problem-solving methods in artificial intelligence, McGraw-Hill Book Company, New York, NY, USA.
• 29. Fernando PEREIRA and David WARREN (1983), Parsing as deduction, in 21st Annual Meeting of the Association for Computational Linguistics, pp. 137–144, Association for Computational Linguistics.
• 30. Daniel L. SCHACTER and Randy L. BUCKNER (1998), Priming and the brain, Neuron, 20(2):185–195.
• 31. Stuart M. SHIEBER (1988), A uniform architecture for parsing and generation, in Coling Budapest 1988 Volume 2: International Conference on Computational Linguistics, pp. 614–619, https://aclanthology.org/C88-2128.
• 32. Josefina SIERRA SANTIBÁÑEZ (2012), A logic programming approach to parsing and production in Fluid Construction Grammar, in Luc STEELS, editor, Computational Issues in Fluid Construction Grammar, volume 7249 of Lecture Notes in Computer Science, pp. 239–255, Springer, Berlin, Germany.
• 33. David SILVER, Aja HUANG, Chris J. MADDISON, Arthur GUEZ, Laurent SIFRE, George VAN DEN DRIESSCHE, Julian SCHRITTWIESER, Ioannis ANTONOGLOU, Veda PANNEERSHELVAM, Marc LANCTOT, et al. (2016), Mastering the game of Go with deep neural networks and tree search, Nature, 529(7587):484–489.
• 34. Luc STEELS, editor (2011), Design patterns in Fluid Construction Grammar, John Benjamins, Amsterdam, Netherlands.
• 35. Luc STEELS and Joachim DE BEULE (2006), Unify and merge in Fluid Construction Grammar, in International Workshop on Emergence and Evolution of Linguistic Communication (EELC 2006), pp. 197–223, Rome, Italy.
• 36. Ilya SUTSKEVER, Oriol VINYALS, and Quoc V. LE (2014), Sequence to sequence learning with neural networks, in Z. GHAHRAMANI, M. WELLING, C. CORTES, N. LAWRENCE, and K.Q. WEINBERGER, editors, Advances in Neural Information Processing Systems, volume 27, pp. 3104–3112, Curran Associates, Inc., Red Hook, NY, USA.
• 37. Takeshi TAKAHASHI, He SUN, Dong TIAN, and Yebin WANG (2019), Learning heuristic functions for mobile robot path planning using deep neural networks, in Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pp. 764–772.
• 38. Michael TOMASELLO (2003), Constructing a language: A usage-based theory of language acquisition, Harvard University Press, Harvard, MA, USA.
• 39. Paul VAN EECKE and Katrien BEULS (2017), Meta-layer problem solving for computational construction grammar, in The 2017 AAAI Spring Symposium Series, pp. 258–265, AAAI Press, Palo Alto, CA, USA.
• 40. Paul VAN EECKE and Katrien BEULS (2018), Exploring the creative potential of computational construction grammar, Zeitschrift für Anglistik und Amerikanistik, 66(3):341–355.
• 41. Rik VAN NOORD, Lasha ABZIANIDZE, Antonio TORAL, and Johan BOS (2018), Exploring neural methods for parsing discourse representation structures, Transactions of the Association for Computational Linguistics, 6:619–633, doi:10.1162/tacl_a_00241.
• 42. Remi VAN TRIJP (2016), Chopping down the syntax tree: What constructions can do instead, Belgian Journal of Linguistics, 30(1):15–38.
• 43. Remi VAN TRIJP, Katrien BEULS, and Paul VAN EECKE (2022), The FCG editor: An innovative environment for engineering computational construction grammars, PLOS ONE, 17(6):e0269708, doi:10.1371/journal.pone.0269708.
• 44. Ashish VASWANI, Noam SHAZEER, Niki PARMAR, Jakob USZKOREIT, Llion JONES, Aidan N. GOMEZ, Lukasz KAISER, and Illia POLOSUKHIN (2017), Attention is all you need, in I. GUYON, U. VON LUXBURG, S. BENGIO, H. WALLACH, R. FERGUS, S. VISHWANATHAN, and R. GARNETT, editors, Advances in Neural Information Processing Systems, volume 30, pp. 6000–6010, Curran Associates, Inc., Red Hook, NY, USA.
• 45. Lara VERHEYEN, Jérôme Botoko EKILA, Jens NEVENS, Paul VAN EECKE, and Katrien BEULS (2022), Hybrid procedural semantics for visual dialogue: An interactive web demonstration, in Workshop on semantic techniques for narrative-based understanding: Workshop at IJCAI-ECAI 2022, pp. 48–52.
• 46. Jingyuan WANG, Ning WU, Wayne Xin ZHAO, Fanzhang PENG, and Xin LIN (2019), Empowering A* search algorithms with neural networks for personalized route recommendation, in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 539–547.
• 47. Pieter WELLENS (2011), Organizing constructions in networks, in Luc STEELS, editor, Design Patterns in Fluid Construction Grammar, pp. 181–201, John Benjamins, Amsterdam, Netherlands.
• 48. Pieter WELLENS and Joachim DE BEULE (2010), Priming through constructional dependencies: a case study in Fluid Construction Grammar, in The Evolution of Language: Proceedings of the 8th International Conference (EVOLANG8), pp. 344–351, World Scientific.
• 49. Tom WILLAERT, Paul VAN EECKE, Katrien BEULS, and Luc STEELS (2020), Building social media observatories for monitoring online opinion dynamics, Social Media + Society, 6(2), doi:10.1177/2056305119898778.
• 50. Tom WILLAERT, Paul VAN EECKE, Jeroen VAN SOEST, and Katrien BEULS (2021), An opinion facilitator for online news media, Frontiers in Big Data, 4:1–10.
• 51. Chen YU and Daniel GILDEA (2022), Sequence-to-sequence AMR parsing with ancestor information, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 571–577, Association for Computational Linguistics, Dublin, Ireland, doi:10.18653/v1/2022.acl- short.63.

Uwagi

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2022-2023).