Natural Language Understanding and Prediction: from Formal Grammars to Large Scale Machine Learning

• Autorzy: Duta N.

Identyfikatory

DOI

10.3233/FI-2014-1023

• Języki publikacji: EN

Abstrakty

EN

Scientists have long dreamed of creating machines humans could interact with by voice. Although one no longer believes Turing's prophecy that machines will be able to converse like humans in the near future, real progress has been made in the voice and text-based human-machine interaction. This paper is a light introduction and survey of some deployed natural language systems and technologies and their historical evolution. We review two fundamental problems involving natural language: the language prediction problem and the language understanding problem. While describing in detail all these technologies is beyond our scope, we do comment on some aspects less discussed in the literature such as language prediction using huge models and semantic labeling using Marcus contextual grammars.

Słowa kluczowe

EN

natural language understanding; language modeling; language prediction

• Wydawca: IOS Press
• Czasopismo: Fundamenta Informaticae
• Rocznik: 2014
• Tom: Vol. 131, nr 3-4
• Strony: 425--440
• Opis fizyczny: Bibliogr. 52 poz., rys., tab.

Twórcy

autor

Duta N.

• New England Research and Development Center, Microsoft, Cambridge, USA

Bibliografia

• [1] Agarwal, G., Kabra, G., Chang, K. C. C.: Towards rich query interpretation: walking back and forth for mining query templates, in Proc of the 19th international conference on World Wide Web, 2010, 1-10.
• [2] Arisoy E., Sainath T. N., Kingsbury B., Ramabhadran B.: Deep neural network language models. In Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, 2012, 20-28.
• [3] Beesley K. R., Karttunen L.: Finite state morphology, Center for the Study of Language and Information Publication, 2003. FST toolkit can be downloaded at http://www.stanford.edu/ laurik/fsmbook/home.html
• [4] Bellegarda J. R.: Statistical language model adaptation: review and perspectives, Speech Communications, 42, 2004, 93-108.
• [5] Bortnikov E., Donmez P., Kagian A., Lempel R.: Modeling transactional queries via templates, in Proc. 34th European conference on Advances in Information Retrieval ECIR 2012, 2012, 13-24.
• [6] Brill E., Moore R. C.: An improved error model for noisy channel spelling correction, in Proc. of the 38th Annual Meeting on Association for Computational Linguistics, 2000, 286-293.
• [7] Broder, A.: A taxonomy of web search, In ACM Sigir forum, 36(2), 2002, 3-10.
• [8] Bulyko I., Ostendorf M., Siu M., Ng T., Stolcke A., Cetin O.: Web resources for language modeling in conversational speech recognition, ACM Trans. on Speech and Language Processing, 5(1), 2007.
• [9] Burns M.: Nuance supercharges Swype, adds new keyboard options, XT9 predictive text and Dragonpowered voice input, TechCrunch.com, 2012, Available at http://techcrunch.com/2012/06/20/ nuancesupercharges-swype-adds- new-keyboard-options- xt9-predictive-text-and- dragon-powered- voice-input/
• [10] Celikyilmaz A., Hakkani-Tur D., Tur G.: Statistical semantic interpretation modeling for spoken language understandingwith enriched semantic features, in Proc of IEEE Workshop on Spoken Language Technologies, 2012, 216-221.
• [11] Charniak E., Gales M.: Personal communication, EARS RT-04 workshop, Yorktown Heights, NY, 2004.
• [12] Chelba C., Brants T., Neveitt W., Xu P.: Study on interaction between entropy pruning and Kneser-Ney smoothing, in Proc of Interspeech, 2242-2245.
• [13] Chelba C., Bikel D. M., Shugrina M., Nguyen P., Kumar S.: Large scale language modeling in automatic speech recognition, Google technical report, 2012, Available at: http://static.googleusercontent.com/ external_content/untrusted dlcp/ research.google.com/en/us/ pubs/archive/40491.pdf
• [14] Chelba C., Xu P., Pereira F., Richardson T.: Distributed acoustic modeling with back-off n-grams, in Proc of ICASSP 2012, 2012, 4129-4132.
• [15] Chomsky N.: Three models for the description of language, IRE Transactions on Information Theory, 2, 1956, 113-124.
• [16] Dean J., Ghemawat S.: MapReduce: simplified data processing on large clusters, OSDI’04: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA, December, 2004.
• [17] Dowding J., Gawron J. M., Appelt D., Bear J., Cherny L., Moore R., Moran D.: Gemini: A natural language system for spoken language understanding, in Proc. ARPA Workshop on Human Language Technology, Princeton, NJ, Mar. 1993.
• [18] Duffy J.: Apple’s Siri versus Dragon Go! and Vlingo, PC Magazine, 10/6/2011, Available at http://www.pcmag.com/article2/0,2817,2394267,00.asp
• [19] Duta N.: Transcription-less call routing using unsupervised language model adaptation, In Proc. Interspeech 2008, Brisbane, Australia, September 22-26, 2008.
• [20] Duta, N., Schwartz R.: Using a large LM, EARS technical workshop, Eurospeech 2003, Martigny, Switzerland.
• [21] Duta, N., Schwartz R., Makhoul J.: Analysis of the errors produced by the 2004 BBN speech recognition system in the DARPA EARS evaluations, IEEE Trans. on Audio, Speech, and Language Processing, 14(5), 2006, 1745-1753.
• [22] Gorin A. L., Riccardi G., Wright J. H.: How may I help you? Speech Communication, 23(1-2), 1997, 113-127.
• [23] Gupta N., Tur G., Hakkani-Tur D., Bangalore S., Riccardi G., Gilbert M.: The AT&T spoken language understanding system, IEEE Trans. on Audio, Speech, and Language Processing, 14(1), 2006, 213 - 222.
• [24] Imielinski, T., Signorini, A.: If you ask nicely, I will answer: semantic search and today’s search engines, In Proc. IEEE International Conference on Semantic Computing, 2009, 184-191.
• [25] Jelinek F.: Statistical methods for speech recognition, MIT Press, 2001.
• [26] Kirchhoff K., Bilmes J., Da, S., Duta N., Egan M., Ji G., He F., Henderson J., Liu D., Noamany M., Schone P., Schwartz R., Vergyri D.: Novel approaches to Arabic speech recognition: Report from the 2002 John-Hopkins summer workshop, In Proc. ICASSP 2003, I 344-347.
• [27] Kneser R., Ney H.: Improved backing-off for m-gram language modeling, In Proc. ICASSP 1995, 181-184.
• [28] Lopez, A.: Statistical machine translation, ACM Computing Surveys, 40(3), 2008, 1-49.
• [29] Marcus S., Paun G., Martin-Vide C.: Contextual grammars as generative models of natural languages Computational Linguistics, 24(2), 1998, 245-274.
• [30] Miller S., Bobrow R., Ingria R., Schwartz R.: Hidden understanding models of natural language, in Proc. Annual Meeting Association for Computational Linguistics, Las Cruces, NM, Jun. 1994.
• [31] Mohri M., Pereira F. C. N., and Riley M.: The design principles of a weighted finite-state transducer library, Theoretical Computer Science, 231, 2000, 17-32.
• [32] Mori S., Nishida H., Yamada H.: Optical character recognition, John Wiley and Sons, 1999.
• [33] Natarajan P., Prasad R., Suhm B., McCarthy D.: Speech enabled natural language call routing: BBN call director, in Proc. Int. Conf. Spoken Language Processing, Denver, CO, Sep. 2002.
• [34] Parameswaran, A., Kaushik, R., Arasu, A.: Efficient parsing-based keyword search over databases, Technical Report, Stanford University, 2012.
• [35] Pasca M., Lin D., Bigham J., Lifchits A., Jain A.: Organizing and searching the World Wide Web of facts - Step one: the one-million fact extraction challenge, in Proc. of the 21st National Conference on Artificial Intelligence (AAAI-06), Boston, Massachusetts, 2006, 1400-1405.
• [36] Paun Gh.: Marcus contextual grammars, Kluwer Academic Publishers, Dordrecht, Boston, London, 1997.
• [37] Pieraccini R., Tzoukermann E., Gorelov Z., Levin E., Lee C., Gauvain JL.: Progress report on the Chronus system: ATIS benchmark results, In Proc. of the workshop on Speech and Natural Language, 1992, 67-71.
• [38] Plamondon R., Srihari S. N.: On-line and off-line handwriting recognition: a comprehensive survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(1), 2000, 63-84.
• [39] Price P. J.: Evaluation of spoken language systems: The ATIS domain, in Proc. DARPA Workshop on Speech and Natural Language, Hidden Valley, PA, Jun. 1990.
• [40] Rosenfeld, R.: Two decades of statistical language modeling: Where do we go from here? , Proceedings of the IEEE, 88(8), 2000.
• [41] Rubinstein, Y. D., Hastie, T.: Discriminative vs. informative learning, In Proc. Third Int. Conf. on Knowledge Discovery and Data Mining, 1997, 49-53.
• [42] Seneff S.: TINA: A natural language system for spoken language applications, Computational linguist, 18(1), 1992, 61-86.
• [43] Solomonoff R.: A formal theory of inductive inference, Information and Control, Part I: 7(1), 1964, 1-22.
• [44] Stolcke A.: Entropy-based pruning of backoff language models, in Proc. DARPA Broadcast News Transcription and Understanding Workshop, Lansdowne, VA, 1998, 270-274.
• [45] Tur G., De Mori R. (eds): Spoken language understanding - Systems for extracting semantic information from speech, JohnWiley and Sons, 2011.
• [46] Turing A. M.: Computing machinery and intelligence, Mind, 49, 1950, 433-460.
• [47] Wang Y. Y., Acero A., Chelba C.: Is word error rate a good indicator for spoken language understanding accuracy?, in Proc. ARPA HLT Workshop, St. Thomas, USA, pp. 577-582, 2003.
• [48] Wang Y. Y., Deng L, Acero A.: Semantic frame-based spoken language understanding, in Tur G., DeMori R. Eds. Spoken Language Understanding, John Wiley and Sons, 2011.
• [49] Wang Y. Y., Yu D., Ju Y.C, Acero A.: Voice search, in Tur G., DeMori R. Eds. Spoken Language Understanding, JohnWiley and Sons, 2011.
• [50] Ward W., Issar S.: Recent improvements in the CMU spoken language understanding system, in Proc. DARPA Broadcast News Transcription and Understanding Workshop, Lansdowne, VA, 1998, 270-274.
• [51] Witten, I., Bell, T.: The zero-frequency problem: Estimating the probabilities of novel events in adaptive text compression, IEEE Trans. on Inform. Theory, 37(4), 1991, 1085-1094.
• [52] The history of automatic speech recognition evaluations at NIST, Available at http://www.itl.nist.gov/iad/mig/publications/ASRhistory