Compressing sentiment analysis CNN models for efficient hardware processing

• Autorzy: Wróbel Krzysztof , Karwatowski Michał , Wielgosz Maciej , Pietroń Marcin , Wiatr Kazimierz

Identyfikatory

DOI

10.7494/csci.2020.21.1.3375

• Języki publikacji: EN

Abstrakty

EN

Convolutional neural networks (CNNs) were created for image classification tasks. Shortly after their creation, they were applied to other domains, including natural language processing (NLP). Nowadays, solutions based on artificial intelligence appear on mobile devices and embedded systems, which places constraints on memory and power consumption, among others. Due to CNN memory and computing requirements, it is necessary to compress them in order to be mapped to the hardware. This paper presents the results of the compression of efficient CNNs for sentiment analysis. The main steps involve pruning and quantization. The process of mapping the compressed network to an FPGA and the results of this implementation are described. The conducted simulations showed that the 5-bit width is enough to ensure no drop in accuracy when compared to the floating-point version of the network. Additionally, the memory footprint was significantly reduced (between 85 and 93% as compared to the original model).

Słowa kluczowe

EN

natural language processing; convolutional neural networks; FPGA; compression

• Wydawca: Wydawnictwa AGH
• Czasopismo: Computer Science
• Rocznik: 2020
• Tom: T. 21 (1)
• Strony: 25--41
• Opis fizyczny: Bibliogr. 20 poz., rys., tab.

Twórcy

autor

Wróbel Krzysztof

• Jagiellonian University, ul. Gołębia 24, 31-007 Cracow, Poland

autor

Karwatowski Michał

• AGH University of Science and Technology, al. Adama Mickiewicza 30, 30-059 Krakow, Poland

autor

Wielgosz Maciej

• AGH University of Science and Technology, al. Adama Mickiewicza 30, 30-059 Krakow, Poland

autor

Pietroń Marcin

• AGH University of Science and Technology, al. Adama Mickiewicza 30, 30-059 Krakow, Poland

autor

Wiatr Kazimierz

• AGH University of Science and Technology, al. Adama Mickiewicza 30, 30-059 Krakow, Poland

Bibliografia

• [1] Al-Hami P.K.M., Pietron M., Casas R., Hijazi S.: Towards a Stable Quantized Convolutional Neural Networks: An Embedded Perspective. In: ICAART 2018 : 10th International Conference on Agents and Artificial Intelligence : proceedings : Funchal, Madeira, Portugal : 16-18 January, 2018. 2018.
• [2] Cheng Y., Wang D., Zhou P., Zhang T.: Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges. In: IEEE Signal Processing Magazine, vol. 35(1), pp. 126-136, 2018. ISSN 1053-5888. https://doi.org/10.1109/MSP.2017.2765695.
• [3] Chollet F., et al.: Keras. https://keras.io, 2015.
• [4] Dozat T.: Incorporating Nesterov Momentum into Adam. In: International Conference on Learning Representations 2016. 2015. https://openreview.net/ forum?id=OM0jvwB8jIp57ZJjtNEZ.
• [5] Grachev A.M., Ignatov D.I., Savchenko A.V.: Neural Networks Compression for Language Modeling. In: B.U. Shankar, K. Ghosh, D.P. Mandal, S.S. Ray, D. Zhang, S.K. Pal, eds., Pattern Recognition and Machine Intelligence, pp. 351-357. Springer International Publishing, Cham, 2017. ISBN 978-3-319-69900-4.
• [6] Gysel P.: Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks. Master's thesis, University of California, 2016. http://arxiv.org/ abs/1605.06402.
• [7] Han S., Pool J., Tran J., Dally W.: Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems, pp. 1135-1143. 2015.
• [8] Hu M., Liu B.: Mining and Summarizing Customer Reviews. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '04, pp. 168-177. ACM, New York, NY, USA, 2004. ISBN 1-58113-888-1. https://doi.org/10.1145/1014052.1014073.
• [9] Kim Y.: Convolutional Neural Networks for Sentence Classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746-1751. Association for Computational Linguistics, Doha, Qatar, 2014. https://doi.org/10.3115/v1/D14-1181.
• [10] Laura J.A., Masi G., Argerich L.: From Imitation to Prediction, Data Compression vs Recurrent Neural Networks for Natural Language Processing, 2017. http://arxiv.org/abs/1705.00697.
• [11] Li X., Roth D.: Learning Question Classifiers. In: COLING, pp. 556-562. 2002.
• [12] Pang B., Lee L.: A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts. In: Proceedings of the 42Nd Annual Meeting on Association for Computational Linguistics, ACL '04. Association for Computational Linguistics, Stroudsburg, PA, USA, 2004. https: //doi.org/10.3115/1218955.1218990.
• [13] Pang B., Lee L.: Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL '05, pp. 115-124. Association for Computational Linguistics, Stroudsburg, PA, USA, 2005. https: //doi.org/10.3115/1219840.1219855.
• [14] Pennington J., Socher R., Manning C.: Glove: Global Vectors forWord Representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543. Association for Computational Linguistics, Doha, Qatar, 2014. https://doi.org/10.3115/v1/D14-1162.
• [15] Shu R., Nakayama H.: Compressing Word Embeddings via Deep Compositional Code Learning, 2017. http://arxiv.org/abs/1711.01068.
• [16] Socher R., Perelygin A., Wu J., Chuang J., Manning C.D., Ng A., Potts C.: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642. Association for Computational Linguistics, Seattle, Washington, USA, 2013. https://www.aclweb.org/anthology/ D13-1170.
• [17] Wiebe J., Wilson T., Cardie C.: Annotating Expressions of Opinions and Emotions in Language. In: Language Resources and Evaluation, vol. 39(2), pp. 165-210, 2005. ISSN 1572-0218. https://doi.org/10.1007/s10579-005-7880-9.
• [18] Wielgosz M., Karwatowski M.: Mapping Neural Networks to FPGA-Based IoT Devices for Ultra-Low Latency Processing. In: Sensors, vol. 19(13), 2019. https: //doi.org/10.3390/s19132981.
• [19] Wróbel K., Wielgosz M., Pietroń M., Duda J., Smywiński-Pohl A.: Improving text classification with vectors of reduced precision. In: ICAART 2018: 10th International Conference on Agents and Artificial Intelligence: proceedings: Funchal, Madeira, Portugal: 16-18 January, 2018. 2018.
• [20] Xu Y., Wang Y., Zhou A., Lin W., Xiong H.: Deep Neural Network Compression With Single and Multiple Level Quantization. In: AAAI18 32nd Conference on Artificial Intelligence Technical Track: Machine Learning Methods. 2018. https: //aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16479.