Multimodal Sarcasm Detection via Hybrid Classifier with Optimistic Logic

• Autorzy: Bavkar Dnyaneshwar Madhukar , Kashyap Ramgopal , Khairnar Vaishali

Identyfikatory

DOI

10.26636/jtit.2022.161622

• Języki publikacji: EN

Abstrakty

EN

This work aims to provide a novel multimodal sarcasm detection model that includes four stages: pre-processing, feature extraction, feature level fusion, and classification. The pre-processing uses multimodal data that includes text, video, and audio. Here, text is pre-processed using tokenization and stemming, video is pre-processed during the face detection phase, and audio is pre-processed using the filtering technique. During the feature extraction stage, such text features as TF-IDF, improved bag of visual words, n-gram, and emojis as well on the video features using improved SLBT, and constraint local model (CLM) are extraction. Similarly the audio features like MFCC, chroma, spectral features, and jitter are extracted. Then, the extracted features are transferred to the feature level fusion stage, wherein an improved multilevel canonical correlation analysis (CCA) fusion technique is performed. The classification is performer using a hybrid classifier (HC), e.g. bidirectional gated recurrent unit (Bi-GRU) and LSTM. The outcomes of Bi-GRU and LSTM are averaged to obtain an effective output. To make the detection results more accurate, the weight of LSTM will be optimally tuned by the proposed opposition learning-based aquila optimization (OLAO) model. The MUStARD dataset is a multimodal video corpus used for automated sarcasm Discovery studies. Finally, the effectiveness of the proposed approach is proved based on various metrics.

Słowa kluczowe

EN

Bi-GRU; improved CCA; LSTM; multimodal sarcasm detection

• Wydawca: Instytut Łączności - Państwowy Instytut Badawczy
• Czasopismo: Journal of Telecommunications and Information Technology
• Rocznik: 2022
• Tom: nr 3
• Strony: 97--114
• Opis fizyczny: Bibliogr. 58 poz., rys., tab.

Twórcy

autor

Bavkar Dnyaneshwar Madhukar

• Department of Computer Science and Engineering, Amity University, Raipur, Chhattisgarh, India

• https://orcid.org/0000-0003-4746-0429

autor

Kashyap Ramgopal

• Department of Computer Science and Engineering, Amity University, Raipur, Chhattisgarh, India

• https://orcid.org/0000-0002-5352-1286

autor

Khairnar Vaishali

• Department of Information Technology, Terna Engineering College, Nerul, Navi Mumbai, India

• https://orcid.org/0000-0002-4867-1263

Bibliografia

• [1] K. Nimala, R. Jebakumar, and M. Saravanan, “Sentiment topic sarcasm mixture model to distinguish sarcasm prevalent topics based on the sentiment bearing words in the tweets”, Journal of Ambient Intelligence and Humanized, vol. 12, pp. 6801–6810, 2021 (DOI:10.1007/s12652-020-02315-1).
• [2] Y. Kumar and N. Goel, “AI-Based Learning Techniques for Sarcasm Detection of Social Media Tweets: State-of-the-Art Survey”, SN Comput. Sci., vol. 1, no. 6, 2020, (DOI: 10.1007/s42979-020-00336-3).
• [3] A. Banerjee, M. Bhattacharjee, K. Ghosh et al., “Synthetic minority oversampling in addressing imbalanced sarcasm detection in social media”, Multimed. Tools Appl., vol. 79, pp. 35995–36031, 2020 (DOI:10.1007/s11042-020-09138-4).
• [4] R. Justo, J.M. Alcaide, M.I. Torres et al., “Detection of Sarcasm and Nastiness: New Resources for Spanish Language”, Cogn. Comput., vol. 10, pp. 1135–1151, 2018 (DOI: 10.1007/s12559-018-9578-5).
• [5] R.A. Potamias, G. Siolas, and A. Stafylopatis “A transformerbased approach to irony and sarcasm detection”, Neural Comput. & Applic., vol. 32, pp. 17309–17320, 2020 (DOI:10.1007/s00521-020-05102-3).
• [6] Y. Du, T. Li, M.S. Pathan et al., “An Effective Sarcasm Detection Approach Based on Sentimental Context and Individual Expression Habits”, Cogn. Comput., vol. 14, pp. 78–90, 2021 (DOI:10.1007/s12559-021-09832-x).
• [7] L. Ren, B. Xua, H. Lin, X. Liu, and L. Yang, “Sarcasm Detection with Sentiment Semantics Enhanced Multi-level Memory Network”, Neurocomputing, vol. 401, pp. 320–326, 2020 (DOI:10.1016/j.neucom.2020.03.081).
• [8] M.S. Razali, A.A. Halin, L.S.Y. Doraisamy, and N.M. Norowi, “Sarcasm Detection Using Deep Learning With Contextual Features”, IEEE Access, vol. 9, pp. 68609–68618, 2021 (DOI: 10.1109/ACCESS.2021.3076789).
• [9] S. Rathod, “Hybrid Metaheuristic Algorithm for Cluster Head Selection in WSN”, Journal of Networking and Communication Systems, vol. 3, no. 4, 2020 (DOI:10.46253/jnacs.v3i4.a1).
• [10] N.S. Lakshmiprabha and S. Majumder, “Face recognition system invariant to plastic surgery”, 12th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 258–263, 2012 (DOI: 10.1109/ISDA.2012.6416547).
• [11] A. Onan and M.A. Toco˘glu, “A Term Weighted Neural Language Model and Stacked Bidirectional LSTM Based Framework for Sarcasm Identification”, IEEE Access, vol. 9, pp. 7701–7722, 2021 (DOI:10.1109/ACCESS.2021.3049734).
• [12] Meherkandukuri, “Deep Convolutional Neural Network for Emotion Recognition via EEG Signal”, Journal of Computational Mechanics, Power System and Control, vol. 4, no. 2, 2021 (DOI:10.46253/jcmps.v4i2.a3).
• [13] S. Rajeyyagari, “Automatic speaker diarization using deep LSTM in audio lecturing of e-Khool platform”, Journal of Networking and Communication Systems, vol. 3, no. 4, 2020 (DOI:10.46253/jnacs.v3i4.a3).
• [14] J. Russel Fernandis, “ALOA: Ant Lion Optimization Algorithmbased Deep Learning for Breast Cancer Classification”, Multimedia Research, vol. 4, no. 1, (DOI: 10.46253/j.mr.v4i1.a5).
• [15] C.I. Eke, A.A. Norman, and L. Shuib, “Context-Based Feature Technique for Sarcasm Identification in Benchmark Datasets Using Deep Learning and BERT Model”, IEEE Access, vol. 9, pp. 48501–48518, 2021 (DOI: 10.1109/ACCESS.2021.3068323).
• [16] Y. Diao, et al., “A Multi-Dimension Question Answering Network for Sarcasm Detection”, IEEE Access, vol. 8, pp. 135152–135161, 2020 (DOI:10.1109/ACCESS.2020.2967095).
• [17] A. Kumar, V.T. Narapareddy, V. Aditya Srikanth, A. Malapati, and L.B.M. Neti, “Sarcasm Detection Using Multi-Head Attention Based Bidirectional LSTM”, IEEE Access, vol. 8, pp. 6388–6397, 2020 (DOI: 10.1109/ACCESS.2019.2963630).
• [18] Y. Zhang et al., “CFN: A Complex-Valued Fuzzy Network for Sarcasm Detection in Conversations”, IEEE Transactions on Fuzzy Systems, vol. 29, no. 12, pp. 3696–3710, 2021 (DOI:10.1109/TFUZZ.2021.3072492).
• [19] K. Rothermich, A. Ogunlana, and N. Jaworska, “Change in humor and sarcasm use based on anxiety and depression symptom severity during the COVID-19 pandemic”, Journal of Psychiatric Research, vol. 140, pp. 95–100, 2021 (DOI: 10.1016/j.jpsychires.2021.05.027).
• [20] P. Parameswaran, A. Trotman, and D. Eyers, “Detecting the target of sarcasm is hard: Really?”, Information Processing and Management, vol. 58, no. 4, 2021 (DOI: 10.1016/j.ipm.2021.102599).
• [21] N.Z.Z. Wang, “The paradox of sarcasm: Theory of mind and sarcasm use in adults”, Personality and Individual Differences, vol. 163, 2020 (DOI: 10.1016/j.paid.2020.110035).
• [22] R. Pandey, A. Kumar, J.P. Singh, and S. Tripathi, “Hybrid attention-based Long Short-Term Memory network for sarcasm identification”, Applied Soft Computing, vol. 106, 2021 (DOI:10.1016/j.asoc.2021.107348).
• [23] N. Basavaraj Hiremath, and M.M. Patil, “Sarcasm Detection using Cognitive Features of Visual Data by Learning Model”, Expert Systems with Applications, vol. 184, 2021 (DOI:10.1016/j.eswa.2021.115476).
• [24] D. Jain, A. Kumar, and G. Garg, “Sarcasm detection in mash- up language using soft-attention based bi-directional LSTM and feature-rich CNN”, Applied Soft Computing, vol. 91, 2020 (DOI: 10.1016/j.asoc.2020.106198).
• [25] Y. Wu et al., “Modeling Incongruity between Modalities for Multimodal Sarcasm Detection”, IEEE MultiMedia, vol. 28, no. 2, pp. 86–95, 2021, (DOI: 10.1109/MMUL.2021.3069097).
• [26] A. Kamal and M. Abulaish “CAT-BiGRU: Convolution and Attention with Bi-Directional Gated Recurrent Unit for Self-Deprecating Sarcasm Detection”, Cogn. Comput., vol. 14, pp. 91–109, 2022 (DOI:10.1007/s12559-021-09821-0).
• [27] C.I. Eke, A.A. Norman, S. Liyana, and H.F. Nweke, “Sarcasm identification in textual data: systematic review, research challenges and open directions”, Artif. Intell. Rev., vol. 53, pp. 4215–4258, 2020 (DOI: 10.1007/s10462-019-09791-8).
• [28] A. Kumar and G. Garg, “Empirical study of shallow and deep learning models for sarcasm detection using context in benchmark datasets”, Journal of Ambient Intelligence and Humanized Computing, 2019 (DOI: 10.1007/s12652-019-01419-7).
• [29] L. Ren, H. Lin, B. Xu, et al., “Learning to capture contrast In sarcasm with contextual dual-view attention network”, Int. J. Mach. Learn. and Cyber. vol. 12, pp. 2607–2615, 2021 (DOI:10.1007/s13042-021-01344-2).
• [30] Z.L. Chia, M. Ptaszyński, and M. Wroczyński, “Machine Learning and feature engineering-based study into sarcasm and irony classification with application to cyberbullying detection”, Information Processing and Management, vol. 58, no. 4, 2021, (DOI:10.1016/j.ipm.2021.102600).
• [31] A.F. Hidayatullah and M.R. Ma’arif, “Pre-processing Tasks in Indonesian Twitter Messages”, Journal of Physics: Conference Series, vol. 801, 2017 (DOI: 10.1088/1742-6596/801/1/012072).
• [32] N. Hazim Barnouti, et al., “Face Detection and Recognition Using Viola-Jones with PCA-LDA and Square Euclidean Distance”, International Journal of Advanced Computer Science and Applications, vol. 7, no. 5, 2016 (DOI: 10.14569/IJACSA.2016.070550).
• [33] H. Pandey and R. Tiwari, “An Innovative Design Approach of Butterworth Filter for Noise Reduction in ECG Signal Processing based Applications”, Progress In Science in Engineering Research Journal PISER 12, vol. 2, pp. 332–337, 2014.
• [34] D. Kim, D. Seo, S. Cho, and P. Kang, “Multi-co-training for dokument classification using various document representations: TF–IDF, LDA, and Doc2Vec”, Information Sciences, vol. 477, pp. 15–29, 2019 (DOI: 10.1016/j.ins.2018.10.006).
• [35] C. Cheng, L. Chunping, H. Yan, and Y. Zhu, “A semi-supervised deep learning image caption model based on Pseudo Label and N-gram”, International Journal of Approximate Reasoning, vol. 131, pp. 93–107, 2021 (DOI: 10.1016/j.ijar.2020.12.016).
• [36] D. Cristinacce and T. Cootes, “Automatic feature localisation with constrained local models”, Pattern Recognition, vol. 41, no. 10, pp. 3054–3067, 2008 (DOI: 10.1016/j.patcog.2008.01.024).
• [37] O.C. Ai, M. Hariharan, S. Yaacob, and L.S. Chee, “Classification of speech dysfluencies with MFCC and LPCC features”, Expert Systems with Applications, vol. 39, no. 2, pp. 2157–2165, 2012 (DOI:10.1016/j.eswa.2011.07.065).
• [38] T. Kronvall, M. Juhlin, J. Sward, S.I. Adalbjornsson, and A. Jakobsson, “Sparse modeling of chroma features”, Signal Processing, vol. 130, pp. 105–117, 2017 (DOI: 10.1016/j.sigpro.2016.06.020).
• [39] M. Kavitha, R. Gayathri, K. Polat, A. Alhudhaif, and F. Alenezi, “Performance evaluation of deep e-CNN with integrated spatial-spectral features in hyperspectral image classification”, Measurement, vol. 191, 2022 (DOI: 10.1016/j.measurement.2022.110760).
• [40] L. An, et al., “Multi-Level Canonical Correlation Analysis for Standard-Dose PET Image Estimation”, IEEE Transactions on Image Processing, vol. 25, no. 7, pp. 3303–3315, 2016 (DOI:10.1109/TIP.2016.2567072).
• [41] X. Zhou, J. Lin, Z. Zhang, Z. Shao, and H. Liu, “Improved itracker combined with bidirectional long short-term memory for 3D gaze estimation using appearance cues”, Neurocomputing In Press, vol. 390, pp. 217–25, 2019 (DOI: 10.1016/j.neucom.2019.04.099).
• [42] D. Zhao, J. Wang, and Y. Zhang, “Extracting drug–drug interactions with hybrid bidirectional gated recurrent unit and graph convolutional network”, Journal of Biomedical Informatics, vol. 99, 2019 (DOI: 10.1016/j.jbi.2019.103295).
• [43] L. Abualigah, et al., “Aquila Optimizer: A novel meta-heuristic optimization algorithm”, Computers & Industrial Engineering, vol. 157, 2021 (DOI: 10.1016/j.cie.2021.107250).
• [44] B.R. Rajakumar, “Impact of Static and Adaptive Mutation Techniques on Genetic Algorithm”, International Journal of Hybrid Intelligent Systems, vol. 10, no. 1, pp. 11–22, 2013 (DOI: 10.3233/HIS-120161).
• [45] B.R. Rajakumar, “Static and Adaptive Mutation Techniques for Genetic algorithm: A Systematic Comparative Analysis”, International Journal of Computational Science and Engineering, vol. 8, no. 2, pp. 180–193, 2013 (DOI: 10.1504/IJCSE.2013.053087).
• [46] S.M. Swamy, B.R. Rajakumar, and I.R. Valarmathi, “Design of Hybrid Wind and Photovoltaic Power System using Opposition-based Genetic Algorithm with Cauchy Mutation”, IET Chennai Fourth International Conference on Sustainable Energy and Intelligent Systems (SEISCON 2013), 2013 (DOI: 10.1049/ic.2013.0361).
• [47] A. George and B.R. Rajakumar, “APOGA: An Adaptive Population Pool Size based Genetic Algorithm”, AASRI Procedia, vol. 4, pp. 288–296, 2013 (DOI: 10.1016/j.aasri.2013.10.043).
• [48] B.R. Rajakumar and A. George, “A New Adaptive Mutation Technique for Genetic Algorithm”, In proceedings of IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), pp. 1–7, 2012, (DOI: 10.1109/ICCIC.2012.6510293).
• [49] F. Chakraborty, P.K. Roy, and D. Nandi, “Oppositional Elephant herding optimization with dynamic Cauchy mutation for multilevel image thresholding”, Evol. Intel. 12, pp. 445–467, 2019 (DOI:10.1007/s12065-019-00238-1).
• [50] S.H.S. Moosavi and V.K. Bardsiri, “Poor and rich optimization algorithm: A new human-based and multi populations algorithm”, Engineering Applications of Artificial Intelligence, vol. 86, pp. 165–181, 2019 (DOI: 10.1016/j.engappai.2019.08.025).
• [51] F. Ahmed, “Social Spider Optimization Algorithm”, 2015 (DOI:10.13140/RG.2.1.4314.5361).
• [52] M. Dehghani, Š. Hubalovsky, and P, Trojovsky, “Cat and Mouse Based Optimizer: A New Nature-Inspired Optimization Algorithm”, Sensors, vol. 21, no. 15, 2021 (DOI: 10.3390/s21155214).
• [53] M.O. Okwu and L.K. Tartibu, “Ant Lion Optimization Algorithm”, Metaheuristic Optimization: Nature-Inspired Algorithms Swarm and Computational Intelligence, Theory and Applications. Studies in Computational Intelligence, vol. 929, 2020 (DOI:10.1007/978-3-030-61111-8_9).
• [54] Y. LeCun, K. Kavukvuoglu, and C. Farabet, “Convolutional networks and applications in vision”, Circuits and Systems, International Symposium on, pp. 253–256, 2010 (DOI: 10.1109/ISCAS.2010.5537907).
• [55] K. Ling-Jing and C.C. Chiu, “Application of integrated recurrent neural network with multivariate adaptive regression splines on SPC-EPC process”, Journal of Manufacturing Systems, vol. 57, pp. 109–118, 2020 (DOI:10.1016/j.jmsy.2020.07.020).
• [56] Z. Masetic and A. Subasi, “Congestive heart failure detection using random forest classifier”, Computer Methods and Programs in Biomedicine, vol. 130, pp. 54–64, July 2016 (DOI:10.1016/j.cmpb.2016.03.020).
• [57] P.T. Ilia, “Comparison of a logistic regression and Naive Bayes classifier in landslide susceptibility assessments: The influence of models complexity and training dataset size “, Catena, vol. 145, pp. 164–179, 2016 (DOI:10.1016/j.catena.2016.06.004).
• [58] https://github.com/soujanyaporia/MUStARD.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).