Tytuł artykułu
Wybrane pełne teksty z tego czasopisma
Identyfikatory
Warianty tytułu
Konferencja
Federated Conference on Computer Science and Information Systems (19 ; 08-11.09.2024 ; Belgrade, Serbia)
Języki publikacji
Abstrakty
In this paper, we investigated the impact of spelling and editing correctness on the accuracy of detection if an email was written by a human or if it was generated by a language model. As a dataset, we used a combination of publicly available email datasets with our in-house data, with over 10k emails in total. Then, we generated their “copies'' using large language models (LLMs) with specific prompts. As a classifier, we used random forest, which yielded the best results in previous experiments. For English emails, we found a slight decrease in evaluation metrics if error-related features were excluded. However, for the Polish emails, the differences were more significant, indicating a decline in prediction quality by around 2% relative. The results suggest that the proposed detection method can be equally effective for English even if spelling- and grammar-checking tools are used. As for Polish, to compensate for error-related features, additional measures have to be undertaken.
Rocznik
Tom
Strony
603--608
Opis fizyczny
Bibliogr. 21 poz., il., tab., wykr.
Twórcy
autor
- Warsaw University of Technology, ul. Nowowiejska 15/19, 00-665 Warsaw, Poland
autor
- Warsaw University of Technology, ul. Nowowiejska 15/19, 00-665 Warsaw, Poland
autor
- Warsaw University of Technology, ul. Nowowiejska 15/19, 00-665 Warsaw, Poland
autor
- Warsaw University of Technology, ul. Nowowiejska 15/19, 00-665 Warsaw, Poland
autor
- Warsaw University of Technology, ul. Nowowiejska 15/19, 00-665 Warsaw, Poland
Bibliografia
- 1. Europol, “Chatgpt: the impact of large language models on law enforcement,” 2023. http://dx.doi.org/10.2813/255453
- 2. H. Williams and C. McCulloch, “Truth decay and national security: Intersections, insights, and questions for future research,” Santa Monica, CA, USA, 2023. [Online]. Available: https://www.rand.org/pubs/perspectives/PEA112-2.html
- 3. K. T. Gradoń, “Generative artificial intelligence and medical disinformation,” British Medical Journal, no. 384, 2024. http://dx.doi.org/10.1136/bmj.q579
- 4. P. Gryka, K. Gradoń, M. Kozłowski, M. Kutyła, and A. Janicki, “Detection of AI-generated emails – a case study,” in Proc. 13th International Workshop on Cyber Crime (IWCC 2024), Vienna, Austria, 2024, (accepted for publication).
- 5. A. Knott, D. Pedreschi, R. Chatila, T. Chakraborti, S. Leavy, R. Baeza-Yates, D. Eyers, A. Trotman, P. D. Teal, P. Biecek, S. Russell, and Y. Bengio, “Generative AI models should include detection mechanisms as a condition for public release,” Ethics and Information Technology, vol. 25, no. 4, p. 55, 12 2023. http://dx.doi.org/10.1007/s10676-023-09728-4
- 6. K. Krishna, Y. Song, M. Karpinska, J. Wieting, and M. Iyyer, “Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense,” Advances in Neural Information Processing Systems, vol. 36, 3 2024.
- 7. F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker, “Perplexity—a measure of the difficulty of speech recognition tasks,” The Journal of the Acoustical Society of America, vol. 62, no. S1, pp. S63–S63, 1977. http://dx.doi.org/10.1121/1.2016299
- 8. M. Chakraborty, S. T. I. Tonmoy, S. M. M. Zaman, S. Gautam, T. Kumar, K. Sharma, N. Barman, C. Gupta, V. Jain, A. Chadha, A. Sheth, and A. Das, “Counter Turing test (CT2): AI-generated text detection is not as easy as you may think - introducing AI detectability index (ADI),” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023. http://dx.doi.org/10.18653/v1/2023.emnlp-main.136 pp. 2206–2239. [Online]. Available: https://aclanthology.org/2023.emnlp-main.136
- 9. I. Cingillioglu, “Detecting AI-generated essays: the ChatGPT challenge,” International Journal of Information and Learning Technology, vol. 40, pp. 259–268, 5 2023. http://dx.doi.org/10.1108/IJILT-03-2023-0043
- 10. L. Fröhling and A. Zubiaga, “Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover,” PeerJ Computer Science, vol. 7, p. e443, 4 2021. http://dx.doi.org/10.7717/peerj-cs.443
- 11. Y. Shi, Q. Sheng, J. Cao, H. Mi, B. Hu, and D. Wang, “Ten words only still help: Improving black-box AI-generated text detection via proxy-guided efficient re-sampling,” arXiv preprint, vol. https://arxiv.org/abs/2402.09199, 2024. [Online]. Available: http://arxiv.org/abs/2402.09199
- 12. E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, and C. Finn, “DetectGPT: Zero-shot machine-generated text detection using probability curvature,” in Proc. International Conference on Machine Learning. Online: PMLR, 2023, pp. 24 950–24 962.
- 13. F. Harrag, M. Dabbah, K. Darwish, and A. Abdelali, “Bert transformer model for detecting Arabic GPT2 auto-generated tweets,” in Proceedings of the Fifth Arabic Natural Language Processing Workshop, I. Zitouni, M. Abdul-Mageed, H. Bouamor, F. Bougares, M. El-Haj, N. Tomeh, and W. Zaghouani, Eds. Barcelona, Spain (Online): Association for Computational Linguistics, Dec. 2020, pp. 207–214. [Online]. Available: https://aclanthology.org/2020.wanlp-1.19
- 14. J. D. Rodriguez, T. Hay, D. Gros, Z. Shamsi, and R. Srinivasan, “Cross-domain detection of GPT-2-generated technical text,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 2022.naacl-main.88. Association for Computational Linguistics, 2022. http://dx.doi.org/10.18653/v1/2022.naacl-main.88 pp. 1213–1233.
- 15. S. Mukherjee, “Exploring burstiness: Evaluating language dynamics in LLM-generated texts,” 2023, [Online]. Available: https://ramblersm.medium.com/exploring-burstiness-evaluating-language-dynamics-in-llm-generated-texts-8439204c75c1 (Accessed on Apr 30, 2024).
- 16. I. Okulska, D. Stetsenko, A. Kołos, A. Karlińska, K. Głąbińska, and A. Nowakowski, “Stylometrix: An open-source multilingual tool for representing stylometric vectors,” arXiv preprint https://arxiv.org/abs/2309.12810, vol. 2309.12810, 9 2023.
- 17. J. Morris, “LanguageTool Python library,” 2024, https://pypi.org/project/language-tool-python/ (Accessed on May 10, 2024). [Online]. Available: https://pypi.org/project/language-tool-python/
- 18. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
- 19. _w1998, “Spam email dataset,” 2023, (Accessed on Jan 14, 2024). [Online]. Available: https://www.kaggle.com/datasets/jackksoncsie/spam-email-dataset/data
- 20. R. Modi, “Email classification dataset,” 2023, (Accessed on Jan 14, 2024). [Online]. Available: https://github.com/rmodi6/Email-Classification/tree/master
- 21. Apache Public Datasets, “The Spam Assassin Email Classification Dataset,” 2023, (Accessed on Jan 14, 2024). [Online]. Available: https://www.kaggle.com/datasets/ganiyuolalekan/spam-assassin-email-classification-dataset/data
Uwagi
Thematic Sessions: Short Papers
Typ dokumentu
Bibliografia
Identyfikator YADDA
bwmeta1.element.baztech-836272a0-6838-46c8-8da8-ee2f3e56e699
JavaScript jest wyłączony w Twojej przeglądarce internetowej. Włącz go, a następnie odśwież stronę, aby móc w pełni z niej korzystać.