Temporal Language Modeling for Short Text Document Classification with Transformers

• Autorzy: Pokrywka Jakub , Graliński Filip

Identyfikatory

DOI

10.15439/2022F174

• Języki publikacji: EN

Abstrakty

EN

Language models are typically trained on solely text data, not utilizing document timestamps, which are available in most internet corpora. In this paper, we examine the impact of incorporating timestamp into transformer language model in terms of downstream classification task and masked language modeling on 2 short texts corpora. We examine different timestamp components: day of the month, month, year, weekday. We test different methods of incorporating date into the model: prefixing date components into text input and adding trained date embeddings. Our study shows, that such a temporal language model performs better than a regular language model for both documents from training data time span and unseen time span. That holds true for classification and language modeling. Prefixing date components into text performs no worse than training special date components embeddings.

• Wydawca: Polskie Towarzystwo Informatyczne
• Czasopismo: Annals of Computer Science and Information Systems
• Rocznik: 2022
• Tom: Vol. 30
• Strony: 121--128
• Opis fizyczny: Bibliogr. 19 poz.

Twórcy

autor

Pokrywka Jakub

autor

Graliński Filip