Wyniki wyszukiwania - BazTech

Ograniczanie wyników

1 Annals of Computer Science and Information Systems

1 2022

Znaleziono wyników: 1

Liczba wyników na stronie

Wyniki wyszukiwania

Sortuj według:

Ogranicz wyniki do:

Applying SoftTriple loss for supervised language model fine tuning

Sosnowski Witold, Wróblewska Anna, Gawrysiak Piotr

Annals of Computer Science and Information Systems

2022

Vol. 30

141--147

We introduce a new loss function based on cross entropy and SoftTriple loss, TripleEntropy, to improve classification performance for fine-tuning general knowledge pre-trained language models. This loss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss by about 0.02 - 2.29 percentage points. Thorough tests on popular datasets using our loss function indicate a steady gain. The fewer samples in the training dataset, the higher gain - thus, for small-sized dataset, it is about 0.71 percentage points, for medium-sized - 0.86 percentage points, for large - 0.20 percentage points, and for extra-large 0.04 percentage points.