Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings Using a Joint Loss Function

1. Attieh, J.: Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings using a Joint Loss Function. Master’s thesis, Aalto University. School of Science (2022). http://urn.fi/URN:NBN:fi:aalto-202209255727

2. Biś, D., Podkorytov, M., Liu, X.: Too much in common: Shifting of embeddings in transformer language models and its implications. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 5117–5130. Association for Computational Linguistics, Online (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.403, http://aclanthology.org/2021.naacl-main.403

3. Ethayarajh, K.: How contextual are contextualized word representations? comparing the geometry of BERT, ELMO, and GPT-2 embeddings. vol. abs/1909.00512 (2019). arXiv: abs/1909.00512

4. Gao, J., He, D., Tan, X., Qin, T., Wang, L., Liu, T.: Representation degeneration problem in training natural language generation models. In: International Conference on Learning Representations (2019). http://openreview.net/forum?id=SkEYojRqtm

5. Gong, C., He, D., Tan, X., Qin, T., Wang, L., Liu, T.Y.: Frage: Frequency-agnostic word representation. ArXiv arXiv:1809.06858 (2018)

6. Kalyan, K.S., Rajasekharan, A., Sangeetha, S.: Ammus : A survey of transformer-based pretrained models in natural language processing (2021). https://doi.org/10.48550/ARXIV.2108.05542,

7. Li, B., Zhou, H., He, J., Wang, M., Yang, Y., Li, L.: On the sentence embeddings from pre-trained language models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119–9130. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.733

8. Liang, Y., Cao, R., Zheng, J., Ren, J., Gao, L.: Learning to remove: Towards isotropic pre-trained Bert embedding. In: Artificial Neural Networks and Machine Learning - ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part V, p. 448–459. Springer-Verlag, Berlin, Heidelberg (2021). https://doi.org/10.1007/978-3-030-86383-8_36

9. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization (2017). https://doi.org/10.48550/ARXIV.1711.05101

10. Mu, J., Viswanath, P.: All-but-the-top: Simple and effective post-processing for word representations (2018), publisher Copyright: © Learning Representations, ICLR 2018 - Conference Track Proceedings. All right reserved.; 6th International Conference on Learning Representations, ICLR 2018; Conference date: 30–04-2018 Through 03–05-2018

    Google Scholar 

11. Rajaee, S., Pilehvar, M.T.: A cluster-based approach for improving isotropy in contextual embedding space. In: ACL (2021)

    Google Scholar 

12. Rajaee, S., Pilehvar, M.T.: How does fine-tuning affect the geometry of embedding space: a case study on isotropy. In: EMNLP (2021)

    Google Scholar 

13. Rudman, W., Gillman, N., Rayne, T., Eickhoff, C.: IsoScore: Measuring the uniformity of embedding space utilization. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 3325–3339. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.findings-acl.262, http://aclanthology.org/2022.findings-acl.262

14. Su, J., Cao, J., Liu, W., Ou, Y.: Whitening sentence representations for better semantics and faster retrieval (2021). https://doi.org/10.48550/ARXIV.2103.15316, http://arxiv.org/abs/2103.15316

15. Timkey, W., van Schijndel, M.: All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4527–4546. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.372, http://aclanthology.org/2021.emnlp-main.372

16. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.: GLUE: A multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353–355. Association for Computational Linguistics, Brussels, Belgium (Nov 2018). https://doi.org/10.18653/v1/W18-5446, http://aclanthology.org/W18-5446

17. Wang, L., Huang, J., Huang, K., Hu, Z., Wang, G., Gu, Q.: Improving neural language generation with spectrum control. In: International Conference on Learning Representations (2020). http://openreview.net/forum?id=ByxY8CNtvr

18. Wolf, T., et al.: Huggingface’s transformers: State-of-the-art natural language processing (2019). https://doi.org/10.48550/ARXIV.1910.03771

19. Zabihzadeh, D.: Ensemble of loss functions to improve generalizability of deep metric learning methods. arXiv:2107.01130 (2021)

20. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., Kumar, R.: Predicting the Type and Target of Offensive Posts in Social Media. In: Proceedings of NAACL (2019)

    Google Scholar 

21. Zhang, Z., Gao, C., Xu, C., Miao, R., Yang, Q., Shao, J.: Revisiting representation degeneration problem in language modeling. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 518–527. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.findings-emnlp.46, http://aclanthology.org/2020.findings-emnlp.46

22. Zhou, W., Lin, B.Y., Ren, X.: Isobn: Fine-tuning bert with isotropic batch normalization. Proc. AAAI Conf. Artif. Intell. 35(16), 14621–14629 (May 2021), http://ojs.aaai.org/index.php/AAAI/article/view/17718

Download references