Showing 1–1 of 1 results for author: Lapastora, A

Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence

Authors: Abhinav Patil, Jaap Jumelet, Yu Ying Chiu, Andy Lapastora, Peter Shen, Lexie Wang, Clevis Willrich, Shane Steinert-Threlkeld

Abstract: This paper introduces Filtered Corpus Training, a method that trains language models (LMs) on corpora with certain linguistic constructions filtered out from the training data, and uses it to measure the ability of LMs to perform linguistic generalization on the basis of indirect evidence. We apply the method to both LSTM and Transformer LMs (of roughly comparable size), developing filtered corpor… ▽ More This paper introduces Filtered Corpus Training, a method that trains language models (LMs) on corpora with certain linguistic constructions filtered out from the training data, and uses it to measure the ability of LMs to perform linguistic generalization on the basis of indirect evidence. We apply the method to both LSTM and Transformer LMs (of roughly comparable size), developing filtered corpora that target a wide range of linguistic phenomena. Our results show that while transformers are better qua LMs (as measured by perplexity), both models perform equally and surprisingly well on linguistic generalization measures, suggesting that they are capable of generalizing from indirect evidence. △ Less

Submitted 6 August, 2024; v1 submitted 24 May, 2024; originally announced May 2024.

Comments: Forthcoming in Transactions of the Association for Computational Linguistics (TACL). This is a pre-MIT Press publication version. For code and trained models, see http://github.com/CLMBRs/corpus-filtering