Assessing the Structural Profiles of L2-textbook Dataset on Transformer LMs

Abstract

Transformer-based language models (TLMs) employing multi-head self-attention methods have led to substantial enhancements in performance across diverse domains in Natural Language Processing (NLP). While current TLMs have demonstrated impressive capabilities by training on datasets hundreds of times larger than those akin to children's learning data, BabyBERTa has achieved meaningful performance by leveraging developmentally plausible datasets of comparable size. This study delves into the detailed evaluation of BabyBERTa's performance, aiming to gain insights into TLMs' language acquisition abilities and explore the feasibility of utilizing second language textbook dataset. Our analysis indicates that the dataset encompassing sentences with varied structures can effectively facilitate TLMs in acquiring grammatical knowledge.