Hierarchical Inductive Bias in the L2 Textbook-T5 and Child-T5 Language Model: A Study of Data and Architecture

Abstract

This study aims to investigate neural language models in alignment with Chomsky's (1965, 1980) proposition regarding the innate human tendency to acquire syntactic rules based on hierarchical structures rather than linear order. We have examined the architectural and dataset factors influencing the acquisition of a syntactic inductive bias during the pre-training of the T5 model. To achieve this objective, particularly concerning the pre-training dataset, we inquire whether there are differences when using datasets of varying complexity. We use two distinct pre-training datasets: Child-Directed Speech and L2-Textbook dataset. Upon examination, we observe that these datasets exhibit different levels of syntactic complexity. Then, with our models, we employ two syntactic transformation tasks: (i) question formation and (ii) passivization. Our results indicate that model depth (number of layers) exerts a more significant influence on hierarchical generalization compared to other model components. Furthermore, we observed that models learn the hierarchical aspects of language more efficiently when exposed to the simpler complexity of the dataset.