신경망 언어 모델의 내부 표상: 탐침 분류기 기법을 중심으로

Abstract

This paper appraises the validity of using a probing classifier that serves as the most rigorous method in investigating the internal representation of a deep neural-network language model. Recently, such a model has been reported to display a high performance in undertaking various linguistic tasks. However, it is not easy to assess what kind of linguistic knowledge it acquires, and how robustly such knowledge is encoded in internal artificial neural networks. A probing classifier has been developed as a method to analyze the internal mechanism of such a model. We first evaluate the validity of this method in three respects by taking into account such control factors as task, function, and dataset that can cause defects when applying a probing classifier. Second, we discuss what kind of probing classifier should be applied, simple versus complex, and ponder on other alternative methods aside from using a probing classifier. Third, we consider the issues of correlation and causation in studying the relationship between probed linguistic properties and an original language model.