Layer-wise Semantic Role Labeling with the KR-BERT Language Model

Abstract

The purpose of this study is to assess the performance of semantic role labeling (SRL) predicted by the neural language models (NLMs, or Transformer-based pre-trained models) of Korean. First, the study built two models: the KR-BERT-BiLSTM-CRF model and the KR-BERT-Verb Position Feature (VPF)-BiLSTM-CRF model. The results from testing these two models show that the KR-BERT-VPF-BiLSTM-CRF model (67.3%) outperformed the KR-BERT-BiLSTM-CRF model (66.4%). In addition, this study examined which hidden layer improved the performance of NLMs during training. As expected, the NLM that was trained on the last hidden layer performed better than other alternative options such as the second-to-last-hidden layer and the concatenated last four layers. Thus, this study renders support to the general observation that an NLM should be trained on the last hidden layer to reach the highest performance. This study is meaningful since it is the first attempt to investigate which hidden layer is useful to train NLMs in SRL tasks of Korean.