Knowledge Distillation based Online Learning Methodology using Unlabeled Data Stream

Abstract

In supervised learning, the performance of the learning model decreases with the change of time step due to concept drift caused by overfitting of the training data. As a methodology to mitigate such concept drift, an online learning methodology has been proposed that trains the learning model on continuously input data stream. In this paper, we proposed an online learning methodology in which teacher model continuously trains student model based on knowledge distillation theory. The teacher model generates the output distribution called soft label to make a label for the unlabeled data stream and the student model trained by the unlabeled data stream with the soft label from teacher model. Experimental results show that the proposed method has better performances such as classification accuracy than that of the batch learning model trained by labeled data stream only.