Adversarial representation teaching with perturbation-agnostic student-teacher structure for semi-supervised learning

Abstract

Consistency regularization (CR) is representative semi-supervised learning (SSL) technique that maintains the consistency of predictions from multiple views on the same unlabeled data during the training. In recent SSL studies, the approaches of self-supervised learning with CR, which conducts the pre-training based on unsupervised learning and fine-tuning based on supervised learning, have provided excellent classification accuracy. However, the data augmentation used to generate multiple views in CR has a limitation for expanding the training data distribution. In addition, the existing self-supervised learning using CR cannot provide the high-density clustering result for each class of the labeled data in a representation space, thus it is vulnerable to outlier samples of the unlabeled data with strong augmentation. Consequently, the unlabeled data with augmentation for SSL may not improve the classification performance but rather degrade it. To solve these, we propose a new training methodology called adversarial representation teaching (ART), which consists of the labeled sample-guided representation teaching and adversarial noise-based CR. In our method, the adversarial attack-robust teacher model guides the student model to form a high-density distribution in representation space. This allows for maximizing the improvement by the strong embedding augmentation in the student model for SSL. For the embedding augmentation, the adversarial noise attack on the representation is proposed to successfully expand a class-wise subspace, which cannot be achieved by the existing adversarial attack or embedding expansion. Experimental results showed that the proposed method provided outstanding classification accuracy up to 1.57% compared to the existing state-of-the-art methods under SSL conditions. Moreover, ART significantly outperforms the classification accuracies up to 1.57%, 0.53%, and 0.3% over our baseline method on the CIFAR-10, SVHN, and ImageNet datasets, respectively.