Online eigenvector transformation reflecting concept drift for improving network intrusion detection

Abstract

Currently, large data streams are constantly being generated in diverse environments, and continuous storage of the data and periodic batch-type principal component analysis (PCA) are becoming increasingly difficult. Various online PCA algorithms have been proposed to solve this problem. In this study, we propose an online PCA methodology based on online eigenvector transformation with the moving average of the data stream that can reflect concept drift. We compared the network intrusion detection performance based on online transformation of eigenvectors with that of offline methods by applying three machine learning algorithms. Both online and offline methods demonstrated excellent performance in terms of precision. However, in terms of the recall ratio, the performance of the proposed methodology with integrated online eigenvector transformation was better; thus, the F1-measure also indicated better performance. The visualization of the principal component score shows the effectiveness of our method.