Research on log anomaly detection method based on autoencoder
Yang Guang1, Lei Yufang2, Wang Peng2, Sun Qiang2, Yan Kaixin1, Zhu Yan1, Pan Haolong1, Wang Xuren3
1. Institute of Information Engineering, Chinese Academy of Sciences;2. SinoRail (Beijing) Information Technology Service Co., Ltd.;3. Capital Normal University
Abstract： System logs contain key operational information and problem clues. However, as the system scale expands, log data becomes increasingly large and complex, making automated anomaly detection a research focus. Current studies face challenges such as imbalanced log data and insufficient labeled data, which lead to low detection accuracy. To address these challenges, a log anomaly detection method based on MultiWindow Long ShortTerm Memory (LSTM) Autoencoder is proposed, focusing on three aspects: log data processing, autoencoder model, and log event classification. This method combines the advantages of LSTM and autoencoder, and uses a multiwindow strategy to capture contextual information at different time scales, providing a more effective anomaly detection solution for timesensitive logs. Experimental results show that this method achieves high F1scores on two public datasets, Hadoop Distributed File System (HDFS) and Blue Gene/L (BGL), and exhibits better anomaly detection performance compared with other methods.
Key words : system logs; anomaly detection; deep learning; autoencoder
