Fast Botnet Detection From Streaming Logs Using Online Lanczos Method
This work addresses real-time botnet detection for web security, offering incremental improvements in efficiency for streaming log analysis.
The paper tackled botnet detection from streaming web server logs by adapting the Lanczos method to reduce the time complexity of PCA-based detection from cubic to sub-cubic, enabling more accurate detection with sliding time windows; experiments on e-commerce logs showed the Lanczos method's time cost was only 20% to 25% of PCA.
Botnet, a group of coordinated bots, is becoming the main platform of malicious Internet activities like DDOS, click fraud, web scraping, spam/rumor distribution, etc. This paper focuses on design and experiment of a new approach for botnet detection from streaming web server logs, motivated by its wide applicability, real-time protection capability, ease of use and better security of sensitive data. Our algorithm is inspired by a Principal Component Analysis (PCA) to capture correlation in data, and we are first to recognize and adapt Lanczos method to improve the time complexity of PCA-based botnet detection from cubic to sub-cubic, which enables us to more accurately and sensitively detect botnets with sliding time windows rather than fixed time windows. We contribute a generalized online correlation matrix update formula, and a new termination condition for Lanczos iteration for our purpose based on error bound and non-decreasing eigenvalues of symmetric matrices. On our dataset of an ecommerce website logs, experiments show the time cost of Lanczos method with different time windows are consistently only 20% to 25% of PCA.