AI based Log Analyser: A Practical Approach
This work addresses a practical problem in system and cyber resilience for IT professionals, but it appears incremental as it builds on existing AI methods for log analysis.
The paper tackles the challenge of analyzing heterogeneous logs with limited labeled data by using a Transformer model trained on normal log entries, augmented through perturbations and fine-tuned with reinforcement learning on few labels, showing promising results in comparative evaluations.
The analysis of logs is a vital activity undertaken for fault or cyber incident detection, investigation and technical forensics analysis for system and cyber resilience. The potential application of AI algorithms for Log analysis could augment such complex and laborious tasks. However, such solution has its constraints the heterogeneity of log sources and limited to no labels for training a classifier. When such labels become available, the need for the classifier to be updated. This practice-based research seeks to address these challenges with the use of Transformer construct to train a new model with only normal log entries. Log augmentation through multiple forms of perturbation is applied as a form of self-supervised training for feature learning. The model is further finetuned using a form of reinforcement learning with a limited set of label samples to mimic real-world situation with the availability of labels. The experimental results of our model construct show promise with comparative evaluation measurements paving the way for future practical applications.