Delog: A Privacy Preserving Log Filtering Framework for Online Compute Platforms
This work addresses the problem of efficient and private log analysis for developers using online compute platforms, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the challenge of debugging long-running applications by filtering logs to show only relevant errors, using a privacy-preserving framework for PaaS providers and a distributed log parsing algorithm based on Locality Sensitive Hashing, achieving state-of-the-art performance on multiple datasets and scaling to 211 million lines with 27,000 unique patterns.
In many software applications, logs serve as the only interface between the application and the developer. However, navigating through the logs of long-running applications is often challenging. Logs from previously successful application runs can be leveraged to automatically identify errors and provide users with only the logs that are relevant to the debugging process. We describe a privacy preserving framework which can be employed by Platform as a Service (PaaS) providers to utilize the user logs generated on the platform while protecting the potentially sensitive logged data. Further, in order to accurately and scalably parse log lines, we present a distributed log parsing algorithm which leverages Locality Sensitive Hashing (LSH). We outperform the state-of-the-art on multiple datasets. We further demonstrate the scalability of Delog on publicly available Thunderbird log dataset with close to 27,000 unique patterns and 211 million lines.