Locality-Sensitive Hashing for Efficient Web Application Security Testing
This addresses a key bottleneck in web application security testing for developers and testers, though it is an incremental improvement over existing methods.
The paper tackled the state similarity problem in crawling Rich Internet Applications for security testing by using locality-sensitive hashing with MinHash sketches to estimate DOM similarity, enabling successful scans of previously uncrawlable RIAs.
Web application security has become a major concern in recent years, as more and more content and services are available online. A useful method for identifying security vulnerabilities is black-box testing, which relies on an automated crawling of web applications. However, crawling Rich Internet Applications (RIAs) is a very challenging task. One of the key obstacles crawlers face is the state similarity problem: how to determine if two client-side states are equivalent. As current methods do not completely solve this problem, a successful scan of many real-world RIAs is still not possible. We present a novel approach to detect redundant content for security testing purposes. The algorithm applies locality-sensitive hashing using MinHash sketches in order to analyze the Document Object Model (DOM) structure of web pages, and to efficiently estimate similarity between them. Our experimental results show that this approach allows a successful scan of RIAs that cannot be crawled otherwise.