Reinforcement learning guided fuzz testing for a browser's HTML rendering engine
This addresses the problem of time-consuming test case generation for software security testing, specifically for browser rendering engines, though it appears incremental as it builds on existing fuzzing and reinforcement learning techniques.
The paper tackled the challenge of developing efficient generation-based fuzz testing for browser HTML rendering engines by combining a deep learning test case generator with a double deep Q-network guided by code coverage, resulting in up to 18.5% improved code coverage for Firefox compared to baseline grammar-based fuzzing.
Generation-based fuzz testing can uncover various bugs and security vulnerabilities. However, compared to mutation-based fuzz testing, it takes much longer to develop a well-balanced generator that produces good test cases and decides where to break the underlying structure to exercise new code paths. We propose a novel approach to combine a trained test case generator deep learning model with a double deep Q-network (DDQN) for the first time. The DDQN guides test case creation based on a code coverage signal. Our approach improves the code coverage performance of the underlying generator model by up to 18.5\% for the Firefox HTML rendering engine compared to the baseline grammar based fuzzer.