An Empirical Study of Fuzz Harness Degradation
For developers and maintainers of continuous fuzzing platforms, this work provides empirical evidence that harnesses are surprisingly resilient but also identifies root causes of degradation, guiding detection and prevention efforts.
This study investigates whether fuzz harnesses in Google's OSS-Fuzz platform degrade over time as the underlying projects evolve. Analyzing 510 open-source C/C++ projects, they found that overall coverage remains consistent and bug-finding capability persists as long as harnesses build, though they identified and categorized specific cases of degradation.
The purpose of continuous fuzzing platforms is to enable fuzzing for software projects via fuzz harnesses -- but as the projects continue to evolve, are these harnesses updated in lockstep, or do they run out of date? If these harnesses remain unmaintained, will they degrade over time in terms of coverage achieved or number of bugs found? This is the subject of our study. We study Google's OSS-Fuzz continuous fuzzing platform containing harnesses for 510 open-source C/C++ projects, many of which are security-critical. A harness is the glue code between the fuzzer and the project, so it needs to adapt to changes in the project. It is often added by a project maintainer or as part of a, sometimes short-lived, testing effort. Our analysis shows a consistent overall fuzzer coverage percentage for projects in OSS-Fuzz and a surprising longevity of the bug-finding capability of harnesses even without explicit updates, as long as they still build. However, we also identify and manually examine individual cases of harness coverage degradation and categorize their root causes. Furthermore, we contribute to OSS-Fuzz and Fuzz Introspector to support metrics to detect harness degradation in OSS-Fuzz projects guided by this research.