Adrian Herrera

Alvin Charles, Adrian Herrera, Peter Oslington et al.

Binary-only fuzzing is a key technique for finding bugs in close-source software. Without access to source code, the fuzzer must rely on static or dynamic binary instrumentation for coverage guidance. In practice, most fuzzers favor dynamic binary instrumentation (DBI), accepting runtime overhead to avoid the perceived accuracy and soundness challenges associated with static binary instrumentation (SBI). We show that these concerns are unwarranted, and that accurate, scalable~SBI is achievable using off-the-shelf frameworks. Building on these frameworks, we develop PeAR, an extensible binary-only fuzzing framework. We demonstrate PeAR's versatility by implementing several modern fuzzer features -- including, deferred initialization, persistent mode, and shared-memory fuzzing. We evaluate PeAR over 4.25 CPU-yrs of fuzzing on the FUZZBENCH benchmark and find that PeAR: (i) successfully instruments 88% of FUZZBENCH targets, comparable to the best SBI-based fuzzers; (ii) achieves a median throughput improvement of 4x when using persistent mode and shared memory fuzzing; and (iii) attains coverage comparable to compiler-based instrumentation. Our results show that SBI is a practical and effective technique for binary-only fuzzing, and that modern binary rewriting frameworks can apply complex instrumentation with high granularity and negligible performance compromise.

Adrian Herrera, Hendra Gunadi, Liam Hayes et al.

Mutation-based fuzzing typically uses an initial set of non-crashing seed inputs (a corpus) from which to generate new inputs by mutation. A corpus of potential seeds will often contain thousands of similar inputs. This lack of diversity can lead to wasted fuzzing effort by exhaustive mutation from all available seeds. To address this, fuzzers come with distillation tools (e.g., afl-cmin) that select the smallest subset of seeds that triggers the same range of instrumentation data points as the full corpus. Common practice suggests that minimizing the number and cumulative size of the seeds leads to more efficient fuzzing, which we explore systematically. We present results of 34+ CPU-years of fuzzing with five distillation approaches to understand their impact in finding bugs in real-world software. We evaluate a number of techniques, includibng the existing afl-cmin and Minset, and also MoonLight---a freely available, configurable, state-of-the-art, open-source, tool. Our experiments compare the effectiveness of distillation approaches, targeting the Google Fuzzer Test Suite and a diverse set of six real-world libraries and programs, covering 13 different input file formats across 16 programs. Our results show that distillation is a necessary precursor to any fuzzing campaign when starting with a large initial corpus. We compare the effectiveness of alternative distillation approaches. Notably, our experiments reveal that state-of-the-art distillation tools (such as MoonLight and Minset) do not exclusively find all of the 33 bugs (in the real-world targets) exposed by our combined campaign: each technique appears to have its own strengths. We find (and report) new bugs with MoonLight that are not found by Minset, and vice versa. Moreover, afl-cmin fails to reveal many of these bugs. Of the 33 bugs revealed in our campaign, seven new bugs have received CVEs.

Adrian Herrera

JavaScript is a popular attack vector for releasing malicious payloads on unsuspecting Internet users. Authors of this malicious JavaScript often employ numerous obfuscation techniques in order to prevent the automatic detection by antivirus and hinder manual analysis by professional malware analysts. Consequently, this paper presents SAFE-Deobs, a JavaScript deobfuscation tool that we have built. The aim of SAFE-Deobs is to automatically deobfuscate JavaScript malware such that an analyst can more rapidly determine the malicious script's intent. This is achieved through a number of static analyses, inspired by techniques from compiler theory. We demonstrate the utility of SAFE-Deobs through a case study on real-world JavaScript malware, and show that it is a useful addition to a malware analyst's toolset.

Ahmad Hazimeh, Adrian Herrera, Mathias Payer

High scalability and low running costs have made fuzz testing the de facto standard for discovering software bugs. Fuzzing techniques are constantly being improved in a race to build the ultimate bug-finding tool. However, while fuzzing excels at finding bugs in the wild, evaluating and comparing fuzzer performance is challenging due to the lack of metrics and benchmarks. For example, crash count, perhaps the most commonly-used performance metric, is inaccurate due to imperfections in deduplication techniques. Additionally, the lack of a unified set of targets results in ad hoc evaluations that hinder fair comparison. We tackle these problems by developing Magma, a ground-truth fuzzing benchmark that enables uniform fuzzer evaluation and comparison. By introducing real bugs into real software, Magma allows for the realistic evaluation of fuzzers against a broad set of targets. By instrumenting these bugs, Magma also enables the collection of bug-centric performance metrics independent of the fuzzer. Magma is an open benchmark consisting of seven targets that perform a variety of input manipulations and complex computations, presenting a challenge to state-of-the-art fuzzers. We evaluate seven widely-used mutation-based fuzzers (AFL, AFLFast, AFL++, FairFuzz, MOpt-AFL, honggfuzz, and SymCC-AFL) against Magma over 200,000 CPU-hours. Based on the number of bugs reached, triggered, and detected, we draw conclusions about the fuzzers' exploration and detection capabilities. This provides insight into fuzzer performance evaluation, highlighting the importance of ground truth in performing more accurate and meaningful evaluations.