Edward Yao

Edward Yao, Wes Dyer, Georgios Gousios

Modern mobile applications include instrumentation that sample internal application metrics at regular intervals. Following a crash, sample metrics are collected and can potentially be valuable for root-causing difficult to diagnose crashes. However, the fine-grained nature and overwhelming wealth of available application metrics, coupled with frequent application updates, renders their use for root-causing crashes extremely difficult. We propose KabOOM, a method to automatically cluster telemetry reports in intuitive, distinct crash categories. Uniquely, KabOOM relies on multivariate timeseries fingerprinting; an auto-encoder coupled with a cluster centroid optimization technique learns embeddings of each crash report, which are then used to cluster metric timeseries based crash reports. We demonstrate the effectiveness of KabOOM on both reducing the dimensionality of the incoming crash reports and producing crash categories that are intuitive to developers.

Vijayaraghavan Murali, Edward Yao, Umang Mathur et al.

Despite engineering workflows that aim to prevent buggy code from being deployed, bugs still make their way into the Facebook app. When symptoms of these bugs, such as user submitted reports and automatically captured crashes, are reported, finding their root causes is an important step in resolving them. However, at Facebook's scale of billions of users, a single bug can manifest as several different symptoms according to the various user and execution environments in which the software is deployed. Root cause analysis (RCA) therefore requires tedious manual investigation and domain expertise to extract out common patterns that are observed in groups of reports and use them for debugging. We propose Minesweeper, a technique for RCA that moves towards automatically identifying the root cause of bugs from their symptoms. The method is based on two key aspects: (i) a scalable algorithm to efficiently mine patterns from telemetric information that is collected along with the reports, and (ii) statistical notions of precision and recall of patterns that help point towards root causes. We evaluate Minesweeper's scalability and effectiveness in finding root causes from symptoms on real world bug and crash reports from Facebook's apps. Our evaluation demonstrates that Minesweeper can perform RCA for tens of thousands of reports in less than 3 minutes, and is more than 85% accurate in identifying the root cause of regressions.