Systematic Generation of Conformance Tests for JavaScript
This addresses the tedious and error-prone process of hand-writing test suites for JavaScript implementations, benefiting developers and standardizers, though it is incremental as it builds on existing symbolic execution techniques.
The paper tackled the problem of generating conformance tests for JavaScript by using dynamic symbolic execution of polyfills, identifying 17 divergences in the core-js polyfill and increasing branch coverage by up to 15%.
JavaScript implementations are tested for conformance to the ECMAScript standard using a large hand-written test suite. Not only in this a tedious approach, it also relies solely on the natural language specification for differentiating behaviors, while hidden implementation details can also affect behavior and introduce divergences. We propose to generate conformance tests through dynamic symbolic execution of polyfills, drop-in replacements for newer JavaScript language features that are not yet widely supported. We then run these generated tests against multiple implementations of JavaScript, using a majority vote to identify the correct behavior. To facilitate test generation for polyfill code, we introduce a model for structured symbolic inputs that is suited to the dynamic nature of JavaScript. In our evaluation, we found 17 divergences in the widely used core-js polyfill and were able to increase branch coverage in interpreter code by up to 15%. Because polyfills are typically written even before standardization, our approach will allow to maintain and extend standardization test suites with reduced effort.