CLAug 1, 2020
The test set for the TransCoder system
arXiv:2008.00293v1
Originality Synthesis-oriented
AI Analysis
This highlights a critical evaluation gap for code translation systems, but it is incremental as it points out a specific flaw in an existing benchmark.
The paper identifies that the TransCoder system's test set lacks key Java features like class definitions and non-recursive user-defined function calls, leaving its accuracy for such programs unknown.
The TransCoder system translates source code between Java, C++, and Python 3. The test set that was used to evaluate its quality is missing important features of Java, including the ability to define and use classes and the ability to call user-defined functions other than recursively. Therefore, the accuracy of TransCoder over programs with those features remains unknown.