Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries
This work provides empirical evidence that pass@k must be the primary metric for neural decompilation, highlighting the risk of superficial similarity gains from fine-tuning, which is relevant for LLM-based code generation tasks.
The paper evaluates fine-tuning and metrics for neural decompilation of Dart AOT binaries, finding that no fine-tuning configuration yields statistically significant pass@k improvement, with the best case only +0.71 pp (p=0.21) and fine-tuning the strongest base causing a -5.65 pp regression (p<0.001). It also demonstrates metric divergence where CodeBLEU and compile@k improve while pass@k declines.
Neural decompilation is increasingly studied as a code-generation problem, yet its evaluation methodology remains underdeveloped for modern languages. We present a systematic empirical study of fine-tuning effectiveness and metric validity for Dart Ahead-of-Time (AOT) neural decompilation. We evaluate six fine-tuned model variants across three base architectures (4B-8B parameters) using three metrics: CodeBLEU, compile@k, and pass@k on a new 154-task HumanEval-Dart benchmark. Our study yields three principal findings grounded in paired task-level statistical tests. First, no fine-tuning configuration produces a statistically significant pass@k improvement. The sole positive case yields +0.71 pp (McNemar p=0.21), while fine-tuning the strongest base (Qwen3-8B) causes a highly significant regression of -5.65 pp (p<0.001). This capacity-dependent trend is consistent across architectures but needs broader scale sweeps. Second, cross-lingual interference from Swift training is highly significant at 4B (-2.66 pp, p<0.001) but statistically indistinguishable from zero at 8B, consistent with the scaling hypothesis. Third, we demonstrate metric divergence: CodeBLEU and compile@k can improve significantly while pass@k moves in the opposite direction. This has implications for any LLM code generation task where fine-tuning targets superficial similarity. Error analysis reveals assembly sequence length is the strongest predictor of task difficulty (p=0.001), with a capability cliff at 200 instructions. We contribute the HumanEval-Dart benchmark, a Dart-adapted CodeBLEU, and empirical evidence that pass@k must be the primary evaluation metric for neural decompilation.