Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark
For practitioners of screen-conditioned action prediction, the paper demonstrates that fine-tuning a smaller model can dramatically outperform larger frontier zero-shot models, but the choice of base model and fine-tuning recipe is critical.
The paper benchmarks supervised fine-tuned models against frontier zero-shot baselines on a 661-row test set from the PiSAR corpus. Fine-tuned Qwen3-VL-8B-Instruct achieves sem_sim 0.783, outperforming frontier baselines (Claude Opus 4.7: 0.459, GPT-5.5: 0.482) by 0.30 absolute, while Gemma-4-26B-A4B-IT fine-tuned with the same recipe scores only 0.441, indicating a recipe-vs-model mismatch.
We benchmark three supervised fine-tuned models against frontier zero-shot baselines on a 661-row held-out slice of PiSAR (Persona, intent, Screen, Action, Rationale), a 12,929-tuple corpus of screen-anchored behavioural rationales curated from public app-store reviews, Pew American Trends Panel demographics, and the OPeRA shopper traces. Every model, frontier or fine-tuned, is evaluated on the same 661-row slice with the same scoring pipeline. Two findings. First, frontier zero-shot baselines (Claude Opus 4.7 and GPT-5.5) reach sem_sim 0.459 and 0.482 respectively; a fine-tuned Qwen3-VL-8B-Instruct reaches 0.783 and clears sem_sim >= 0.7 on 79% of rows, against 1-2% for either frontier baseline, a gap of 0.30 absolute on the same test set. Second, the same training data and recipe on Gemma-4-26B-A4B-IT scores only 0.441, in the same band as the frontier zero-shot baselines rather than the fine-tuned Qwen. We read this as a recipe-vs-model mismatch: the reasoning-tuned high-parameter model resists displacement and would likely need either more data or a stronger fine-tuning method.