Predicting Through Generation: Why Generation Is Better for Prediction
This addresses a method for improving structured prediction tasks using autoregressive models, which is incremental as it builds on existing generation-based approaches.
This paper tackles the problem of using autoregressive models for structured prediction tasks by arguing that token-level generation retains more mutual information than pooled representations, and introduces PredGen with scheduled sampling and a task adapter to address exposure bias and format mismatch. The results show PredGen consistently outperforms standard baselines on multiple classification and regression benchmarks.
This paper argues that generating output tokens is more effective than using pooled representations for prediction tasks because token-level generation retains more mutual information. Since LLMs are trained on massive text corpora using next-token prediction, generation aligns naturally with their learned behavior. Using the Data Processing Inequality (DPI), we provide both theoretical and empirical evidence supporting this claim. However, autoregressive models face two key challenges when used for prediction: (1) exposure bias, where the model sees ground truth tokens during training but relies on its own predictions during inference, leading to errors, and (2) format mismatch, where discrete tokens do not always align with the tasks required output structure. To address these challenges, we introduce PredGen(Predicting Through Generating), an end to end framework that (i) uses scheduled sampling to reduce exposure bias, and (ii) introduces a task adapter to convert the generated tokens into structured outputs. Additionally, we introduce Writer-Director Alignment Loss (WDAL), which ensures consistency between token generation and final task predictions, improving both text coherence and numerical accuracy. We evaluate PredGen on multiple classification and regression benchmarks. Our results show that PredGen consistently outperforms standard baselines, demonstrating its effectiveness in structured prediction tasks.