On Learning Prediction-Focused Mixtures
This addresses the trade-off between interpretability and performance in mixture models for practitioners needing compact, task-specific models.
The paper tackles the problem of learning interpretable mixture models with few components while maintaining prediction performance by introducing prediction-focused modeling that automatically selects task-relevant dimensions. The approach outperforms non-prediction-focused models and includes theoretical characterization of when it works.
Probabilistic models help us encode latent structures that both model the data and are ideally also useful for specific downstream tasks. Among these, mixture models and their time-series counterparts, hidden Markov models, identify discrete components in the data. In this work, we focus on a constrained capacity setting, where we want to learn a model with relatively few components (e.g. for interpretability purposes). To maintain prediction performance, we introduce prediction-focused modeling for mixtures, which automatically selects the dimensions relevant to the prediction task. Our approach identifies relevant signal from the input, outperforms models that are not prediction-focused, and is easy to optimize; we also characterize when prediction-focused modeling can be expected to work.