Meta-Statistical Learning: Supervised Learning of Statistical Inference
This work addresses the problem of adapting machine learning to statistical inference tasks for researchers and practitioners, offering a novel but incremental approach by repurposing existing Transformer architectures.
The authors tackled the challenge of applying supervised learning to distribution-level tasks like parameter estimation and hypothesis testing by proposing meta-statistical learning, a framework that treats entire datasets as inputs to neural networks, and demonstrated strong performance in applications such as hypothesis testing and mutual information estimation, especially for small datasets.
This work demonstrates that the tools and principles driving the success of large language models (LLMs) can be repurposed to tackle distribution-level tasks, where the goal is to predict properties of the data-generating distribution rather than labels for individual datapoints. These tasks encompass statistical inference problems such as parameter estimation, hypothesis testing, or mutual information estimation. Framing these tasks within traditional machine learning pipelines is challenging, as supervision is typically tied to individual datapoint. We propose meta-statistical learning, a framework inspired by multi-instance learning that reformulates statistical inference tasks as supervised learning problems. In this approach, entire datasets are treated as single inputs to neural networks, which predict distribution-level parameters. Transformer-based architectures, without positional encoding, provide a natural fit due to their permutation-invariance properties. By training on large-scale synthetic datasets, meta-statistical models can leverage the scalability and optimization infrastructure of Transformer-based LLMs. We demonstrate the framework's versatility with applications in hypothesis testing and mutual information estimation, showing strong performance, particularly for small datasets where traditional neural methods struggle.