A Little Confidence Goes a Long Way
This addresses the problem of high computational costs in LLM-based classification for researchers and practitioners, though it appears incremental as it builds on existing probing techniques.
The paper tackles binary classification by using probes on hidden state activations in large language models (LLMs), achieving performance comparable to advanced LLMs while requiring far fewer computational resources and no labeled data.
We introduce a group of related methods for binary classification tasks using probes of the hidden state activations in large language models (LLMs). Performance is on par with the largest and most advanced LLMs currently available, but requiring orders of magnitude fewer computational resources and not requiring labeled data. This approach involves translating class labels into a semantically rich description, spontaneous symmetry breaking of multilayer perceptron probes for unsupervised learning and inference, training probes to generate confidence scores (prior probabilities) from hidden state activations subject to known constraints via entropy maximization, and selecting the most confident probe model from an ensemble for prediction. These techniques are evaluated on four datasets using five base LLMs.