Mixing Configurations for Downstream Prediction
This addresses the need for better unsupervised feature extraction in machine learning, particularly for tasks like bioinformatics and tabular data, though it is incremental as it builds on existing community detection and configuration concepts.
The paper tackles the problem of selecting or composing configurations for downstream tasks by introducing GraMixC, a plug-and-play module that extracts, aligns, and fuses configurations, improving the R2 score from 0.6 to 0.9 on the DSN1 16S rRNA cultivation-media prediction task.
Humans possess an innate ability to group objects by similarity, a cognitive mechanism that clustering algorithms aim to emulate. Recent advances in community detection have enabled the discovery of configurations -- valid hierarchical clusterings across multiple resolution scales -- without requiring labeled data. In this paper, we formally characterize these configurations and identify similar emergent structures in register tokens within Vision Transformers. Unlike register tokens, configurations exhibit lower redundancy and eliminate the need for ad hoc selection. They can be learned through unsupervised or self-supervised methods, yet their selection or composition remains specific to the downstream task and input. Building on these insights, we introduce GraMixC, a plug-and-play module that extracts configurations, aligns them using our Reverse Merge/Split (RMS) technique, and fuses them via attention heads before forwarding them to any downstream predictor. On the DSN1 16S rRNA cultivation-media prediction task, GraMixC improves the R2 score from 0.6 to 0.9 across multiple methods, setting a new state of the art. We further validate GraMixC on standard tabular benchmarks, where it consistently outperforms single-resolution and static-feature baselines.