Towards Emergent Language Symbolic Semantic Segmentation and Model Interpretability
This addresses the need for model interpretability in medical imaging by providing symbolic explanations for segmentation decisions, though it is incremental as it builds on existing grounding and segmentation techniques.
The paper tackles the problem of interpretable semantic segmentation by developing a Symbolic Semantic (S^2) framework where two agents communicate via a private language to co-generate segmentation masks, achieving similar or better performance than state-of-the-art methods on tumor segmentation in the TCGA dataset.
Recent advances in methods focused on the grounding problem have resulted in techniques that can be used to construct a symbolic language associated with a specific domain. Inspired by how humans communicate complex ideas through language, we developed a generalized Symbolic Semantic ($\text{S}^2$) framework for interpretable segmentation. Unlike adversarial models (e.g., GANs), we explicitly model cooperation between two agents, a Sender and a Receiver, that must cooperate to achieve a common goal. The Sender receives information from a high layer of a segmentation network and generates a symbolic sentence derived from a categorical distribution. The Receiver obtains the symbolic sentences and co-generates the segmentation mask. In order for the model to converge, the Sender and Receiver must learn to communicate using a private language. We apply our architecture to segment tumors in the TCGA dataset. A UNet-like architecture is used to generate input to the Sender network which produces a symbolic sentence, and a Receiver network co-generates the segmentation mask based on the sentence. Our Segmentation framework achieved similar or better performance compared with state-of-the-art segmentation methods. In addition, our results suggest direct interpretation of the symbolic sentences to discriminate between normal and tumor tissue, tumor morphology, and other image characteristics.