Theory of Hallucinations based on Equivariance
This work addresses hallucinations in language models, which is a critical issue for AI reliability, but it appears incremental as it builds on existing equivariance concepts and T5 models.
The study tackled the problem of hallucinations in large language models by hypothesizing that understanding real-world social relationships is key, and proposed using equivariant models to measure and improve this understanding. It introduced a hallucination scale based on a specialized cross-entropy error function, tested character-level equivariance with a T5-based technique, and discovered scale laws to aid in developing hallucination-free models.
This study aims to acquire knowledge for creating very large language models that are immune to hallucinations. Hallucinations in contemporary large language models are often attributed to a misunderstanding of real-world social relationships. Therefore, I hypothesize that very large language models capable of thoroughly grasping all these relationships will be free from hallucinations. Additionally, I propose that certain types of equivariant language models are adept at learning and understanding these relationships. Building on this, I have developed a specialized cross-entropy error function to create a hallucination scale for language models, which measures their extent of equivariance acquisition. Utilizing this scale, I tested language models for their ability to acquire character-level equivariance. In particular, I introduce and employ a novel technique based on T5 (Text To Text Transfer Transformer) that efficiently understands permuted input texts without the need for explicit dictionaries to convert token IDs (integers) to texts (strings). This T5 model demonstrated a moderate ability to acquire character-level equivariance. Additionally, I discovered scale laws that can aid in developing hallucination-free language models at the character level. This methodology can be extended to assess equivariance acquisition at the word level, paving the way for very large language models that can comprehensively understand relationships and, consequently, avoid hallucinations.