The flow of ideas in word embeddings
This work addresses how language models process ideas, potentially linking to creativity, but it is incremental as it combines existing methods from different fields.
The paper tackled the problem of analyzing idea flow in language by applying microrheology tools to study random walks in word embeddings, finding signatures of anomalous diffusion similar to complex systems like biological cells.
The flow of ideas has been extensively studied by physicists, psychologists, and machine learning engineers. This paper adopts specific tools from microrheology to investigate the similarity-based flow of ideas. We introduce a random walker in word embeddings and study its behavior. Such similarity-mediated random walks through the embedding space show signatures of anomalous diffusion commonly observed in complex structured systems such as biological cells and complex fluids. The paper concludes by proposing the application of popular tools employed in the study of random walks and diffusion of particles under Brownian motion to assess quantitatively the incorporation of diverse ideas in a document. Overall, this paper presents a self-referenced method combining microrheology and machine learning concepts to explore the meandering tendencies of language models and their potential association with creativity.