Multivariate Dependence Beyond Shannon Information
This work is foundational for researchers in complex systems and information theory, highlighting a critical limitation in existing methods for causal discovery.
The paper tackles the problem of measuring multivariate dependencies using Shannon information measures, showing they are inadequate for determining meaningful dependency structure and intrinsic causal relations, and demonstrates that such problematic distributions exist across arbitrary sets of variables.
Accurately determining dependency structure is critical to discovering a system's causal organization. We recently showed that the transfer entropy fails in a key aspect of this---measuring information flow---due to its conflation of dyadic and polyadic relationships. We extend this observation to demonstrate that this is true of all such Shannon information measures when used to analyze multivariate dependencies. This has broad implications, particularly when employing information to express the organization and mechanisms embedded in complex systems, including the burgeoning efforts to combine complex network theory with information theory. Here, we do not suggest that any aspect of information theory is wrong. Rather, the vast majority of its informational measures are simply inadequate for determining the meaningful dependency structure within joint probability distributions. Therefore, such information measures are inadequate for discovering intrinsic causal relations. We close by demonstrating that such distributions exist across an arbitrary set of variables.