Wlodek W. Zadrozny

Wlodek W. Zadrozny

The ability to abstract, count, and use System~2 reasoning are well-known manifestations of intelligence and understanding. In this paper, we argue, using the example of the ``Look and Say" puzzle, that although deep neural networks can exhibit high `competence' (as measured by accuracy) when trained on large data sets (2 million examples in our case), they do not show any sign on the deeper understanding of the problem, or what D. Dennett calls `comprehension'. We report on two sets experiments: first, computing the next element of the sequence, and ,then, the previous element. We view both problems as building a translator from one set of tokens to another. We apply both standard LSTMs and Transformer/Attention-based neural networks, using publicly available machine translation software. We observe that despite the amazing accuracy, the performance of the trained programs on the actual L\&S sequence is bad, and shows no understanding of the principles behind the sequences. The ramifications of this finding include: (1) from the cognitive science perspective, we argue that we need better mathematical models of abstraction; (2) the universality of neural networks should be re-examined for functions acting on discrete data sets; (3) we hypothesize topology can provide a definition of without the reference to the concept of distance.

Wlodek W. Zadrozny

In the last couple of years there were a few attempts to apply topological data analysis to text, and in particular to natural language inference. A recent work by Tymochko et al. suggests the possibility of capturing `the notion of logical shape in text,' using `topological delay embeddings,' a technique derived from dynamical systems, applied to word embeddings. In this note we reconstruct their argument and show, using several old and new examples, that the problem of connecting logic, topology and text is still very much unsolved. We conclude that there is no clear answer to the question: ``Can we find a circle in a circular argument?'' We point out some possible avenues of exploration. The code used in our experiment is also shown.

Wlodek W. Zadrozny

This article contains a proposal to add coinduction to the computational apparatus of natural language understanding. This, we argue, will provide a basis for more realistic, computationally sound, and scalable models of natural language dialogue, syntax and semantics. Given that the bottom up, inductively constructed, semantic and syntactic structures are brittle, and seemingly incapable of adequately representing the meaning of longer sentences or realistic dialogues, natural language understanding is in need of a new foundation. Coinduction, which uses top down constraints, has been successfully used in the design of operating systems and programming languages. Moreover, implicitly it has been present in text mining, machine translation, and in some attempts to model intensionality and modalities, which provides evidence that it works. This article shows high level formalizations of some of such uses. Since coinduction and induction can coexist, they can provide a common language and a conceptual model for research in natural language understanding. In particular, such an opportunity seems to be emerging in research on compositionality. This article shows several examples of the joint appearance of induction and coinduction in natural language processing. We argue that the known individual limitations of induction and coinduction can be overcome in empirical settings by a combination of the the two methods. We see an open problem in providing a theory of their joint use.