Paul Tupper

Simone Brugiapaglia, Matthew Liu, Paul Tupper

Often in language and other areas of cognition, whether two components of an object are identical or not determine whether it is well formed. We call such constraints identity effects. When developing a system to learn well-formedness from examples, it is easy enough to build in an identify effect. But can identity effects be learned from the data without explicit guidance? We provide a simple framework in which we can rigorously prove that algorithms satisfying simple criteria cannot make the correct inference. We then show that a broad class of algorithms including deep neural networks with standard architecture and training with backpropagation satisfy our criteria, dependent on the encoding of inputs. Finally, we demonstrate our theory with computational experiments in which we explore the effect of different input encodings on the ability of algorithms to generalize to novel inputs.

Paul Tupper, Paul Smolensky, Pyeong Whan Cho

Gradient Symbolic Computation is proposed as a means of solving discrete global optimization problems using a neurally plausible continuous stochastic dynamical system. Gradient symbolic dynamics involves two free parameters that must be adjusted as a function of time to obtain the global maximizer at the end of the computation. We provide a summary of what is known about the GSC dynamics for special cases of settings of the parameters, and also establish that there is a schedule for the two parameters for which convergence to the correct answer occurs with high probability. These results put the empirical results already obtained for GSC on a sound theoretical footing.

Benjamin Goodman, Paul Tupper

In spoken languages, speakers divide up the space of phonetic possibilities into different regions, corresponding to different phonemes. We consider a simple exemplar model of how this division of phonetic space varies over time among a population of language users. In the particular model we consider, we show that, once the system is initialized with a given set of phonemes, that phonemes do not become extinct: all phonemes will be maintained in the system for all time. This is in contrast to what is observed in more complex models. Furthermore, we show that the boundaries between phonemes fluctuate and we quantitatively study the fluctuations in a simple instance of our model. These results prepare the ground for more sophisticated models in which some phonemes go extinct or new phonemes emerge through other processes.

B. Goodman, P. F. Tupper

Exemplar models are a popular class of models used to describe language change. Here we study how limiting the memory capacity of an individual in these models affects the system's behaviour. In particular we demonstrate the effect this change has on the extinction of categories. Previous work in exemplar dynamics has not addressed this question. In order to investigate this, we will inspect a simplified exemplar model. We will prove for the simplified model that all the sound categories but one will always become extinct, whether memory storage is limited or not. However, computer simulations show that changing the number of stored memories alters how fast categories become extinct.

Paul Tupper, Bobak Shahriari

We propose a novel framework for the analysis of learning algorithms that allows us to say when such algorithms can and cannot generalize certain patterns from training data to test data. In particular we focus on situations where the rule that must be learned concerns two components of a stimulus being identical. We call such a basis for discrimination an identity-based rule. Identity-based rules have proven to be difficult or impossible for certain types of learning algorithms to acquire from limited datasets. This is in contrast to human behaviour on similar tasks. Here we provide a framework for rigorously establishing which learning algorithms will fail at generalizing identity-based rules to novel stimuli. We use this framework to show that such algorithms are unable to generalize identity-based rules to novel inputs unless trained on virtually all possible inputs. We demonstrate these results computationally with a multilayer feedforward neural network.

P. F. Tupper

We develop a model of phonological contrast in natural language. Specifically, the model describes the maintenance of contrast between different words in a language, and the elimination of such contrast when sounds in the words merge. An example of such a contrast is that provided by the two vowel sounds 'i' and 'e', which distinguish pairs of words such as 'pin' and 'pen' in most dialects of English. We model language users' knowledge of the pronunciation of a word as consisting of collections of labeled exemplars stored in memory. Each exemplar is a detailed memory of a particular utterance of the word in question. In our model an exemplar is represented by one or two phonetic variables along with a weight indicating how strong the memory of the utterance is. Starting from an exemplar-level model we derive integro-differential equations for the evolution of exemplar density fields in phonetic space. Using these latter equations we investigate under what conditions two sounds merge, thus eliminating the contrast. Our main conclusion is that for the preservation of phonological contrast, it is necessary that anomalous utterances of a given word are discarded, and not merely stored in memory as an exemplar of another word.

P. F. Tupper

We develop a model for the stability and maintenance of phonological categories. Examples of phonological categories are vowel sounds such as "i" and "e". We model such categories as consisting of collections of labeled exemplars that language users store in their memory. Each exemplar is a detailed memory of an instance of the linguistic entity in question. Starting from an exemplar-level model we derive integro-differential equations for the long-term evolution of the density of exemplars in different portions of phonetic space. Using these latter equations we investigate under what conditions two phonological categories merge or not. Our main conclusion is that for the preservation of distinct phonological categories, it is necessary that anomalous speech tokens of a given category are discarded, and not merely stored in memory as an exemplar of another category.

P. F. Tupper

An open problem in numerical analysis is to explain why molecular dynamics works. The difficulty is that numerical trajectories are only accurate for very short times, whereas the simulations are performed over long time intervals. It is believed that statistical information from these simulations is accurate, but no one has offered a rigourous proof of this. In order to give mathematicians a clear goal in understanding this problem, we state a precise mathematical conjecture about molecular dynamics simulation of a particular system. We believe that if the conjecture is proved, we will then understand why molecular dynamics works.

P. F. Tupper

We consider the numerical simulation of Hamiltonian systems of ordinary differential equations. Two features of Hamiltonian systems are that energy is conserved along trajectories and phase space volume is preserved by the flow. We want to determine if there are integration schemes that preserve these two properties for all Hamiltonian systems, or at least for all systems in a wide class. This paper provides provides a negative result in the case of two dimensional (one degree of freedom) Hamiltonian systems, for which phase space volume is identical to area. Our main theorem shows that there are no computationally reasonable numerical integrators for which all Hamiltonian systems of one degree of freedom can be integrated while conserving both area and energy. Before proving this result we define what we mean by a computationally reasonable integrator. We then consider what obstructions this result places on the existence of volume- and energy-conserving integrators for Hamiltonian systems with an arbitrary number of degrees of freedom.