David M. W. Powers

David M. W. Powers

Commonly used evaluation measures including Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic. Using these measures a system that performs worse in the objective sense of Informedness, can appear to perform better under any of these commonly used measures. We discuss several concepts and measures that reflect the probability that prediction is informed versus chance. Informedness and introduce Markedness as a dual measure for the probability that prediction is marked versus chance. Finally we demonstrate elegant connections between the concepts of Informedness, Markedness, Correlation and Significance as well as their intuitive relationships with Recall and Precision, and outline the extension from the dichotomous case to the general multi-class case.

David M. W. Powers

There has been considerable interest in boosting and bagging, including the combination of the adaptive techniques of AdaBoost with the random selection with replacement techniques of Bagging. At the same time there has been a revisiting of the way we evaluate, with chance-corrected measures like Kappa, Informedness, Correlation or ROC AUC being advocated. This leads to the question of whether learning algorithms can do better by optimizing an appropriate chance corrected measure. Indeed, it is possible for a weak learner to optimize Accuracy to the detriment of the more reaslistic chance-corrected measures, and when this happens the booster can give up too early. This phenomenon is known to occur with conventional Accuracy-based AdaBoost, and the MultiBoost algorithm has been developed to overcome such problems using restart techniques based on bagging. This paper thus complements the theoretical work showing the necessity of using chance-corrected measures for evaluation, with empirical work showing how use of a chance-corrected measure can improve boosting. We show that the early surrender problem occurs in MultiBoost too, in multiclass situations, so that chance-corrected AdaBook and Multibook can beat standard Multiboost or AdaBoost, and we further identify which chance-corrected measures to use when.

Davoud Mougouei, David M. W. Powers

Software requirement selection is to find an optimal set of requirements that gives the highest value for a release of software while keeping the cost within the budget. However, value-related dependencies among software requirements may impact the value of an optimal set. Moreover, value-related dependencies can be of varying strengths. Hence, it is important to consider both the existence and the strengths of value-related dependencies during a requirement selection. The existing selection models however, either assume that software requirements are independent or they ignore strengths of requirement dependencies. This paper presents a cost-value optimization model that considers the impacts of value-related requirement dependencies on the value of selected requirements (optimal set). We have exploited algebraic structure of fuzzy graphs for modeling value-related requirement dependencies and their strengths. Validity and practicality of the work are verified through carrying out several simulations and studying a real world software project.

Nasim Nematzadeh, David M. W. Powers

The Café Wall illusion is one of a class of tilt illusions where lines that are parallel appear to be tilted. We demonstrate that a simple Differences of Gaussian model provides an explanatory mechanism for the illusory tilt perceived in a family of Café Wall illusion generalizes to the dashed versions of Café Wall. Our explanation models the visual mechanisms in low level stages and the lateral inhibition of simple cells that can reveal tilt cues in Geometrical distortion illusions such as Tile illusions particularly Café Wall illusions. For this, we simulate the activations of the retinal/cortical simple cells in responses to these patterns based on a Classical Receptive Field (CRF) model (referred to as Vis-CRF) to explain tilt effects in these illusions. Previously, it was assumed that all these visual experiences of tilt arise from the orientation selectivity properties described for more complex cortical cells. An estimation of an overall tilt angle perceived in these illusions is based on the integration of the local tilts detected by simple cells which is presumed to be a key mechanism utilized by the complex cells to create our final perception of tilt.

Nasim Nematzadeh, David M. W. Powers, Trent Lewis

Illusions are fascinating and immediately catch people's attention and interest, but they are also valuable in terms of giving us insights into human cognition and perception. A good theory of human perception should be able to explain the illusion, and a correct theory will actually give quantifiable results. We investigate here the efficiency of a computational filtering model utilised for modelling the lateral inhibition of retinal ganglion cells and their responses to a range of Geometric Illusions using isotropic Differences of Gaussian filters. This study explores the way in which illusions have been explained and shows how a simple standard model of vision based on classical receptive fields can predict the existence of these illusions as well as the degree of effect. A fundamental contribution of this work is to link bottom-up processes to higher level perception and cognition consistent with Marr's theory of vision and edge map representation.

Nasim Nematzadeh, David M. W. Powers

Geometrical illusions are a subclass of optical illusions in which the geometrical characteristics of patterns such as orientations and angles are distorted and misperceived as the result of low- to high-level retinal/cortical processing. Modelling the detection of tilt in these illusions and their strengths as they are perceived is a challenging task computationally and leads to development of techniques that match with human performance. In this study, we present a predictive and quantitative approach for modeling foveal and peripheral vision in the induced tilt in Café Wall illusion in which parallel mortar lines between shifted rows of black and white tiles appear to converge and diverge. A bioderived filtering model for the responses of retinal/cortical simple cells to the stimulus using Difference of Gaussians is utilized with an analytic processing pipeline introduced in our previous studies to quantify the angle of tilt in the model. Here we have considered visual characteristics of foveal and peripheral vision in the perceived tilt in the pattern to predict different degrees of tilt in different areas of the fovea and periphery as the eye saccades to different parts of the image. The tilt analysis results from several sampling sizes and aspect ratios, modelling variant foveal views are used from our previous investigations on the local tilt, and we specifically investigate in this work, different configurations of the whole pattern modelling variant Gestalt views across multiple scales in order to provide confidence intervals around the predicted tilts. The foveal sample sets are verified and quantified using two different sampling methods. We present here a precise and quantified comparison contrasting local tilt detection in the foveal sets with a global average across all of the Café Wall configurations tested in this work.

Nasim Nematzadeh, David M. W. Powers

This paper explores the tilt illusion effect in the Cafe Wall pattern using a classical Gaussian Receptive Field model. In this illusion, the mortar lines are misperceived as diverging or converging rather than horizontal. We examine the capability of a simple bioplausible filtering model to recognize different degrees of tilt effect in the Cafe Wall illusion based on different characteristics of the pattern. Our study employed a Difference of Gaussians model of retinal to cortical ON center and/or OFF center receptive fields. A wide range of parameters of the stimulus, for example mortar thickness, luminance, tiles contrast, phase of the tile displacement, have been studied. Our model constructs an edge map representation at multiple scales that reveals tilt cues and clues involved in the illusory perception of the Cafe Wall pattern. We present here that our model can not only detect the tilt in this pattern, but also can predict the strength of the illusion and quantify the degree of tilt. For the first time quantitative predictions of a model are reported for this stimulus. The results of our simulations are consistent with previous psychophysical findings across the full range of Cafe Wall variations tested. Our results also suggest that the Difference of Gaussians mechanism is the heart of the effects explained by, and the mechanisms proposed for, the Irradiation, Brightness Induction, and Bandpass Filtering models.

Nasim Nematzadeh, David M. W. Powers, Trent W. Lewis

Why does our visual system fail to reconstruct reality, when we look at certain patterns? Where do Geometrical illusions start to emerge in the visual pathway? How far should we take computational models of vision with the same visual ability to detect illusions as we do? This study addresses these questions, by focusing on a specific underlying neural mechanism involved in our visual experiences that affects our final perception. Among many types of visual illusion, Geometrical and, in particular, Tilt Illusions are rather important, being characterized by misperception of geometric patterns involving lines and tiles in combination with contrasting orientation, size or position. Over the last decade, many new neurophysiological experiments have led to new insights as to how, when and where retinal processing takes place, and the encoding nature of the retinal representation that is sent to the cortex for further processing. Based on these neurobiological discoveries, we provide computer simulation evidence from modelling retinal ganglion cells responses to some complex Tilt Illusions, suggesting that the emergence of tilt in these illusions is partially related to the interaction of multiscale visual processing performed in the retina. The output of our low-level filtering model is presented for several types of Tilt Illusion, predicting that the final tilt percept arises from multiple-scale processing of the Differences of Gaussians and the perceptual interaction of foreground and background elements. Our results suggest that this model has a high potential in revealing the underlying mechanism connecting low-level filtering approaches to mid- and high-level explanations such as Anchoring theory and Perceptual grouping.

Davoud Mougouei, David M. W. Powers, Asghar Moeini

Binary Knapsack Problem (BKP) is to select a subset of an element (item) set with the highest value while keeping the total weight within the capacity of the knapsack. This paper presents an integer programming model for a variation of BKP where the value of each element may depend on selecting or ignoring other elements. Strengths of such Value-Related Dependencies are assumed to be imprecise and hard to specify. To capture this imprecision, we have proposed modeling value-related dependencies using fuzzy graphs and their algebraic structure.

Davoud Mougouei, David M. W. Powers

Considering user preferences is a determining factor in optimizing the value of a software release. This is due to the fact that user preferences for software features specify the values of those features and consequently determine the value of the release. Certain features of a software however, may encourage or discourage users to prefer (select or use) other features. As such, value of a software feature could be positively or negatively influenced by other features. Such influences are known as Value-related Feature (Requirement) Dependencies. Value-related dependencies need to be considered in software release planning as they influence the value of the optimal subset of the features selected by the release planning models. Hence, we have proposed considering value-related feature dependencies in software release planning through mining user preferences for software features. We have demonstrated the validity and practicality of the proposed approach by studying a real world software project.

Tom A. F. Anderson, David M. W. Powers

We report investigations into speaker classification of larger quantities of unlabelled speech data using small sets of manually phonemically annotated speech. The Kohonen speech typewriter is a semi-supervised method comprised of self-organising maps (SOMs) that achieves low phoneme error rates. A SOM is a 2D array of cells that learn vector representations of the data based on neighbourhoods. In this paper, we report a method to evaluate pronunciation using multilevel SOMs with /hVd/ single syllable utterances for the study of vowels, for Australian pronunciation.

Nasim Nematzadeh, David M. W. Powers

The biological characteristics of human visual processing can be investigated through the study of optical illusions and their perception, giving rise to intuitions that may improve computer vision to match human performance. Geometric illusions are a specific subfamily in which orientations and angles are misperceived. This paper reports quantifiable predictions of the degree of tilt for a typical geometric illusion called Café Wall, in which the mortar between the tiles seems to tilt or bow. Our study employs a common bioplausible model of retinal processing and we further develop an analytic processing pipeline to quantify and thus predict the specific angle of tilt. We further study the effect of resolution and feature size in order to predict the different perceived tilts in different areas of the fovea and periphery, where resolution varies as the eye saccades to different parts of the image. In the experiments, several different minimal portions of the pattern, modeling monocular and binocular foveal views, are investigated across multiple scales, in order to quantify tilts with confidence intervals and explore the difference between local and global tilt.

David M. W. Powers

Prof. Robert Berwick's abstract for his forthcoming invited talk at the ACL2016 workshop on Cognitive Aspects of Computational Language Learning revives an ancient debate. Entitled "Why take a chance?", Berwick seems to refer implicitly to Chomsky's critique of the statistical approach of Harris as well as the currently dominant paradigms in CoNLL. Berwick avoids Chomsky's use of "innate" but states that "the debate over the existence of sophisticated mental grammars was settled with Chomsky's Logical Structure of Linguistic Theory (1957/1975)", acknowledging that "this debate has often been revived". This paper agrees with the view that this debate has long since been settled, but with the opposite outcome! Given the embers have not yet died away, and the questions remain fundamental, perhaps it is appropriate to refuel the debate, so I would like to join Bob in throwing fuel on this fire by reviewing the evidence against the Chomskian position!

Somaiyeh Mahmoud Zadeh, David M. W. Powers, AmirMehdi Yazdani

Promoting the levels of autonomy facilitates the vehicle in performing long-range operations with minimum supervision. The capability of Autonomous Underwater Vehicles (AUVs) to fulfill the mission objectives is directly influenced by route planning and task assignment system performance. The system fives the error of "Bad character(s) in field Abstract" for no reason. Please refer to manuscript for the full abstract

David M. W. Powers

Evaluation often aims to reduce the correctness or error characteristics of a system down to a single number, but that always involves trade-offs. Another way of dealing with this is to quote two numbers, such as Recall and Precision, or Sensitivity and Specificity. But it can also be useful to see more than this, and a graphical approach can explore sensitivity to cost, prevalence, bias, noise, parameters and hyper-parameters. Moreover, most techniques are implicitly based on two balanced classes, and our ability to visualize graphically is intrinsically two dimensional, but we often want to visualize in a multiclass context. We review the dichotomous approaches relating to Precision, Recall, and ROC as well as the related LIFT chart, exploring how they handle unbalanced and multiclass data, and deriving new probabilistic and information theoretic variants of LIFT that help deal with the issues associated with the handling of multiple and unbalanced classes.

David M. W. Powers

Over the last decade there has been increasing concern about the biases embodied in traditional evaluation methods for Natural Language Processing/Learning, particularly methods borrowed from Information Retrieval. Without knowledge of the Bias and Prevalence of the contingency being tested, or equivalently the expectation due to chance, the simple conditional probabilities Recall, Precision and Accuracy are not meaningful as evaluation measures, either individually or in combinations such as F-factor. The existence of bias in NLP measures leads to the 'improvement' of systems by increasing their bias, such as the practice of improving tagging and parsing scores by using most common value (e.g. water is always a Noun) rather than the attempting to discover the correct one. The measures Cohen Kappa and Powers Informedness are discussed as unbiased alternative to Recall and related to the psychologically significant measure DeltaP. In this paper we will analyze both biased and unbiased measures theoretically, characterizing the precise relationship between all these measures as well as evaluating the evaluation measures themselves empirically using a Monte Carlo simulation.

David M. W. Powers

The F-measure or F-score is one of the most commonly used single number measures in Information Retrieval, Natural Language Processing and Machine Learning, but it is based on a mistake, and the flawed assumptions render it unsuitable for use in most contexts! Fortunately, there are better alternatives.