Alberto Casagrande

Alberto Casagrande, Francesco Fabris, Rossano Girometti et al.

Agreement measures, such as Cohen's kappa or intraclass correlation, gauge the matching between two or more classifiers. They are used in a wide range of contexts from medicine, where they evaluate the effectiveness of medical treatments and clinical trials, to artificial intelligence, where they can quantify the approximation due to the reduction of a classifier. The consistency of different classifiers to a golden standard can be compared simply by using the order induced by their agreement measure with respect to the golden standard itself. Nevertheless, labelling an approach as good or bad exclusively by using the value of an agreement measure requires a scale or a significativity index. Some quality scales have been proposed in the literature for Cohen's kappa, but they are mainly naïve, and their boundaries are arbitrary. This work proposes a general approach to evaluate the significativity of any agreement value between two classifiers and introduces two significativity indices: one dealing with finite data sets, the other one handling classification probability distributions. Moreover, this manuscript addresses the computational challenges of evaluating such indices and proposes some efficient algorithms for their evaluation.

Alberto Casagrande, Francesco Fabris, Rossano Girometti

Agreement measures are useful to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. Information Agreement (IA) is an information-theoretic-based agreement measure introduced to overcome all the limitations and alleged pitfalls of Cohen's Kappa. However, it is only able to deal with agreement matrices whose values are positive natural numbers. This work extends IA admitting also 0 as a possible value for the agreement matrix cells.

Alberto Casagrande, Joanna Jarmolowska, Marcello Turconi et al.

P300 is an electric signal emitted by brain about 300 milliseconds after a rare, but relevant-for-the-user event. One of the applications of this signal is sentence spelling that enables subjects who lost the control of their motor pathways to communicate by selecting characters in a matrix containing all the alphabet symbols. Although this technology has made considerable progress in the last years, it still suffers from both low communication rate and high error rate. This article presents a P300 speller, named PolyMorph, that introduces two major novelties in the field: the selection matrix polymorphism, that reduces the size of the selection matrix itself by removing useless symbols, and sentence-based predictions, that exploit all the spelt characters of a sentence to determine the probability of a word. In order to measure the effectiveness of the presented speller, we describe two sets of tests: the first one in vivo and the second one in silico. The results of these experiments suggest that the use of PolyMorph in place of the naive character-by-character speller both increases the number of spelt characters per time unit and reduces the error rate.