Transformation of Mean Opinion Scores to Avoid Misleading of Ranked based Statistical Techniques
This work solves a methodological problem for researchers in quality of experience and subjective data analysis, though it is incremental as it builds on existing statistical techniques.
The paper addresses the issue that rank-based statistical techniques can be misleading when applied to Mean Opinion Scores (MOS) due to inappropriate tied rank definitions, and introduces a transformation method using 95% Confidence Intervals to enable safe application of these techniques, with open-source software provided for implementation.
The rank correlation coefficients and the ranked-based statistical tests (as a subset of non-parametric techniques) might be misleading when they are applied to subjectively collected opinion scores. Those techniques assume that the data is measured at least at an ordinal level and define a sequence of scores to represent a tied rank when they have precisely an equal numeric value. In this paper, we show that the definition of tied rank, as mentioned above, is not suitable for Mean Opinion Scores (MOS) and might be misleading conclusions of rank-based statistical techniques. Furthermore, we introduce a method to overcome this issue by transforming the MOS values considering their $95\%$ Confidence Intervals. The rank correlation coefficients and ranked-based statistical tests can then be safely applied to the transformed values. We also provide open-source software packages in different programming languages to utilize the application of our transformation method in the quality of experience domain.