Analysis of Problem Tokens to Rank Factors Impacting Quality in VoIP Applications
This work addresses the need for VoIP providers to prioritize product improvements by quantifying the impact of quality issues, though it is incremental as it builds on existing QoE evaluation methods.
The paper tackled the problem of identifying specific factors that degrade user-perceived quality in VoIP applications by introducing a problem token questionnaire, and it demonstrated its efficacy using data from over 700,000 surveys from Skype to rank issues and isolate independent factors.
User-perceived quality-of-experience (QoE) in internet telephony systems is commonly evaluated using subjective ratings computed as a Mean Opinion Score (MOS). In such systems, while user MOS can be tracked on an ongoing basis, it does not give insight into which factors of a call induced any perceived degradation in QoE -- it does not tell us what caused a user to have a sub-optimal experience. For effective planning of product improvements, we are interested in understanding the impact of each of these degrading factors, allowing the estimation of the return (i.e., the improvement in user QoE) for a given investment. To obtain such insights, we advocate the use of an end-of-call "problem token questionnaire" (PTQ) which probes the user about common call quality issues (e.g., distorted audio or frozen video) which they may have experienced. In this paper, we show the efficacy of this questionnaire using data gathered from over 700,000 end-of-call surveys gathered from Skype (a large commercial VoIP application). We present a method to rank call quality and reliability issues and address the challenge of isolating independent factors impacting the QoE. Finally, we present representative examples of how these problem tokens have proven to be useful in practice.