Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)
This provides a quick and simple method for analyzing new technologies and performing pre-tests in subjective assessment, though it is incremental.
The paper tackles the problem of subjective assessment tests by proposing a design where three to four team members rate stimuli multiple times, achieving results comparable to a high-performing objective metric.
Recent studies have shown that it is possible to characterize subject bias and variance in subjective assessment tests. Apparent differences among subjects can, for the most part, be explained by random factors. Building on that theory, we propose a subjective test design where three to four team members each rate the stimuli multiple times. The results are comparable to a high performing objective metric. This provides a quick and simple way to analyze new technologies and perform pre-tests for subjective assessment.