Revisiting Hotels-50K and Hotel-ID
This work addresses the challenge of evaluating models for countering human trafficking by improving dataset realism, though it is incremental as it modifies existing datasets rather than introducing new methods.
The authors tackled the problem of hotel recognition datasets not aligning with real-world unseen scenarios by proposing revisited versions of Hotels50K and Hotel-ID with varying difficulty levels. They tested state-of-the-art image retrieval models, showing performance decreases and ranking changes as evaluation settings approached real-world conditions.
In this paper, we propose revisited versions for two recent hotel recognition datasets: Hotels50K and Hotel-ID. The revisited versions provide evaluation setups with different levels of difficulty to better align with the intended real-world application, i.e. countering human trafficking. Real-world scenarios involve hotels and locations that are not captured in the current data sets, therefore it is important to consider evaluation settings where classes are truly unseen. We test this setup using multiple state-of-the-art image retrieval models and show that as expected, the models' performances decrease as the evaluation gets closer to the real-world unseen settings. The rankings of the best performing models also change across the different evaluation settings, which further motivates using the proposed revisited datasets.