Improving Baselines in the Wild
This work provides incremental insights for researchers developing robust models in domain shift scenarios, specifically using the WILDS benchmark.
The paper tackles the problem of improving model robustness to domain shifts using the WILDS benchmark, showing that minor hyperparameter changes and separate cross-validation for each metric significantly boost baseline performance, with improvements mainly on the FMoW dataset.
We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain shifts. Several experiments yield a couple of critical observations which we believe are of general interest for any future work on WILDS. Our study focuses on two datasets: iWildCam and FMoW. We show that (1) Conducting separate cross-validation for each evaluation metric is crucial for both datasets, (2) A weak correlation between validation and test performance might make model development difficult for iWildCam, (3) Minor changes in the training of hyper-parameters improve the baseline by a relatively large margin (mainly on FMoW), (4) There is a strong correlation between certain domains and certain target labels (mainly on iWildCam). To the best of our knowledge, no prior work on these datasets has reported these observations despite their obvious importance. Our code is public.