The Omniglot challenge: a 3-year progress report
This is an incremental report assessing community progress on a benchmark for few-shot learning, highlighting gaps in achieving human-level performance.
The authors reviewed progress on the Omniglot dataset for one-shot learning over three years, finding notable improvements in one-shot classification but less advancement on other tasks, with current approaches still far from human-like concept learning.
Three years ago, we released the Omniglot dataset for one-shot learning, along with five challenge tasks and a computational model that addresses these tasks. The model was not meant to be the final word on Omniglot; we hoped that the community would build on our work and develop new approaches. In the time since, we have been pleased to see wide adoption of the dataset. There has been notable progress on one-shot classification, but researchers have adopted new splits and procedures that make the task easier. There has been less progress on the other four tasks. We conclude that recent approaches are still far from human-like concept learning on Omniglot, a challenge that requires performing many tasks with a single model.