Wild Guesses and Mild Guesses in Active Concept Learning
This research addresses the trade-off between informativeness and stability in active learning for human-like concept learning, with implications for cognitive science and AI, though it is incremental as it builds on existing neuro-symbolic and Bayesian frameworks.
The study investigated active concept learning by comparing a Rational Active Learner using expected information gain (EIG) with a human-like Positive Test Strategy (PTS) in tasks like the Number Game, finding that EIG excels for complex rules but underperforms on simple concepts due to a support-mismatch trap, while PTS leads to faster convergence on simple rules by maintaining proposal validity.
Human concept learning is typically active: learners choose which instances to query or test in order to reduce uncertainty about an underlying rule or category. Active concept learning must balance informativeness of queries against the stability of the learner that generates and scores hypotheses. We study this trade-off in a neuro-symbolic Bayesian learner whose hypotheses are executable programs proposed by a large language model (LLM) and reweighted by Bayesian updating. We compare a Rational Active Learner that selects queries to maximize approximate expected information gain (EIG) and the human-like Positive Test Strategy (PTS) that queries instances predicted to be positive under the current best hypothesis. Across concept-learning tasks in the classic Number Game, EIG is effective when falsification is necessary (e.g., compound or exception-laden rules), but underperforms on simple concepts. We trace this failure to a support mismatch between the EIG policy and the LLM proposal distribution: highly diagnostic boundary queries drive the posterior toward regions where the generator produces invalid or overly specific programs, yielding a support-mismatch trap in the particle approximation. PTS is information-suboptimal but tends to maintain proposal validity by selecting "safe" queries, leading to faster convergence on simple rules. Our results suggest that "confirmation bias" may not be a cognitive error, but rather a rational adaptation for maintaining tractable inference in the sparse, open-ended hypothesis spaces characteristic of human thought.