Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization
This reveals a critical flaw in widely-used benchmarks for virtual screening, potentially misleading researchers in drug discovery by rewarding memorization over generalization.
The paper tackled the problem of overfitting in ligand-based classification benchmarks by introducing AVE, a measure of training-validation redundancy, and found that performance of methods strongly correlates with this bias across seven benchmarks, suggesting reported successes may be due to overfitting rather than true generalization.
Undetected overfitting can occur when there are significant redundancies between training and validation data. We describe AVE, a new measure of training-validation redundancy for ligand-based classification problems that accounts for the similarity amongst inactive molecules as well as active. We investigated seven widely-used benchmarks for virtual screening and classification, and show that the amount of AVE bias strongly correlates with the performance of ligand-based predictive methods irrespective of the predicted property, chemical fingerprint, similarity measure, or previously-applied unbiasing techniques. Therefore, it may be that the previously-reported performance of most ligand-based methods can be explained by overfitting to benchmarks rather than good prospective accuracy.