A Priori Generalizability Estimate for a CNN
This provides a diagnostic tool for researchers and practitioners to estimate CNN performance on individual samples, addressing reliability issues in image tasks, though it is incremental as it builds on existing singular value decomposition methods.
The paper tackles the problem of predicting when a convolutional neural network (CNN) will perform poorly on specific images by formulating truncated singular value decompositions of entire CNNs and defining Right and Left Projection Ratios as diagnostic metrics. The result shows that these ratios can identify class imbalance in image classification and that the Right Projection Ratio, using only unlabeled data, correlates with model performance in image segmentation, suggesting it can estimate sample-wise generalizability.
We formulate truncated singular value decompositions of entire convolutional neural networks. We demonstrate the computed left and right singular vectors are useful in identifying which images the convolutional neural network is likely to perform poorly on. To create this diagnostic tool, we define two metrics: the Right Projection Ratio and the Left Projection Ratio. The Right (Left) Projection Ratio evaluates the fidelity of the projection of an image (label) onto the computed right (left) singular vectors. We observe that both ratios are able to identify the presence of class imbalance for an image classification problem. Additionally, the Right Projection Ratio, which only requires unlabeled data, is found to be correlated to the model's performance when applied to image segmentation. This suggests the Right Projection Ratio could be a useful metric to estimate how likely the model is to perform well on a sample.