Overhead-MNIST: Machine Learning Baselines for Image Classification
This provides a practical baseline for selecting image classification algorithms for mission-critical satellite imaging systems, though it is incremental as it applies existing methods to a new dataset.
The paper established baseline performance metrics for image classification on the Overhead-MNIST satellite imagery dataset by evaluating 23 machine learning algorithms, finding that a convolutional neural network achieved the best accuracy of 0.965 on unseen test data.
Twenty-three machine learning algorithms were trained then scored to establish baseline comparison metrics and to select an image classification algorithm worthy of embedding into mission-critical satellite imaging systems. The Overhead-MNIST dataset is a collection of satellite images similar in style to the ubiquitous MNIST hand-written digits found in the machine learning literature. The CatBoost classifier, Light Gradient Boosting Machine, and Extreme Gradient Boosting models produced the highest accuracies, Areas Under the Curve (AUC), and F1 scores in a PyCaret general comparison. Separate evaluations showed that a deep convolutional architecture was the most promising. We present results for the overall best performing algorithm as a baseline for edge deployability and future performance improvement: a convolutional neural network (CNN) scoring 0.965 categorical accuracy on unseen test data.