Ali Gholami

Saeid Asgari Taghanaki, Aliasghar Khani, Fereshte Khani et al. · stanford

A fundamental challenge of over-parameterized deep learning models is learning meaningful data representations that yield good performance on a downstream task without over-fitting spurious input features. This work proposes MaskTune, a masking strategy that prevents over-reliance on spurious (or a limited number of) features. MaskTune forces the trained model to explore new features during a single epoch finetuning by masking previously discovered features. MaskTune, unlike earlier approaches for mitigating shortcut learning, does not require any supervision, such as annotating spurious features or labels for subgroup samples in a dataset. Our empirical results on biased MNIST, CelebA, Waterbirds, and ImagenNet-9L datasets show that MaskTune is effective on tasks that often suffer from the existence of spurious correlations. Finally, we show that MaskTune outperforms or achieves similar performance to the competing methods when applied to the selective classification (classification with rejection option) task. Code for MaskTune is available at https://github.com/aliasgharkhani/Masktune.

Wele Gedara Chaminda Bandara, Naman Patel, Ali Gholami et al.

Masked Autoencoders (MAEs) learn generalizable representations for image, text, audio, video, etc., by reconstructing masked input data from tokens of the visible data. Current MAE approaches for videos rely on random patch, tube, or frame-based masking strategies to select these tokens. This paper proposes AdaMAE, an adaptive masking strategy for MAEs that is end-to-end trainable. Our adaptive masking strategy samples visible tokens based on the semantic context using an auxiliary sampling network. This network estimates a categorical distribution over spacetime-patch tokens. The tokens that increase the expected reconstruction error are rewarded and selected as visible tokens, motivated by the policy gradient algorithm in reinforcement learning. We show that AdaMAE samples more tokens from the high spatiotemporal information regions, thereby allowing us to mask 95% of tokens, resulting in lower memory requirements and faster pre-training. We conduct ablation studies on the Something-Something v2 (SSv2) dataset to demonstrate the efficacy of our adaptive sampling approach and report state-of-the-art results of 70.0% and 81.7% in top-1 accuracy on SSv2 and Kinetics-400 action classification datasets with a ViT-Base backbone and 800 pre-training epochs.

Saeid Asgari Taghanaki, Ali Gholami, Fereshte Khani et al. · stanford

Batch normalization (BN) is a ubiquitous technique for training deep neural networks that accelerates their convergence to reach higher accuracy. However, we demonstrate that BN comes with a fundamental drawback: it incentivizes the model to rely on low-variance features that are highly specific to the training (in-domain) data, hurting generalization performance on out-of-domain examples. In this work, we investigate this phenomenon by first showing that removing BN layers across a wide range of architectures leads to lower out-of-domain and corruption errors at the cost of higher in-domain errors. We then propose Counterbalancing Teacher (CT), a method which leverages a frozen copy of the same model without BN as a teacher to enforce the student network's learning of robust representations by substantially adapting its weights through a consistency loss function. This regularization signal helps CT perform well in unforeseen data shifts, even without information from the target domain as in prior works. We theoretically show in an overparameterized linear regression setting why normalization leads to a model's reliance on such in-domain features, and empirically demonstrate the efficacy of CT by outperforming several baselines on robustness benchmarks such as CIFAR-10-C, CIFAR-100-C, and VLCS.

Yingpin Chen, Zhenming Peng, Ali Gholami et al.

Time-frequency analysis has been applied successfully in many fields. However, the traditional methods, like short time Fourier transform and Cohen distribution, suffer from the low resolution or the interference of the cross terms. To solve these issues, we put forward a new sparse time-frequency analysis model by using the Lp-quasinorm constraint, which is capable of fitting the sparsity prior knowledge in the frequency domain. In the proposed model, we regard the short time truncated data as the observation of sparse representation and design a dictionary matrix, which builds up the relationship between the short time measurement and the sparse spectrum. Based on the relationship and the Lp-quasinorm feasible domain, the proposed model is established. The alternating direction method of multipliers (ADMM) is adopted to solve the proposed model. Experiments are then conducted on several theoretical signals and applied to the seismic signal spectrum decomposition, indicating that the proposed method is able to obtain a higher time-frequency distribution than state-of-the-art time-frequency methods. Thus, the proposed method is of great importance to reservoir exploration.