Funnel Activation for Visual Recognition
This work addresses a bottleneck in activation functions for computer vision researchers, offering a simple yet effective enhancement for visual recognition tasks, though it is incremental as it builds on existing ReLU variants.
The authors tackled the problem of limited activation function expressiveness in visual recognition by introducing Funnel activation (FReLU), a 2D extension of ReLU and PReLU that adds spatial conditioning, resulting in improved performance on ImageNet, COCO detection, and semantic segmentation tasks.
We present a conceptually simple but effective funnel activation for image recognition tasks, called Funnel activation (FReLU), that extends ReLU and PReLU to a 2D activation by adding a negligible overhead of spatial condition. The forms of ReLU and PReLU are y = max(x, 0) and y = max(x, px), respectively, while FReLU is in the form of y = max(x,T(x)), where T(x) is the 2D spatial condition. Moreover, the spatial condition achieves a pixel-wise modeling capacity in a simple way, capturing complicated visual layouts with regular convolutions. We conduct experiments on ImageNet, COCO detection, and semantic segmentation tasks, showing great improvements and robustness of FReLU in the visual recognition tasks. Code is available at https://github.com/megvii-model/FunnelAct.