Actor-Centric Spatio-Temporal Feature Extraction for Action Recognition

Action understanding involves the recognition and detection of specific actions within videos. This crucial task in computer vision gained significant attention due to its multitude of applications across various domains. The current action detection models, inspired by 2D object detection methods, employ two-stage architectures. The first stage is to extract actor-centric video sub-clips, i.e. tubelets of individuals, and the second stage is to classify these tubelets using action recognition networks. The majority of these recognition models utilize a frame-level pre-trained 3D Convolutional Neural Networks (3D CNN) to extract spatio-temporal features of a given tubelet. This, however, results in suboptimal spatio-temporal feature representation for action recognition, primarily because the actor typically occupies a relatively small area in the frame. This work proposes the use of actor-centric tubelets instead of frames to learn spatio-temporal feature representation for action recognition. We present an empirical study of the actor-centric tubelet and frame-level action recognition models and propose a baseline for actor-centric action recognition. We evaluated the proposed method on the state-of-the-art C3D, I3D, and SlowFast 3D CNN architectures using the NTURGBD dataset. Our results demonstrate that the actor-centric feature extractor consistently outperforms the frame-level and large pre-trained fine-tuned models. The source code for the tubelet generation is available at https://github.com/anilkunchalaece/ntu_tubelet_parser.