HandReader: Advanced Techniques for Efficient Fingerspelling Recognition
Abstract
AbstractFingerspelling is a significant component of Sign Language (SL), allowing the interpretation of proper names, characterized by fast hand movements during signing. Although previous works on fingerspelling recognition have focused on processing the temporal dimension of videos, there remains room for improving the accuracy of these approaches. This paper introduces HandReader, a group of three architectures designed to address the fingerspelling recognition task. HandReaderRGB employs the novel Temporal Shift-Adaptive Module (TSAM) to process RGB features from videos of varying lengths while preserving important sequential information. HandReaderKP is built on the proposed Temporal Pose Encoder (TPE) operating on keypoints as tensors. Such keypoint composition in a batch allows the encoder to pass them through 2D and 3D convolution layers, utilizing temporal and spatial information and accumulating keypoint coordinates. We also introduce HandReaderRGB+KP — architecture with a joint encoder to benefit from RGB and keypoint modalities. Each HandReader model possesses distinct advantages: on the ChicagoFSWild and ChicagoFSWild+ datasets, HandReader models achieve the best results among methods trained on a single training set, including a new state-of-the-art result on ChicagoFSWild+.