Lightweight Transformer-Based Efficient Two-Step Network for Temporal Action Segmentation in Ultralow Frame Rate Excavator Work Videos at the Construction Site

Abstract

In this paper, we propose a novel lightweight Transformer-based architecture for Temporal Action Segmentation (TAS) in an ultralow frame rate video setting, called a Transformer-Based Efficient Two-Step Network (ETSNFormer). This study makes three main contributions to the literature. First, we demonstrate that the choice of the temporal window size for feature extraction significantly affects the segmentation performance in an ultralow frame rate video setting. Second, we enhance the Efficient Two-Step Network (ETSN) baseline by integrating a modified Transformer-based decoder block, yielding improved segmentation accuracy while using fewer computational resources. Third, we employ a genetic algorithm-based hyperparameter-Tuning approach to automatically tune the hyperparameters of Local Burr Suppression (LBS), which is the post-processing method used in the ETSN baseline to mitigate the over-segmentation problem. On our ultralow frame rate excavator video dataset, ETSNFormer achieved state-of-The-Art results while using fewer parameters than prior approaches: A frame-wise accuracy of 89.91%, an edit score of 72.09%, and segmental F1 scores of 79.11%, 78.42%, and 73.51% at thresholds of 0.1, 0.25, and 0.5, respectively.