A Video-based Lifting Action Recognition Using Rank-Altered Kinematic Feature Pairs

Abstract

Low-back musculoskeletal disorders (MSDs) are the primary work-related injuries among manual material handling (MMH) workers, who are frequently exposed to repetitive lifting. To prevent low-back MSDs in the workplace, we present a video-based lifting action recognition method using rank-altered kinematic feature pairs, called top-scoring pairs (TSPs). We derive TSPs from a video dataset containing lifting and other activities commonly seen in MMH. These TSPs collectively classify each frame as lifting and non-lifting. The validation process involves evaluating classification performance. The proposed method minimizes computational and memory requirements while achieving performance comparable to more complex methods with greater computational demands. This makes it suitable for systems with limited hardware resources, thereby providing extensive feasibility across a variety of MMH environments to improve workplace safety.

Introduction

Between 2021 and 2022, 502,280 cases of work-related musculoskeletal disorders (MSDs) resulting in days away from work were reported in the U.S. (U.S. Bureau of Labor Statistics, 2023). Most of these cases occur among manual material handling (MMH) workers frequently exposed to repetitive lifting tasks. Epidemiological studies link repetitive lifting to an increased risk of low-back MSDs (Maher et al., 2017). From a biomechanical standpoint, repetitive lifting fatigues the back muscles and increases spinal loading, particularly on the L5/S1 intervertebral disc, leading to compression and shear forces (Marras et al., 2006). The L5/S1 disc also experiences significant bending moments during repetitive lifting due to the lumbar spine’s structure (Dolan & Adams, 1998). These stresses are key risk factors for chronic low-back pain and disc impairments, such as herniation (Desmoulin et al., 2020). Therefore, managing lifting frequency in the workplace is critical to prevent work-related low-back injuries.

In current safety evaluations, practitioners manually observe workers to identify lifting actions and determine frequencies, either through field surveys or video analysis. This process is time-consuming and labor-intensive, requiring real-time monitoring throughout shifts. Thus, developing an automated system for monitoring lifting tasks is essential. To this end, the first step is acquiring motion data from workers. With the widespread availability of cameras, computer vision-based neural networks offer a promising solution for collecting body motion data. However, technical challenges remain for workplace implementation. Current approaches often stack multiple convolutional layers to capture features from local to global scales (Meng et al., 2022). Despite their power, these networks are computationally and memory intensive, limiting their use on hardware-constrained systems (Han et al., 2015).

To address these limitations, we propose a video-based lifting action recognition method that reduces computational and memory demands while maintaining high classification accuracy. The method uses BlazePose (Bazarevsky et al., 2020), a lightweight CNN architecture, to detect 18 key body joints. Kinematic features are extracted from these joints, and top-scoring pairs (TSPs) form an ensemble classifier that identifies lifting actions by classifying each video frame as lifting or non-lifting. We validated the method against baseline classifiers using a video dataset of seven common MMH activities.

Method

To identify lifting actions in videos, we adopt a four-stage process: pose estimation, pre-processing, feature extraction, and classification. Figure 1 illustrates the overall workflow of the proposed method.

Figure 1.

Overall workflow of the proposed method.

Pose Estimation

Recently, Google researchers introduced BlazePose, a lightweight CNN architecture designed for real-time human pose estimation on mobile devices (Bazarevsky et al., 2020). In this study, we employed BlazePose for motion data collection due to its low latency and high accuracy. From the 33 joints detected by BlazePose, we selected 18 key joints to capture full-body movements for lifting action recognition (Figure 2).

Figure 2.

Selected key joints of the human body. These include shoulders, elbows, wrists, pinkies, index fingers, thumbs, hips, knees, and ankles.

Pre-processing

The initial step in data pre-processing involves smoothing each joint’s trajectory by filtering out high-frequency noise. We utilize a fifth-order Butterworth filter with a cutoff frequency of 3 Hz, given that typical walking frequencies range from 1.2to 2.2 Hz (Luo et al., 2021). After filtering, the joint trajectories become more precise, with reduced noise and improved accuracy. Next, we normalize the joint positions by centering and scaling: each joint is shifted by subtracting the mid-hip point and then scaled by dividing it by the average trunk length over a short time window.

Feature Extraction

From the pre-processed joints $\left\{x_1,y_1,x_2,y_2,\cdots ,x_{18},y_{18}\right\}$ , we extract kinematic feature vectors $\mathbf{x}={\left\{\mathbf{a},\mathbf{p},\mathbf{d},\mathbf{v}\right\}}^T$ using a sliding window $w$ . Each vector consists of joint angle $\mathbf{a}$ , joint position $\mathbf{p}$ , inter-joint distance $\mathbf{d}$ , and joint velocity $\mathbf{v}$ .

Classification

Consider a data matrix $\mathbf{X}=\left\{x_{ln}\right\}\in {ℝ}^{L\times N}$ , where each element $x_{ln}$ denotes the $n$ -th $(1\le n\le N)$ observation of a random variable $X_l$ for feature $l(1\le l\le L)$ . Each observation ${\mathbf{x}}_n={\left[x_{1n}{x}_{2n}\dots x_{Ln}\right]}^T$ has a class label $y_n\in \left\{1,2\right\}$ . The TSP algorithm is interested in finding a feature pair $(i,j)(1\le i,j\le L;i\ne j)$ for which there is a significant difference in the probability of $X_i<X_j$ between class 1 and 2. For a feature pair $\left(i,j\right)$ , the conditional ordering probability $p_{ij}\left(c\right)$ is defined as $p_{ij}\left(c\right):=P\left(X_i<X_j\mathrm{|}Y=c\right)$ for $c\in \{1,2\}$ . $p_{ij}\left(1\right)$ represents the probability of feature $j$ being greater than feature $i$ when $c=1$ , and similarly $p_{ij}\left(2\right)$ when $c=2$ . For the data matrix $\mathbf{X}$ , these two probabilities are estimated by the relative frequencies of occurrences of $X_i<X_j$ for class $c$ , that is,

$${\widehat{p}}_{ij}\left(c\right)=\frac{{\sum }_{n=1}^N\left[I(x_{in}<x_{jn})I(y_n=c)\right]}{{\sum }_{n=1}^{N}I(y_n=c)},$$

(1)

where $I(\cdot )$ is an indicator function that generates a value of one when the statement inside is true and zero otherwise. The score of a feature pair $(i,j)$ , which measures the discriminative power of the pair, is defined as ${\Delta }_{ij}:=\left|p_{ij}\left(1\right)-p_{ij}(2)\right|$ . To find the best feature pair $(i^{*},j^{*})$ , called the TSP, the TSP algorithm estimates the scores ${\Delta }_{ij}$ for every distinct pair and selects the pair with the highest score.

Suppose we derive feature pairs with scores comparable to the TSP from the data matrix $\mathbf{X}$ . The $k$ -TSP classifier $f_{k-\mathrm{TSP}}$ produces a prediction for a new observation ${\mathbf{x}}_{\mathrm{new}}={\left[x_{1\mathrm{new}}{x}_{2\mathrm{new}}\dots x_{L\mathrm{new}}\right]}^T$ as follows:

$${\widehat{y}}_{\mathrm{new}}=f_{k-\mathrm{TSP}}\left({\mathbf{x}}_{\mathrm{new}}\right)=\{\begin{array}{c}1,\mathrm{if}\sum _{u=1}^{k}I(f_{\mathrm{TSP}}^u\left({\mathbf{x}}_{\mathrm{new}}\right)=2)<c_k,\\ 2,\mathrm{if}\sum _{u=1}^{k}I(f_{\mathrm{TSP}}^u\left({\mathbf{x}}_{\mathrm{new}}\right)=2)\ge c_k.\end{array}$$

(3)

where $c_k$ is the number of votes required to classify the new observation as class 2. We determined the hyper-parameters $k$ and $c_k$ to be 9 and 3, respectively.

Results

For validation, we created a benchmark video dataset for lifting action recognition in MMH by conducting experiments involving 25 healthy participants. Each participant completed four sessions, during which they performed seven common MMH activities: pushing, pulling, sitting, standing, walking, and lifting.

Classification Performance

We applied the $k$ -TSP classifier to the MMH dataset above. The baseline classifiers were constructed using the kinematic features used to identify the TSPs. Table 1 shows the performance results of each classifier.

Table 1.

Performance Comparison. The Latency is the Run Time It Takes for a Classifier to Produce a Single Prediction.

Classifier	Accuracy	Precision	Recall	F1-Score	Latency (ms)
$k$ -TSP	0.89	0.89	0.93	0.91	0.062
RF	0.91	0.91	0.95	0.93	5.057
SVM	0.91	0.98	0.88	0.93	0.774
MLP	0.93	0.97	0.92	0.94	2.380
LSTM	0.97	0.98	0.98	0.98	311.188

Discussions

All classifiers achieved a performance of 0.88 or higher across all metrics. The $k$ -TSP classifier demonstrates superior latency by comparing only a small set of features for class prediction, leading to high computational efficiency. This efficiency is essential for real-time monitoring of lifting tasks on platforms with limited hardware resources, such as mobile devices and embedded systems. As a result, the proposed method offers a practical and cost-effective ergonomic solution for various MMH environments.

This study has several limitations. First, the use of 2D pose estimation may lead to inaccuracies in feature extraction due to the projection of joints onto a 2D plane. Second, the window length should be adjusted based on the frame rate of the videos being processed to maintain consistent temporal resolution for feature extraction. Third, a lift counting algorithm based on class predictions needs to be developed to count lifts and assess the risks associated with lifting frequency.