Using a Binary Classification Approach to Assess the Accuracy of Hand Posture and Force Estimation with Machine Learning Models

Abstract

Recent studies have successfully reported the accuracy of using artificial neural networks to predict grip force in controlled settings. However, only relying on accuracy to evaluate the machine learning models may lead to overoptimistic results, especially on imbalanced datasets. The Matthews correlation coefficient (MCC) showed an advantage in capturing all the data characteristics in the confusion matrix. Therefore, a binary classification approach and the MCC value were introduced to assess the performance of previously proposed machine learning models. Our results show that the overall correlations ranging between 0.48 and 0.59 indicate a strong relationship between predictions and actual scenarios. The binary classification approach and the MCC values could be used for future performance comparison with other machine learning models.

Get full access to this article

View all access options for this article.

References

Chicco

Jurman

(2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC genomics, 21(1), 1–13.

Chicco

Tötsch

Jurman

(2021). The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData mining, 14(1), 1–22.

Choi

Kwon

Park

Lee

dong Kim

(2010). Real-time pinch force estimation by surface electromyography using an artificial neural network. Medical Engineering and Physics, 32(5), 429–436. https://doi.org/10.1016/j.medengphy.2010.04.004

Jiang

Chen

(2019). Force Estimation Based on sEMG using Wavelet Analysis and Neural Network. 9th International Conference on Information Science and Technology, ICIST 2019, 320–326. https://doi.org/10.1109/ICIST.2019.8836897

Lever

Krzywinski

Altman

(2016). Classification evaluation. Ma, R., Zhang, L., Li, G., Jiang, D., Xu, S., & Chen, D. (2020). Grasping force prediction based on sEMG signals. Alexandria Engineering Journal, 59(3), 1135–1147. https://doi.org/10.1016/j.aej.2020.01.007

Subasi

Yilmaz

Ozcalik

H. R.

(2006). Classification of EMG signals using wavelet neural network. Journal of Neuroscience Methods, 156(1–2), 360–367. https://doi.org/10.1016/j.jneumeth.2006.03.004

Vihinen

(2012). How to evaluate performance of prediction methods? Measures and their interpretation in variation effect analysis. BMC Genomics, 13 Suppl 4(Suppl 4). https://doi.org/10.1186/1471-2164-13S4-S2

Wang

Zhao

Barr

Kapellusch

Harris Adamson

(2020, December). Hand Posture and Force Estimation using Surface Electromyography and an Artificial Neural Network. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 64, No. 1, pp. 1247–1248). Sage CA: Los Angeles, CA: SAGE Publications.