Robust hand motion tracking holds promise for improved human–machine interaction in diverse fields, including virtual reality, and automated sign language translation. However, current wearable hand motion tracking approaches are typically limited in detection performance, wearability, and durability. This article presents a hand motion tracking system using multiple soft polymer acoustic waveguides (SPAWs). The innovative use of SPAWs as strain sensors offers several advantages that address the limitations. SPAWs are easily manufactured by casting a soft polymer shaped as a soft acoustic waveguide and containing a commercially available small ceramic piezoelectric transducer. When used as strain sensors, SPAWs demonstrate high stretchability (up to 100%), high linearity (R2 > 0.996 in all quasi-static, dynamic, and durability tensile tests), negligible hysteresis (<0.7410% under strain of up to 100%), excellent repeatability, and outstanding durability (up to 100,000 cycles). SPAWs also show high accuracy for continuous finger angle estimation (average root-mean-square errors [RMSE] <2.00°) at various flexion-extension speeds. Finally, a hand-tracking system is designed based on a SPAW array. An example application is developed to demonstrate the performance of SPAWs in real-time hand motion tracking in a three-dimensional (3D) virtual environment. To our knowledge, the system detailed in this article is the first to use soft acoustic waveguides to capture human motion. This work is part of an ongoing effort to develop soft sensors using both time and frequency domains, with the goal of extracting decoupled signals from simple sensing structures. As such, it represents a novel and promising path toward soft, simple, and wearable multimodal sensors.
Get full access to this article
View all access options for this article.
References
1.
Maceira-ElviraP, PopaT, SchmidAC, et al.Wearable technology in stroke rehabilitation: Towards improved diagnosis and treatment of upper-limb motor impairment. J Neuroeng Rehabil, 2019; 16:1–18.
2.
RadiantiJ, MajchrzakTA, FrommJ, et al.A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Comp Educ, 2020; 147:103778.
3.
WangM, YanZ, WangT, et al.Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors. Nat Electron, 2020; 3:563–570.
4.
KudrinkoK, FlavinE, ZhuX, et al.Wearable sensor-based sign language recognition: A comprehensive review. IEEE Rev Biomed Eng, 2020; 14:82–97.
5.
ErolA, BebisG, NicolescuM, et al.Vision-based hand pose estimation: A review. Comput Vis Image Und, 2007; 108:52–73.
6.
TianJ, ZhangW, ZhangT, et al.Research status of gesture recognition based on vision: A review. IOP Conf Ser Earth Environ Sci, 2021; 632:042019.
7.
HammondFL, MengüçY, WoodRJ. Toward a Modular Soft Sensor-Embedded Glove for Human Hand Motion and Tactile Pressure Measurement. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE: Chicago, IL, 2014; pp. 4000–4007.
8.
JhaCK, AgarwalS, ChakrabortyAL, et al.An FBG-based sensing glove to measure dynamic finger flexure with an angular resolution of 0.1 deg up to speeds of 80 deg/s. J Lightwave Tech, 2019; 37:4734–4740.
9.
SaggioG. Mechanical model of flex sensors used to sense finger movements. Sensor Actuat A Phys, 2012; 185:53–58.
10.
ShenZ, YiJ, LiX, et al.A soft stretchable bending sensor and data glove applications. Robotics Biomim, 2016; 3:1–8.
11.
SirohiJ, ChopraI. Fundamental understanding of piezoelectric strain sensors. J Intell Mater Syst Struct, 2020; 11:246–257.
12.
HuB, DingT, PengY, et al.Flexible and attachable inertial measurement unit (IMU)-based motion capture instrumentation for the characterization of hand kinematics: A pilot study. Instrum Sci Technol, 2020; 49:125–145.
13.
DuanL, D'hoogeDR, CardonL. Recent progress on flexible and stretchable piezoresistive strain sensors: From design to application. Prog Mater Sci, 2020; 114:100617.
14.
TangZ, JiaS, WangF, et al.Highly stretchable core–sheath fibers via wet-spinning for wearable strain sensors. ACS Appl Mater Interfaces, 2018; 10:6624–6635.
15.
ShiG, ZhaoZ, PaiJH, et al.Highly sensitive, wearable, durable strain sensors and stretchable conductors using graphene/silicon rubber composites. Adv Funct Mater, 2016; 26:7614–7625.
16.
ParkYL, ChenBR, WoodRJ. Design and fabrication of soft artificial skin using embedded microchannels and liquid conductors. IEEE Sens J, 2012; 12:2711–2718.
17.
ChossatJB, ParkYL, WoodRJ, et al.A soft strain sensor based on ionic and metal liquids. IEEE Sens J, 2013; 13:3405–3414.
18.
TangH, BanerjeeH, JusufiA, et al.Wearable and stretchable strain sensors: Materials, sensing mechanisms, and applications. Adv Intel Sys, 2020; 2:2000039.
19.
ToC, HellebrekersTL, ParkYL. Highly Stretchable Optical Sensors for Pressure, Strain, and Curvature Measurement. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE: Hamburg, 2015; pp. 5898–5903.
20.
ZhaoH, O'BrienK, LiS, et al.Optoelectronically innervated soft prosthetic hand via stretchable optical waveguides. Sci Robot, 2016; 1(1):eaai7529.
21.
TeepleCB, BeckerKP, WoodRJ. Soft Curvature and Contact Force Sensors for Deep-Sea Grasping via Soft Optical Waveguides. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE: Madrid, 2018; pp. 1621–1627.
22.
WangD, ShengB, PengL, et al.Flexible and optical fiber sensors composited by graphene and pdms for motion detection. Polymers, 2019; 11:1433.
LeberA, CholstB, SandtJ, et al.Stretchable thermoplastic elastomer optical fibers for sensing of extreme deformations. Adv Func Mater, 2019; 29:1802629.
25.
CobosS, FerreM, UranMS, et al.Efficient Human Hand Kinematics for Manipulation Tasks. In: 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE: Nice, France; 2008; pp. 2246–2251.
26.
ChossatJB, ShullPB. Soft acoustic waveguides for strain, deformation, localization, and twist measurements. IEEE Sens J, 2020; 21:222–230.
27.
ZöllerG, WallV, BrockO. Acoustic Sensing for Soft Pneumatic Actuators. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE: Madrid, Spain; 2018; pp. 6986–6991.
28.
ZöllerG, WallV, BrockO. Active Acoustic Contact Sensing for Soft Pneumatic Actuators. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020; pp.7966–7972.
29.
YangX, YanJ, FangY, et al.Simultaneous prediction of wrist/hand motion via wearable ultrasound sensing. IEEE Trans Neural Syst Rehabil Eng, 2020; 28:970–977.
30.
YinZ, ChenH, YangX, et al.A wearable ultrasound interface for prosthetic hand control. IEEE J Biomed Health Inform, 2022; 26(11):5384–5393.
31.
JonesSB, WraithJM, OrD. Time domain reflectometry measurement principles and applications. Hydrol Process, 2002; 16:141–153.
32.
PapadakisEP. Ultrasonic phase velocity by the pulse-echo-overlap method incorporating diffraction phase corrections. J Acoust Soc Am, 1967; 42:1045–1051.
33.
SinhaM, BuckleyDJ. Acoustic Properties of Polymers. In: Physical Properties of Polymers Handbook (Mark JM. ed.). Springer: New York, USA; 2007; pp. 1021–1031.
34.
GentA, MartenyP. The effect of strain upon the velocity of sound and the velocity of free retraction for natural rubber. J Appl Phys, 1982; 53:6069–6075.
35.
KobayashiH, VanderbyR. Acoustoelastic analysis of reflected waves in nearly incompressible, hyper-elastic materials: Forward and inverse problems. J Acoust Soc Am, 2007; 121:879–887.
36.
JerriAJ. The shannon sampling theorem—Its various extensions and applications: A tutorial review. Proc IEEE, 1977; 65:1565–1596.
37.
VirtanenP, GommersR, OliphantTE, et al.Scipy 1.0: Fundamental algorithms for scientific computing in python. Nat Methods, 2020; 17:261–272.
38.
JohanssonM.The Hilbert Transform. Mathematics Master's Thesis. Växjö University: Suecia; 1999. Available from: https://w3.msi.vxu.se/exarb/mj_ex.pdf, consultado el, 19 [Last accessed: September 21, 2021].
39.
RoseJL. Physically Based Signal Processing Concepts for Guided Waves. Cambridge University Press: London, UK; 2014; pp. 456–477; doi: 10.1017/CBO9781107273610.026
40.
BagleyRL, TorvikPJ. On the fractional calculus model of viscoelastic behavior. J Rheology, 1986; 30:133–155.
41.
WuY, KarakurtI, BekerL, et al.Piezoresistive stretchable strain sensors with human machine interface demonstrations. Sens Actuat A: Phys, 2018; 279:46–52.
42.
ShiX, WangH, XieX, et al.Bioinspired ultrasensitive and stretchable mxene-based strain sensor via nacre-mimetic microscale “brick-and-mortar” architecture. ACS Nano, 2018; 13:649–659.
43.
JeongYR, ParkH, JinSW, et al.Highly stretchable and sensitive strain sensors using fragmentized graphene foam. Adv Funct Mater, 2015; 25:4228–4236.
44.
ChenF, AppendinoS, BattezzatoA, et al.Constraint study for a hand exoskeleton: Human hand kinematics and dynamics. J Robotics, 2013; 2013:910961.
45.
ParkY, LeeJ, BaeJ. Development of a wearable sensing glove for measuring the motion of fingers using linear potentiometers and flexible wires. IEEE Trans Indus Inform, 2014; 11:198–206.
46.
MEngKL, ChenIM, YeoSH, et al.Development of finger-motion capturing device based on optical linear encoder. J Rehabil Res Dev, 2011; 48:69–82.
47.
PfisterA, WestAM, BronnerS, et al.Comparative abilities of microsoft kinect and Vicon 3D motion capture for gait analysis. J Med Eng Tech, 2014; 38:274–280.
48.
MerriauxP, DupuisY, BoutteauR, et al.A study of Vicon system positioning performance. Sensors, 2017; 17:1591.
49.
ShullPB, XuJ, YuB, et al.Magneto-gyro wearable sensor algorithm for trunk sway estimation during walking and running gait. IEEE Sens J, 2016; 17:480–486.
50.
GarrettJW.Anthropometry of the Hands of Male Air Force Flight Personnel. Technical Report. Air Force Aerospace Medical Research Lab Wright-Patterson AFB: Dayton, OH; 1970.
51.
IngramJN, KördingKP, HowardIS, et al.The statistics of natural hand movements. Exp Brain Res, 2018; 188:223–236.
52.
CarpinellaI, JonsdottirJ, FerrarinM. Multi-finger coordination in healthy subjects and stroke patients: A mathematical modelling approach. J Neuroeng Rehabil, 2011; 8:1–20.
53.
CockcroftJ, LouwQ, BakerR. Proximal placement of lateral thigh skin markers reduces soft tissue artefact during normal gait using the conventional gait model. Com Methods Biomech Biomed Eng, 2016; 19:1497–1504.
54.
CappozzoA, Della CroceU, LeardiniA, et al.Human movement analysis using stereophotogrammetry: Part 1: Theoretical background. Gait Posture, 2005; 21:186–196.
CobosS, FerreM, Ángel Sánchez-UránM, et al.Human hand descriptions and gesture recognition for object manipulation. Com Methods Biomech Biomed Eng, 2010; 13:305–317.
58.
ChossatJB, TaoY, DuchaineV, et al.Wearable Soft Artificial Skin for Hand Motion Detection with Embedded Microfluidic Strain Sensing. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), Washington, USA; 2015; pp. 2568–2573.
59.
XuanX, KimJY, HuiX, et al.A highly stretchable and conductive 3D porous graphene metal nanocomposite based electrochemical-physiological hybrid biosensor. Biosens Bioelectron, 2018; 120:160–167.
60.
LiH, ChenJ, ChangX, et al.A highly stretchable strain sensor with both an ultralow detection limit and an ultrawide sensing range. J Mater Chem A, 2021; 9:1795–1802.
61.
YuJ, WangM, DangC, et al.Highly stretchable, transparent and conductive double-network ionic hydrogels for strain and pressure sensors with ultrahigh sensitivity. J Mater Chem C, 2021; 9:3635–3641.
62.
WangY, HaoJ, HuangZ, et al.Flexible electrically resistive-type strain sensors based on reduced graphene oxide-decorated electrospun polymer fibrous mats for human motion monitoring. Carbon, 2018; 126:360–371.
63.
ZhengH, LinN, HeY, et al.Self-healing, self-adhesive silk fibroin conductive hydrogel as a flexible strain sensor. ACS Appl Mater Interfaces, 2021; 13(33):40013–40031.
64.
XuH, LvY, QiuD, et al.An ultra-stretchable, highly sensitive and biocompatible capacitive strain sensor from an ionic nanocomposite for on-skin monitoring. Nanoscale, 2019; 11(4):1570–1578.
65.
HuX, YangF, WuM, et al.A super-stretchable and highly sensitive carbon nanotube capacitive strain sensor for wearable applications and soft robotics. Adv Mater Tech, 2022; 7(3):2100769.
66.
YaoS, ZhuY. Wearable multifunctional sensors using printed stretchable conductors made of silver nanowires. Nanoscale, 2014; 6(4):2345–2352.
67.
CohenDJ, MitraD, PetersonK, et al.A highly elastic, capacitive strain gauge based on percolating nanotube networks. Nano Lett, 2012; 12(4):1821–1825.
68.
XingZ, LinJ, McCoulD, et al.Inductive strain sensor with high repeatability and ultra-low hysteresis based on mechanical spring. IEEE Sens J, 2020; 20(24):14670–14675.
69.
GuoJ, ZhouB, ZongR, et al.Stretchable and highly sensitive optical strain sensors for human-activity monitoring and healthcare. ACS Appl Mater Interfaces, 2019; 11(37):33589–33598.
70.
TangZ, YuH, YangH, et al.Effect of velocity and acceleration in joint angle estimation for an EMG-Based upper-limb exoskeleton control. Comput Biol Med, 2022; 141:105156.
71.
RoK, KimS, ParkW, et al.Development of a Wearable Soft Sensor System for Measuring Finger Motions. In: 2016 16th International Conference on Control, Automation and Systems (ICCAS). IEEE: Gyeongju, Korea (South); 2016; pp. 549–554.
72.
LedbetterH. Sound velocities, elastic constants: Temperature dependence. Mater Sci Eng A, 2006; 442(1–2):31–34.
LiuH, ZhangH, HanW, et al.3D printed flexible strain sensors: From printing to devices and signals. Adv Mater, 2021; 33(8):2004782.
75.
RathodVT. A review of acoustic impedance matching techniques for piezoelectric sensors and transducers. Sensors, 2020; 20:4051.
76.
RoseJL. A baseline and vision of ultrasonic guided wave inspection potential. J Pressure Vessel Technol, 2002; 124:273–282.
77.
ChoY. Estimation of ultrasonic guided wave mode conversion in a plate with thickness variation. IEEE Trans Ultrason Ferroelectr Freq Control, 2000; 47:591–603.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
