Sage Journals: Discover world-class research

Abstract

We propose to perform wearable sensors-based human physical activity recognition. This is further extended to an Internet-of-Things (IoT) platform which is based on a web-based application that integrates wearable sensors, smartphones, and activity recognition. To this end, a smartphone collects the data from wearable sensors and sends it to the server for processing and recognition of the physical activity. We collect a novel data set of 13 physical activities performed both indoor and outdoor. The participants are from both the genders where their number per activity varies. During these activities, the wearable sensors measure various body parameters via accelerometers, gyroscope, magnetometers, pressure, and temperature. These measurements and their statistical are then represented in features vectors that used to train and test supervised machine learning algorithms (classifiers) for activity recognition. On the given data set, we evaluate a number of widely known classifiers such random forests, support vector machine, and many others using the WEKA machine learning suite. Using the default settings of these classifiers in WEKA, we attain the highest overall classification accuracy of 90%. Consequently, such a recognition rate is encouraging, reliable, and effective to be used in the proposed platform.

Keywords

Wearable Internet-of-Things activity recognition health and well-being accelerometer gyroscope magnetometer temperature sensor pressure sensor

Introduction

The Internet of Things (IoT) in recent years has gained significant importance in daily life. The IoT combines sensors, actuators, and communication networks to allow sensing and collecting information from the environment and human body for further computing and processing.^1,2 The proliferation and changing trends of smartness in every physical object enables human-centric pervasive application development to facilitate people’s daily lives. Diverse and powerful embedded sensors make a ubiquitous platform to acquire and analyze data. This provides greater potential for efficient resource management and utilization, and for the ability to monitor human activities for health and wellness.³ The lightweight and miniature size of devices allow them to be worn and carried on the move. In recent years, wearable sensors have emerged in various fields, such as entertainment, medicine, and security, changing the IoT trend to the wearable IoT (WIoT).⁴ Mobile, wearable, and IoT devices provide a large amount of important personal data and help elucidate the context of the user.⁵ Mining of this big data provides new approaches to measure and/or track welfare problems. Wearable sensors provide reliable and accurate information on human activities and behavior to ensure a sound and safe living environment.⁶ The WIoT permits observing, tracking, and measuring individual functions in daily life. This article extends this line of application by proposing a wearable sensor IoT platform to acquire the data and automatically recognize activities.

Physical activity is the movement of body produced by skeletal muscle contractions resulting in energy expenditure above the resting level.⁷ Human physical activities play a major role in a human being’s mental and physical health. Lack of physical activity can negatively affect physical fitness.⁸ Recent studies show that the physical inactivity (aside from poor nutrition, smoking, and use of alcohol) is a significant cause of premature death.⁹ This problem can be alleviated significantly with the help of physical activity tracking platforms. To this end, the data acquired from them can be used in recognizing activities. Consequently, with such tracking, the health issues related to physical inactivity can be reduced significantly. This concept can also be extended to other healthcare application areas such as tracking the patient or an elderly person to determine whether they are performing prescribed activities.

Based on the type of data, the physical activity recognition and tracking systems can broadly be divided into two classes. First is the video/image-based^10,11 systems that use images or videos to recognize and track the physical activities. The second type relies on data generated by wearable sensor.^12,13 The image-based systems face challenges due to image variations caused by occlusion, background clutter, and various object deformations. Apart from that, video- and image-based systems may cause the problem of privacy invasion.⁵ However, wearable-sensors-based systems do not face such challenges as they are worn closely on the body without any visual contact with the user. Due to this reason, the data generated by these sensors are relatively accurate, reliable, and non-intrusive than images.

A human physical activities recognition (HPAR) system can be used to recognize and track activities of various magnitudes such as short, simple, and complex. A short activity is one with a short duration or a transition from one state of activity to another, such as a change from standing to sitting. Simple activities, such as walking, standing, and reading, are periodic in nature and can be recognized using a single sensor. Complex activities, such as cooking and presentations, are non-periodic in nature, and multiple sensors are required for their accurate recognition. Recognizing complex activities is a challenging task due to similarity, concurrency, and their interleaved nature. Two different activities can have a similar nature (such as sitting, compared to reading while sitting). Concurrent activities are when a person can perform more than one activity at the same time, such as standing while brushing teeth. Interleaved activities are defined as ones where an interruption can occur when someone is performing an activity (e.g. someone who is running might stop to drink some water, and after drinking, starts running again). Our proposed platform has the capability to recognize physical activities ranging from simple to complex.

HPAR is divided into several steps.^13–15 First, either one sensor or a combination of body-worn, wrist-worn, and smartphone motion sensors collect activity time-series data. Various sensors, such as the tri-axil accelerometer, the gyroscope, and the global positioning system (GPS), are used at sampling rates from 1 to 100 Hz to collect data. In some cases, for signal noise removal, pre-processing (like Kalman filtering) is performed on the acquired data. The time-series data are segmented using a segment window size of 1 to 30 s for feature extraction.^14,16,17 The segmented data are used to extract various time-domain features, such as the mean, variance, and root mean square values, as well as the frequency domain, such as the magnitude of the highest frequency. Feature data with labels are used to train supervised learning algorithms (classifiers). More recently deep learning has also been used to recognize activities for problem like health care.¹⁸ Activities can be recognized in two different ways. One approach is the client/server model^19,20 with any worn device or smartphone locally used for data collection and activity recognition.^20,21 In the client/server architecture, the mobile phone is used for data collection and functions as a client node. It collects the data from mobile phone sensors or collects body-worn sensor data through Bluetooth. Collected data are sent to the server using the Internet, either in raw form or in some pre-processed format. At the server, features are extracted from the data and a trained classifier is used to classify the activity. The mobile-based local approach uses the smartphone for data acquisition, pre-processing, feature extraction, and classification. In the local scheme, the classifier is trained either offline or online. In offline training, the classifier is trained on a desktop PC and then used on the mobile device for activity recognition.

Concretely, following are the main challenges in our proposed platform:

Collection of various physical activities data from subjects of different ages and genders using wearable sensors.

Representation of data acquired from wearable sensors in the form of feature vectors and physical activity classification using these features.

Deployment of a real-time framework to asses and recognize users’ physical activities.

In face of the above-mentioned challenges, following are our contributions:

We collect a novel data set named “CUI-HPAR-Dataset” of physical activities performed by participants of various ages and genders. These activities range from simple to complex where some of them are performed outdoors such as presentation and laundry folding, whereas others are performed indoors such as cooking and reading.

The collected data are then represented in feature vectors that are calculated using various statistical measures of the input data such as mean and standard deviation. A detailed performance evaluation of supervised learning-based classifier is then performed on this data to determine the one with best recognition rate.

Finally, we propose a novel real-time and web-based framework that collects the data from wearable sensors and sends it to the server via a smartphone. Based on the received data, the activity is recognized and sent to the user.

The rest of this article is organized as follows. Section “Background and related works” briefly introduces the background and related works on the WIoT, activity data collection using ambient sensors, on-body wearable sensors and smartphone sensors, feature extraction, and recognition algorithms. The proposed WIoT platform, data collection, feature extraction, and classifiers are explained in section “The proposed WIoT platform for HPAR.” Section “Simulation results and discussion” introduces the properties of the acquired data set and includes a performance comparison of various classifiers using different features with different window sizes. Finally, we conclude the proposed work in section “Conclusion.”

Background and related works

This section briefly introduces the background and related work about the platforms used to collect the sensor data, kind of sensors, sensors placement, features extracted from raw data, recognized activities, and classifiers used for recognition.

Platforms and sensors used for HPAR

HPAR can be recognized either using ambient or body-worn sensors. The ambient sensors used for activity recognition include a camera-based platform,⁹ a passive infrared sensor,^22,23 an ultrasonic sensor,²³ radio frequency identification,²⁴ and WiFi- and vision-based sensors.²⁵ These sensors are installed in offices and at home. Ambient sensor-based HPAR has the drawback of being static in nature and cannot perform outside of the installed area. However, wearable sensors can overcome this problem. Kumari et al.²⁶ proposed the WIoT architecture that interfaced the accelerometer, gyroscope, and magnetometer sensors with particle photon board. Using WiFi interface the particle photon board is connected to Raspberry Pi to collect the data and send it to ThingSpeak cloud for storage. They used smartphone to start recording and control the storage (sampling). However, they did not provide any statistical analysis and placement of sensor module on body. Also, Raspberry Pi and particle photon both have WiFi connectivity so the architecture can be minimized components-wise using only one device to transfer data to the cloud. Gia et al.²⁷ proposed energy-efficient WIoT sensor node for fall detection and analyzed energy consumption considering various parameters such as sampling rate, communication interfaces, transmission rate, and transmission protocols. They showed that Bluetooth (Bluetooth Low Energy) consume about 70% less energy than WiFi. A remote patient monitoring system is proposed to track a person’s physiological data to detect specific disorders which can aid in Early Intervention Practices.²⁸ Most people are using smartphones, and the number is increasing day by day. Modern smartphones and smart watches are equipped with sensors such as a gyroscope, an accelerometer, a magnetometer, and a GPS. The Android™ smartphone is the most common platform used to collect activities data. Table 1 provides detailed insight into mobile platforms and wearable sensors for HPAR.

Table 1.

Mobile and wearable sensor platform for human physical activity recognition.

Reference	Platforms	Sensors used for HPAR
Ignatov¹³ and Bayat et al.²⁹	Android Smartphone	Triaxial accelerometer
Anjum and Ilyas³⁰ and Ronao and Cho³¹	Android Smartphone Y	Triaxial accelerometer, three-axis gyroscope
Sztyler et al.¹⁶	Samsung Galaxy S4, LG G Watch R	Accelerometer, GPS, three-axis gyroscope, Light, Magnetic field, Sound level
Dao et al.³²	Samsung Galaxy S3/iPhone 6S	Accelerometer, gyroscope, orientation
Ceja³³	LG Optimus Me with Android 2.2	Accelerometer
Lee and Kim³⁴	LG Optimus Pro	MPU-6050 Triaxial accelerometer
Cao et al.³⁵	Samsung Galaxy SII	Triaxial accelerometer and gyroscope
Cvetković et al.³⁶	Samsung Galaxy (SII and S4) and Wrist-Band (Microsoft Band, and Empatica E4)	Triaxial accelerometer and gyroscope

HPAR: human physical activities recognition; GPS: global positioning system.

Sensor placement for HPAR

Sensor placement on the human body mainly includes relative position to the body and orientation of the sensor. In the literature on HPAR, single or multiple sensors are attached or worn on the body in a predefined position and orientation. The number of activities recognized and the accuracy increases as the number of sensors increases at various positions.³⁷ The research shows that the best locations for sensors to recognize an activity depend on the activity performed. Bao and Intille³⁸ stated that placing accelerometers on the hip, thigh, and ankle are good indicators for activities that include some kind of ambulation, movement, or posture, while placement of accelerometers on the wrist, elbow, and arm are good indicators for activities that mostly involved the upper body. Cleland et al.³⁹ placed six triaxial accelerometers at six different positions and found that the sensors installed at the hip had higher recognition accuracy, compared to other positions. Figure 1 shows a graphical illustration of sensors placed for HPAR, as found in the literature.

Figure 1.

Graphical illustration of sensor placement for HPAR from the literature review.

Features computation, extraction, and selection for HPAR

In HPAR, the recognition step is generally preceded by features extraction from the acquired sensor data. Features are the characteristics of sensor signals in both the time and frequency domain. Features extraction is then performed on various length (1–30 s) segments of these signals. However, selecting significant features is an important and non-trivial step as increasing the number of features does not necessarily improve classification accuracy.⁴⁰ Sztyler et al.¹⁶ used features such as correlation, entropy, gravity, mean, absolute deviation, standard deviation, and Fourier transform. According to them, these features are helpful in finding the orientation information of the device by calculating angles. Li⁴¹ proposed a two-step process for feature selection where in the first step, features are extracted while, in the second step, unnecessary features are eliminated by a specialized filter. Consequently, 37 initially selected features are reduced to just seven. Bayat et al.²⁹ employ a low-pass filter to remove the DC component and then perform the features extraction. They extract various features such as mean, MinMax, standard deviation (STD), average peak frequency, and root mean square. Table 2 shows the features used for activity recognition, as described in the literature.

Table 2.

Literature summary of physical activities, selected features, window size for feature extraction, and classifiers for HPAR.

References	Activities	Selected features	Windowsize (s)	Classifiers with accuracy
Sztyler et al.¹⁶	A1, A2, A3, A4, A5,A6, A7, A9	Correlation, entropy, gravity, mean, mean absolute deviation, STD, variance, Fourier	1	RF: 95%, NB: 89%, KNN: 92%, ANN: 93%, SVM: 92%, DT: 93%
Bayat et al.²⁹	A1, A2, A3, A4, A7, A8	Average of Peak Frequency (APF), Variance Average of Peak Frequency (VAPF), Mean, MinMax, Standard Deviation (STD), Root Mean Square (RMS), Correlation	1.28	Multilayer Perceptron (MLP): 89.48%–89.72%, SVM: 72.27%–88.76%, RF: 85.15%–87.55%, LMT: 85.04%–85.89%, Simple Logistic: 85.05%–85.41%, Logic Boost: 82.24%–82.54%
Dao et al.³²	A1, A2, A3, A4, A5, A6	Mean, Median, STD, Skew, kurtosis, Entropy, Zero crossing, Min, Max, spectral Centroid, Spectral Flux, Slope, Short time energy	10	J48:, MLP:, Logistic Regression: (accuracy is provided separately for each activity.)
Ceja³³	A1, A2, A3, A4, A7, A12	Mean, STD, Correlation, Derivatives mean	2	KNN: 70%–90%, HMM: 93%–98.5%, Decision Tree: 90.8%, Bag of Features: 92.7%, Bayesian Classifier 1.5% error rate, Emerging Patterns 78.5%–90.96%, Topic Models: 72.7%
Li⁴¹	A1, A6, A7, A16, A17,A18, A19, A20	Average Increment, STD, Mean, RMS, Binned distribution, Correlation, Mean Cross, Simplified Energy	2	Decision Tree: 95%
Cherian⁴⁷	A19, A26, A27, A28, A29, A30	Mean, STD, Correlation, Number of peaks, RMS, Average Distance, Energy, Entropy	4	KNN (K = 5): 95.3%, C4.5: 96.1%, Multilayer Perceptron: 96.0%, Random Tree: 94.4%, RF: 96.5%
Köping et al.⁴⁸	A1, A5, A6, A7, A9, A10, A11,A12, A13, A14	Codebook based features	2.5	SVM: 87.1%
Shoaib et al.¹⁴ andShoaib et al.^17,49	A17, A18, A19, A21, A22,A23, A24, A25	Mean, STD, Median, Variance, Zero Crossing, RMS, Fourier Transform, Spectral Energy	2	Bayesian Networks, RF, NB, SVM, KNN, J48, Rule-based classifiers

HPAR: human physical activities recognition; RF: random forest; NB: Naïve Bayes; KNN: k-nearest neighbor; ANN: Artificial neural network; SVM: support vector machine; LMT: logistic model trees; HMM: hidden Markov model; DT: decision tree.

Activities: A1: Slow Walking, A2: Jogging, A3: Going up stairs, A4: Going down stairs, A5: Sitting, A6: Standing, A7: Fast walking, A8: Dancing, A9: Lying, A10: Bending, A11: Getting Up, A12: Laying Down, A13: Putting a hand back, A14: Stretching a hand, A15: Jumping, A16: Answering a phone call, A17: Typing, A18: Reading, A19: Drinking, A20: Eating, A21: Talking, A22: Smoking, A23: Biking, A24: Cooking, A25: Washing dishes, A26: Brushing teeth, A27: Combing hair, A28: Scratching the chin, A29: Washing hands, A30: Taking medication.

Classifiers for HPAR

Classifiers are supervised learning algorithms used to perform physical activity recognition. The most widely known and used classifiers are the k-nearest neighbors (KNNs)^50–52 the naive Bayes (NB),⁵³ decision trees,^54,55 random forests (RFs),⁵⁶ support vector machine (SVM),⁵⁷ and deep learning.⁵⁸ Sztyler et al.¹⁶ report better performance of RF. NB and KNN classifiers are used for online activity recognition on smartphones.⁵⁹ Bayat et al.²⁹ analyzed several single classifiers using the WEKA⁶⁰ such as multilayer perceptron (MLP), RF, logistic model trees, SVM, logistic regression, and Logit Boost. They also combined the classifiers and found that the combination has better accuracy compared to the single classifier. Lu et al.⁶¹ used RF, NB, SVM, and KNN classifiers for activity recognition, and according to them, RF is the preferred classifier. Weiss et al.⁶² also showed RF to perform better, while Wang et al.⁶³ got better performance with KNN. Table 2 shows a summary of features and classifiers used for activity recognition, as found in the literature.

The proposed WIoT platform for HPAR

In this section, we briefly describe the proposed platform, the acquiring of physical activity data from sensors, features vector construction, and the classifiers used for physical activity recognition.

Hardware and software of the proposed HPAR framework

Figure 2(a) depicts hardware of the proposed framework for HPAR that consists of three main components. The first is a sensing device with Bluetooth communication link to get connected with a smartphone for data transmission. In this article, we used the Metawear MetaMotion R⁶⁴ module that includes several sensors such as triaxial accelerometer, gyroscope, magnetometer, temperature, and pressure. The second component of the hardware is a smartphone that collects the data from sensor and transmits it to a web-server over the Internet. It also receives the physical activity recognition and analysis report from web-sever over the Internet for user’s information and monitoring. The last component of hardware is the web-server itself which receives the physical activity data from the smartphone, recognizes, and analyzes this data and then sends the feedback to user’s smartphone via the Internet.

Figure 2.

An overview of the (a) hardware and (b) software architecture for human physical activities recognition.

Figure 2(b) shows the software architecture of the proposed framework that consists of two main parts. The first part is based on PHP scripts that are used to transfers the data between a smartphone and web server. The second component is the recognition web server (Bottle server) interfaced with MySQL Database through Python. It receives the time-series data of the sensors, extracts various features from the data, and uses the trained classifiers and labels to recognize a physical activity which is then stored on the server for further use and analysis.

Data collection in the HPAR

In this article, we collected data from the sensor modules installed on various body locations such as wrists and arms. Table 3 shows the technical specifications of various individual sensors in a module such as sampling frequency, quantization levels, and range. The data were collected in both indoor and outdoor environments from participants of both the genders for 13 various activities. Since activity parameters vary with an individual’s fitness, age, height, weight, and physical movements, the participants for each activity were chosen in such a way that they vary from one another based on these parameters. For each activity, the number of participants vary from 6 to 19. The ages of participants are $27 \pm 8 years$ , weights are $54 \pm 10 kg$ , and heights are $165 \pm 15 cm$ . Table 4 summarizes the physical activities and the information about the participants. However, it should be noted that the input about discomfort caused by the wearable sensors is not taken from the participants. We see that as a limitation of the current framework that will further be improved in the future research.

Table 3.

Sensors configurations for data collection.

Sensorsname	Sampling frequency(Hz)	Quantizationlevel (bits)	Range
Accelerometer(x, y, z)	50	16	$\pm 16 gs$
Gyroscope(x, y, z)	50	16	$\pm 2000 ° / s$
Magnetometer(x, y, z)	25	16	$\pm 1300, \pm 2500$
Temperature	1	16	$- 40 ° C to + 85 ° C$
Pressure	1	16	$300 to 1100 hPa$

Table 4.

Information about the subjects who participated in various physical activities data collection.

Activity	Label	Subjects(male andfemale)	Activity durationperformed/person(min)
Writing	$A 1$	4, 4	6.5
Walking	$A 2$	6, 7	6.5
Typing	$A 3$	6, 11	Minimum: 4, Maximum: 6.5
Standing	$A 4$	5, 3	6.5
Sitting	$A 5$	3, 5	6.5
Presentation	$A 6$	5, 7	6.5
Lying	$A 7$	0, 8	Minimum: 4, Maximum: 6.5
Ironing	$A 8$	0, 6	Minimum: 4, Maximum: 6.5
Folding laundry	$A 9$	1, 5	Minimum: 4, Maximum: 6.5
Eating	$A 10$	6, 13	Minimum: 3, Maximum: 6.5
Bending	$A 11$	1, 7	3
UpStairs	$A 12$	2, 6	3
DownStairs	$A 13$	2, 6	3

Feature extraction and selection in the proposed HPAR system

The extraction of features from sensor time-series data depends on the window size. To find the best window size, we divided the data into different segments using empirically selected window sizes of 2, 3, and 4. We calculated both the time-domain and frequency-domain features of the activities data. Details of the features are as follows:

Mean value: the mean value is calculated for accelerometer (x, y, z), magnetometer (x, y, z), temperature, and pressure. In features vector, eight features are the mean values of sensors

μ = \frac{1}{N} \sum_{i = 0}^{N - 1} x_{i}

(1)

where $N$ is the number of samples which is equal to the segment(window) size, and $x$ is the sensor sample output.

Standard deviation (STD): it finds the spread in the sensors data around the mean value. It can be calculated as

σ = \sqrt{\frac{1}{N} \sum_{i = 0}^{N - 1} {(x_{i} - μ)}^{2}}

(2)

we calculate the STD of accelerometer (x, y, z) and magnetometer (x, y, z).

Entropy: entropy is useful to differentiate between the activities who have similar power spectral density but having different patterns of movement.⁶⁵ Equation (3) finds the entropy of accelerometer along each axis

Entropy = - \frac{1}{N} \sum_{i = 0}^{N - 1} p_{i} \log p_{i}

(3)

Cross correlation: we find the correlation between the accelerometer axes such as $Corr (x, y)$ , $Corr (x, z)$ , and $Corr (y, z)$ . The correlation helps to differentiate between the activities such as walking form stairs up and stairs down. Equation (4) finds the correlation between axes

Corr (x, y) = \frac{Cov (x, y)}{σ_{x} σ_{y}}

(4)

where

Cov (x, y) = \frac{\sum_{i = 0}^{N - 1} (x_{i} - μ_{x}) (y_{i} - μ_{y})}{N - 1}

RMS: the $RMS$ value is calculated for the gyroscope (x, y, z). Equation (5) finds the $RMS$

RMS = \sqrt{\frac{1}{N} \sum_{i = 0}^{N - 1} x_{i}^{2}}

(5)

Zero-crossing (ZC): Chang et al.⁶⁶ defined ZC is the number of times the signal crosses zero and its sign is changed. In this article, we consider ZC for the accelerometer along three axes. Mathematically, it can be written by equation (6)

\begin{matrix} ZC = COUNT ({(x_{i} > 0) AND (x_{i + 1} < 0)} OR {(x_{i} < 0) AND \\ (x_{i + 1} > 0)}), 0 < = i < = N - 1 \end{matrix}

(6)

Maximum value: we calculate the maximum value of accelerometer (x, y, z). Equation (7) find the maximum value

Acc_\max = \max (x_{i}), 0 \leq i \leq N - 1

(7)

Frequency-domain features: in this article, we consider six frequency-domain features based on the fast Fourier transform (FFT) of acceleration data. Since it is double-sided, first we shift the FFT. The six features are the FFT magnitude: $peak_f$ , $low_f 1$ , $low_f 2$ , $low_f 3$ , $med_f$ , and $high_f$ considering 3 s window with sampling frequency $50 Hz$ . Equation (8) finds the FFT

H (k) = \sum_{n = 0}^{N - 1} x (n) e^{- j 2 π (k \frac{n}{N})}

(8)

peak_f = \max (H (k))

(9)

low_fj = \sum_{i = \frac{k}{2} + l + 1}^{\frac{k}{2} + j * 5} H (i), l = 0, 5, 10, j = 1, 2, 3

(10)

med_f = \sum_{i = \frac{k}{2} + 16}^{\frac{k}{2} + 30} H (i)

(11)

high_f = \sum_{i = \frac{k}{2} + 36}^{\frac{k}{2} + 75} H (i)

(12)

Equation (10) finds the $low_f 1$ , $low_f 2$ , and $low_f 3$ . To calculate $low_f 1$ , the $j = 0$ and $l = 1$ ; $low_f 2$ , the $j = 2$ and $l = 5$ ; and $low_f 3$ , the $j = 3$ and $l = 10$ . The whole process of features vector construction is summarized in Figure 3.

Figure 3.

Process of features vector construction. The time-series data are collected from various sensors. The statistical measurements of each sensor output are then calculated and summarized to make the final features vector.

Classification algorithms for the proposed HPAR

The task of HPAR is to label a given recorded activity from A1 to A13. In order to perform such labeling, we use supervised machine learning algorithm more commonly known as classifiers. First stage is the training, where the activities represented in the form of features vectors along with their labels are used to train the parameters of a given classifiers. Afterward, this trained model is evaluated by using it to predict the label a given test activity such that this particular test sample is disjoint from the training set.

A multitude of classification algorithms have been studied in the literature on HPAR. However, a universal classifier does not exist that outperforms all others for activity recognition.³⁵ The most common classifiers used are $KNN$ , $NB$ , $RF$ , $Bayesian$ $Networks$ , $SVM$ , $J 48$ , $Logistic$ $Regression$ , $Decision$ $Tree$ , and $ANN$ . Due to this reason, we propose to evaluate of a subset of these classifiers as shown in Table 5 using WEKA. The default settings were used for all the classifiers except for MLP, where the learning rate was changed to $0.1$ .

Table 5.

Classification algorithms in WEKA.

Classifier type	WEKA name	Notation(abbreviation)
Decision tree	Random Forest	RF
Decision tree	J48	J48
k-nearest neighbor	IBK	KNN
Rule-based	PART	PART
Multilayer perceptron	MultilayerPerceptron	MLP
Support vector machine	SMO with PearsonVII kernelfunction (PUK) kernel	SVM

Simulation results and discussion

In this section, we briefly investigate the performance analysis of window size on classifiers for HPAR. We considered the window size $(Window - 2) : 2 s$ , $(Window - 3) : 3 - s$ , and $(Window - 4) : 4 - s$ . The total data for all 13 activities are of $11.1172 h$ . Total instances for Window-2, Window-3, and Window-4 are 20,011, 13,337, and 10,002, respectively. We use classification accuracy, true positive rate (TPR), and precision as performance metrics. Mathematically, performance can be calculated with these equations

Classification Accuracy = \frac{T_{P} + T_{N}}{T_{P} + F_{P} + T_{N} + F_{N}}

(13)

Precision = \frac{T_{P}}{T_{P} + F_{P}}

(14)

We used 10-fold cross validation to evaluate the classifiers’ performance. In 10-fold cross validation, the feature data set is divided into 10 equal bins. Out of these bins, 10% are used for testing and 90% are used for training. Evaluation is repeated 10 times, and each time a different bin is used for testing. It eventually uses the whole data set for training and testing. Finally, an average of the 10 iterations is used for the performance metric.

Role of windows on the performance comparison of classifiers for HPAR

Figure 4 shows the classification accuracy comparison of various classifiers with various window sizes. It clearly depicts that a window size of 3 s used for features extraction has better performance. All the six classifiers have higher accuracy for $Window - 3$ . According to Shoaib et al.,¹⁷ smaller window size is efficient for recognition of simple activities (such as walking, standing) and larger window size for complex activities (such as eating and smoking). Since we are recognizing both simple and complex activities, the window of 3 s performs better. The decision tree–based classifiers such as $RF$ , $J 48$ , and $PART$ show higher accuracy compared to $SVM$ , $MLP$ , and $KNN$ . The rule-based classifier $(PART)$ also uses decision tree concept.⁴⁹ Figure 5 shows the performance comparison in terms of $TPR$ and $precision$ . Figure 4 shows that $RF$ , $J 48$ , and $PART$ achieve higher $TPR$ while the same behavior can be observed for precision in Figure 4 at $Window - 3$ .

Figure 4.

Classifier accuracy comparison for various window sizes.

Figure 5.

Performance comparison of $TPR$ and $precision$ for various windows sizes used for features extraction: (a) TPR of classifiers and (b) precision of classifiers.

Performance comparison of individual activity recognition in the proposed HPAR

Figure 6 shows the performance comparison of individual activity recognition using $Window - 3$ for $RF$ , $J 48$ , and $PART$ classifiers. All acquired activities have higher accuracy except $A 12 : GoingUpStairs$ and $A 13 : GoingDownStairs$ . Activities A2, A3, A5, A7, A8, A9, and $A 11$ have accuracy greater than 96%. $A 1, A 4, A 6$ , and $A 10$ have accuracy greater than 84%, while $A 12$ and $A 13$ were greater than 65%.

Figure 6.

Physical activity recognition rates of $J 48$ , $PART$ , and $RF$ , where x-axis shows various activities and y-axis shows the classification accuracy achieved by each classifier.

Figure 7(a)–(c) shows the confusion matrices of $RF$ , $PART$ , and $J 48$ classifiers, respectively. Since sensors are installed at wrist position only, the $A 1 : Writing$ and $A 10 : Eating$ have more similar event compared to other activities. Similarly, $A 6 : Sitting$ and $A 10 : Eating$ have higher similar event because mostly the $A 10$ activity data are acquired while sitting and standing. Since $Writing$ data are collected both in sitting and standing position so it has similar events with $Standing$ and $Eating$ .

Figure 7.

Confusion matrices of $RF$ , $PART$ , and $J 48$ classifiers for individual activity recognition: confusion matrix of (a) J48 classifier, PART classifier, and (c) RF classifier.

Table 6 shows the $TPR$ and $precision$ achieved by $J 48$ , $PART$ , and $RF$ . The TPRs achieved by each classifier for activities $A 2$ , $A 3$ , $A 5$ , $A 7$ , $A 8$ , $A 9$ , and $A 11$ are higher than others. Consequently, the respective precision values for these activities are also higher. This means that the currently extracted features from the sensors data are good enough to make these classifiers achieve TPRs and precision values more than 94%. It can also be noted that the precision values are almost equal to TPRs, which from equation (14) means very low false negative rates.

Table 6.

True positive rates (TPR) and precision values achieved by $J 48$ , $PART$ , and $RF$ for each physical activity.

	TPR			Precision
	J48	PART	RF	J48	PART	RF
A1	0.88	0.886	0.857	0.854	0.868	0.876
A2	0.944	0.967	0.976	0.954	0.953	0.969
A3	0.957	0.976	0.979	0.964	0.973	0.978
A4	0.829	0.837	0.839	0.875	896	0.837
A5	0.93	0.909	0.846	0.935	0.967	0.99
A6	0.93	0.909	0.846	0.859	0.889	0.804
A7	0.967	0.977	0.97	0.969	0.982	0.987
A8	0.969	0.988	0.992	0.956	0.977	0.985
A9	0.938	0.949	0.967	0.941	0.943	0.957
A10	0.849	0.868	0.829	0.876	0.88	0.828
A11	0.952	0.954	0.95	0.956	0.952	0.996
A12	0.654	0.659	0.656	0.704	0.628	0.649
A13	0.721	0.625	0.644	0.678	0.631	0.679

Bold values represents that these activities are recognized with higher accuracies.

Conclusion

This article proposed a web-based framework for human physical activity recognition that integrates wearable sensors, smartphones, and processing with a recognition server. The smartphone collects data from wearable sensors using Bluetooth and transfers it to the server using HTTP. The transferred data are then represented in features vectors and given to the via already trained classifier in order to recognize the activity. The feedback of such recognition is then sent to the smartphone via the Internet. We collected data for 13 physical activities using several sensors such as accelerometer, gyroscope, magnetometer, body temperature, and pressure sensors. The window size for feature extraction was evaluated, where it was found that a window of 3 s performs the best from among 2-, 3-, and 4-s windows. We evaluated six classifiers for physical activity recognition which are random forests, decision tree, a rule-based classifier, KNN, MLP, and SVM. It was found that on the given data set, RF, the decision tree, and rule-based classifiers perform better than SVM, KNN, and MLP by achieving an overall classification accuracy of more than 94%. In future, we plan to use more sensors such as electrocardiogram (ECG) and breathing sensors for healthcare and evaluate more state-of-the art machine learning algorithms such as deep learning.

Footnotes

Handling Editor: Bingang Xu

Author contributions

All authors contributed to the paper. F.U., A.I., and H.A. contributed to conceptualization, software, formal analysis, and writing. A.U.R., K.S., A.B., and S.A. contributed to data curation, software, and methodology. S.Y. and K.S.K. contributed to validation, funding acquisition, and review and editing

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a National Research Foundation of Korea-Grant funded by the Korean Government (Ministry of Science and ICT)-NRF-2020R1A2B5B02002478.

ORCID iD

Farman Ullah

References

Luong

Hoang

Wang

, et al. Data collection and wireless communication in Internet of Things (IoT) using economic analysis and pricing models: a survey. IEEE Commun Surv Tut 2016; 18(4): 2546–2590.

Iqbal

Ullah

Anwar

, et al. Interoperable Internet-of-Things platform for smart home system using Web-of-Objects and cloud. Sustain Cities Soc 2018; 38: 636–646.

Lee

Han

Cho

, et al. Intelligent positive computing with mobile, wearable, and IoT devices: literature review and research directions. Ad Hoc Netw 2019; 83: 8–24.

Hiremath

Yang

Mankodiya

. Wearable Internet of Things: concept, architectural components and promises for person-centered healthcare. In: 2014 4th international conference on wireless mobile communication and healthcare—transforming healthcare through innovations in mobile and wireless technologies (MOBIHEALTH), Athens, 3–5 November 2014, pp.304–307. New York: IEEE.

Gochoo

Tan

Liu

, et al. Unobtrusive activity recognition of elderly people living alone using anonymous binary sensors and DCNN. IEEE J Biomed Health 2018; 23(2): 693–702.

Mukhopadhyay

SC.

Wearable sensors for human activity monitoring: a review. IEEE Sens J 2015; 15(3): 1321–1330.

Caspersen

Powell

Christenson

GM.

Physical activity, exercise, and physical fitness: definitions and distinctions for health-related research. Public Health Rep 1985; 100(2): 126.

Bosems

A ubiquitous system for smart reasoning for well-being at home and at work. In: 2012 IEEE international conference on pervasive computing and communications workshops (PERCOM Workshops), Lugano, 19–23 March 2012, pp.534–535. New York: IEEE.

Callaway

Rozar

Quantified wellness wearable technology usage and market summary. Chesterfield, MO: RGA Reinsurance Company, 2015.

10.

Lin

Sun

M-T

Poovandran

, et al. Human activity recognition for video surveillance. In: 2008 IEEE international symposium on circuits and systems, Seattle, WA, 18–21 May 2008, pp.2737–2740. New York: IEEE.

11.

Schneider

Banerjee

. Activity recognition using imagery for smart home monitoring. In: Hassanien

Oliva

(eds) Advances in soft computing and machine learning in image processing. Cham: Springer, 2018, pp.355–371.

12.

Avci

Bosch

Marin-Perianu

, et al. Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: a survey. In: 2010 23rd international conference on architecture of computing systems (ARCS), Hannover, 22–23 February 2010, pp.1–10. Berlin: VDE.

13.

Ignatov

. Real-time human activity recognition from accelerometer data using convolutional neural networks. Appl Soft Comput 2018; 62: 915–922.

14.

Shoaib

Bosch

Incel

, et al. Complex human activity recognition using smartphone and wrist-worn motion sensors. Sensors 2016; 16(4): 426.

15.

Bulling

Blanke

Schiele

. A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput Surv 2014; 46(3): 33.

16.

Sztyler

Stuckenschmidt

Petrich

. Position-aware activity recognition with wearable devices. Pervasive Mob Comput 2017; 38: 281–295.

17.

Shoaib

Scholten

Havinga

PJM

. Towards physical activity recognition using smartphone sensors. In: Ubiquitous intelligence and computing, 2013 IEEE 10th international conference on and 10th international conference on autonomic and trusted computing (UIC/ATC), Vietri sul Mere, 18–21 December 2013, pp 80–87. New York: IEEE.

18.

Ravì

Wong

Deligianni

, et al. Deep learning for health informatics. IEEE J Biomed Health 2016; 21(1): 4–21.

19.

Lee

C-T

Chen

W-P

Enabling human activity recognition with smartphone sensors in a mobile environment. In: Proceedings of the world congress on engineering, vol. 1, 2017, http://www.iaeng.org/publication/WCE2017/WCE2017_pp194-198.pdf

20.

Shoaib

Bosch

Incel

, et al. A survey of online activity recognition using mobile phones. Sensors 2015; 15(1): 2059–2085.

21.

Choudhury

Consolvo

Harrison

, et al. The mobile sensing platform: an embedded activity recognition system. IEEE Pervas Comput 2008; 7(2): 32–41.

22.

Lester

Choudhury

Borriello

. A practical approach to recognizing physical activities. In: Fishkin

Schiele

Nixon

, et al. (eds) International conference on pervasive computing. Berlin: Springer, 2006, pp.1–16.

23.

Pham

Buluschek

, et al. Elderly people living alone: detecting home visits with ambient and wearable sensing. In: Proceedings of MMHealth, no. EPFL-CONF-231125, 2017, http://publications.idiap.ch/index.php/publications/show/3696

24.

Yao

Sheng

Benatallah

, et al. WITS: an IoT-endowed computational framework for activity recognition in personalized smart homes. Computing 2018; 100(4): 369–385.

25.

Guo

Wang

Liu

, et al. HuAc: human activity recognition using crowdsourced WiFi signals and skeleton data. Wirel Commun Mob Com 2018; 2018: 6163475.

26.

Kumari

López-Benìtez

Lee

, et al. Wearable Internet of Things-from human activity tracking to clinical integration. In: 2017 39th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Seogwipo, 11–15 July 2017, pp.2361–2364. New York: IEEE.

27.

Gia

Sarker

Tcarenko

, et al. Energy efficient wearable sensor node for IoT-based fall detection systems. Microprocess Microsy 2018; 56: 34–46.

28.

Al-Khafajiy

Baker

Chalmers

, et al. Remote health monitoring of elderly through wearable sensors. Multimed Tools Appl 2019; 78: 24681–24706.

29.

Bayat

Pomplun

Tran

. A study on human activity recognition using accelerometer data from smartphones. Procedia Comput Sci 2014; 34: 450–457.

30.

Anjum

Ilyas

. Activity recognition using smartphone sensors. In: 2013 IEEE 10th consumer communications and networking conference (CCNC), Las Vegas, NV, 11–14 January 2013, pp.914–919. New York: IEEE.

31.

Ronao

Cho

S-B

. Human activity recognition using smartphone sensors with two-stage continuous hidden Markov models. In: 2014 10th international conference on natural computation (ICNC), Xiamen, China, 19–21 August pp.681–686. New York: IEEE, 2014.

32.

Dao

M-S

Nguyen-Gia

T-A

Mai

V-C

. Daily human activities recognition using heterogeneous sensors from smartphones. Procedia Comput Sci 2017; 111: 323–328.

33.

Ceja

EAG

. Human activity recognition using smartphone’s sensors and machine learning-Edición Única. Mexico City: Repositorio Nacional CONACYT.

34.

Lee

Kim

. Energy-efficient real-time human activity recognition on smart mobile devices. Mob Inf Syst 2016; 2016: 2316757.

35.

Cao

Wang

Zhang

, et al. GCHAR: an efficient group—based context—aware human activity recognition on smartphone. J Parallel Distr Com 2018; 118(1): 67–80.

36.

Cvetković

Szeklicki

Janko

, et al. Real-time activity monitoring with a wristband and a smartphone. Inform Fusion 2018; 43: 77–93.

37.

Attal

Mohammed

Dedabrishvili

, et al. Physical human activity recognition using wearable sensors. Sensors 2015; 15(12): 31314–31338.

38.

Bao

Intille

. Activity recognition from user-annotated acceleration data. In: Ferscha

Mattern

(eds) International conference on pervasive computing. Berlin: Springer, 2004, pp.1–17.

39.

Cleland

Kikhia

Nugent

, et al. Optimal placement of accelerometers for the detection of everyday activities. Sensors 2013; 13(7): 9183–9200.

40.

Shardlow

. An analysis of feature selection techniques. Manchester: The University of Manchester, 2016.

41.

. Wearable computing: accelerometer-based human activity classification using decision tree. Utah State University, 2017.

42.

Maurer

Smailagic

Siewiorek

, et al. Activity recognition and monitoring using multiple sensors on different body positions. In: International workshop on wearable and implantable body sensor networks, 2006 (BSN 2006), Cambridge, MA, 3–5 April 2006, p. 4. New York: IEEE.

43.

Parkka

Ermes

Korpipaa

, et al. Activity classification using realistic data from wearable sensors. IEEE T Inf Technol B 2006; 10(1): 119–128.

44.

Frank

Nadales

Josefa

, et al. Reliable real-time recognition of motion related human activities using MEMS inertial sensors, 2010, https://elib.dlr.de/64996/2/activityRecognition.pdf

45.

Favre

Luthi

Jolles

, et al. A new ambulatory system for comparative evaluation of the three-dimensional knee kinematics, applied to anterior cruciate ligament injuries. Knee Surg Sport Tr A 2006; 14(7): 592–604.

46.

Pirttikangas

Fujinami

Nakajima

. Feature selection and activity recognition from wearable sensors. In: Youn

Kim

Morikawa

(eds) International symposium on ubiquitious computing systems. Berlin: Springer, 2006, pp.516–527.

47.

Cherian

. Recognition of everyday activities through wearable sensors and machine learning. Master’s thesis, Texas A&M University, College Station, TX, 2017.

48.

Köping

Shirahama

Grzegorzek

. A general framework for sensor-based human activity recognition. Comput Biol Med 2018; 95: 248–260.

49.

Shoaib

Bosch

Incel

, et al. Fusion of smartphone motion sensors for physical activity recognition. Sensors 2014; 14(6): 10146–10176.

50.

Dudani

. The distance-weighted k-nearest-neighbor rule. IEEE T Syst Man Cyb 1976; 4: 325–327.

51.

Ullah

Sarwar

Lee

. N-screen aware multicriteria hybrid recommender system using weight based subspace clustering. Sci World J 2014; 2014: 679849.

52.

Beyer

Goldstein

Ramakrishnan

, et al. When is “nearest neighbor” meaningful? In: Beeri

Buneman

(eds) International conference on database theory. Berlin: Springer, 1999, pp.217–235.

53.

Rish

. An empirical study of the naive Bayes classifier. In: IJCAI 2001 workshop on empirical methods in artificial intelligence, vol. 3, no. 22, pp.41–46. IBM, 2001, https://www.cc.gatech.edu/∼isbell/reading/papers/Rish.pdf

54.

Safavian

Landgrebe

. A survey of decision tree classifier methodology. IEEE T Syst Man Cyb 1991; 21(3): 660–674.

55.

Breiman

Friedman

Olshen

, et al. Classification and regression trees. Belmont, CA: Wadsworth, 1984.

56.

Liaw

Wiener

. Classification and regression by random forest. R News 2002; 2(3): 18–22.

57.

Hearst

Dumais

Osuna

, et al. Support vector machines. IEEE Intell Syst App 1998; 13(4): 18–28.

58.

Schalkoff

. Artificial neural networks, vol. 1. New York: McGraw-Hill, 1997.

59.

Kose

Incel

Ersoy

. Online human activity recognition on smart phones. In: Workshop on mobile sensing: from smartphones and wearables to big data, vol. 16, no. 2012, pp.11–15, 2012, https://www.researchgate.net/publication/267245099_Online_Human_Activity_Recognition_on_Smart_Phones

60.

Frank

Hall

Witten

. The WEKA Workbench. Online Appendix for “Data Mining: Practical Machine Learning Tools and Techniques.”4th ed. Burlington, MA: Morgan Kaufmann, 2016.

61.

D-N

Nguyen

T-T

Nguyen

T-H

, et al. Mobile online activity recognition system based on smartphone sensors. In: Akagi

Nguyen

, et al. (eds) International conference on advances in information and communication technology. Cham: Springer, 2016, pp.357–366.

62.

Weiss

Lockhart

Pulickal

, et al. Actitracker: a smartphone-based activity recognition system for improving health and well-being. In: 2016 IEEE international conference on data science and advanced analytics (DSAA), Montreal, QC, Canada, 17–19 October 2016, pp.682–688. New York: IEEE.

63.

Wang

Lee

, et al. Detection of optimal activity recognition algorithm for elderly using smartphone. In: Park

D-S

Chao

H-C

Jeong

Y-S

, et al. (eds) Advances in computer science and ubiquitous computing. Singapore: Springer, 2016, pp.1013–1018.

64.

Metawear MetaMotionR Sensor, https://mbientlab.com/product/metamotionr-bundle/

65.

JJC

. Interruptions: using activity transitions to trigger proactive messages. Master’s thesis, Massachusetts Institute of Technology, Cambridge, MA, 2004.

66.

Chang

G-C

Kang

W-J

Luh

J-J

, et al. Real-time implementation of electromyogram pattern recognition as a control command of man-machine interface. Med Eng Phys 1996; 18(7): 529–537.

Wearable Internet-of-Things platform for human activity recognition and health care

Abstract

Keywords

Introduction

Background and related works

Platforms and sensors used for HPAR

Sensor placement for HPAR

Features computation, extraction, and selection for HPAR

Classifiers for HPAR

The proposed WIoT platform for HPAR

Hardware and software of the proposed HPAR framework

Data collection in the HPAR

Feature extraction and selection in the proposed HPAR system

Classification algorithms for the proposed HPAR

Simulation results and discussion

Role of windows on the performance comparison of classifiers for HPAR

Performance comparison of individual activity recognition in the proposed HPAR

Conclusion

Footnotes

Author contributions

Declaration of conflicting interests

Funding

ORCID iD

References