Reliability and concurrent validity of a peripheral pulse oximeter and health–app system for the quantification of heart rate in healthy adults

Abstract

There are downloadable applications (Apps) for cell phones that can measure heart rate in a simple and painless manner. The aim of this study was to assess the reliability of this type of App for a Smartphone using an Android system, compared to the radial pulse and a portable pulse oximeter. We performed a pilot observational study of diagnostic accuracy, randomized in 46 healthy volunteers. The patients’ demographic data and cardiac pulse were collected. Radial pulse was measured by palpation of the radial artery with three fingers at the wrist over the radius; a low-cost portable, liquid crystal display finger pulse oximeter; and a Heart Rate Plus for Samsung Galaxy Note®. This study demonstrated high reliability and consistency between systems with respect to the heart rate parameter of healthy adults using three systems. For all parameters, ICC was > 0.93, indicating excellent reliability. Moreover, CVME values for all parameters were between 1.66−4.06 %. We found significant correlation coefficients and no systematic differences between radial pulse palpation and pulse oximeter and a high precision. Low-cost pulse oximeter and App systems can serve as valid instruments for the assessment of heart rate in healthy adults.

Keywords

heart rate pulse rate smartphone Apps

Introduction

The World Health Organization (WHO) and the International Association of Cardiologists emphasize that regular measurement of heart rate is especially recommended for adults over 50 years as preventive action.¹ People at risk or diagnosed with cardiovascular disorders (tachycardia, bradycardia, etc.) should regularly take their pulse when they feel sick, to aid in early detection of cardiovascular disease or aggravation of existing disease.² Currently, there are downloadable applications (Apps) for cell phones that can measure heart rate in a simple and painless manner.³

When measured accurately, heart rate provides useful diagnostic, therapeutic, and physical activity information. However, there is often a trade-off between the accuracy of the measuring device and its applicability for clinical use. For example, cardiac monitoring systems or pulse oximeters measure heart rate with a high degree of precision but are expensive, technically difficult to use, and labor intensive; therefore, they are not readily applicable to clinical or sport settings. To address such limitations, a “low cost portable pulse oximeter” has recently been introduced. Because this is not a medical device, a prescription from a physician is not required. The mobile health App detects heart rate through the camera on the smartphone and calculates the parameter for an individual.

The aim of this study was to assess the reliability of this type of App for a smartphone using an Android system, compared to the radial pulse and a portable pulse oximeter, in order to guide effective recommendations by health professionals. For patients without cardiovascular disease, this could serve as a quick and effective approach that has the potential to decrease the risk of a stroke or for use in other clinical or sports settings. Our studies assessed the intra-session reliability of both systems and concurrent validity of the “low cost portable pulse oximeter” and cellular health App systems with peripheral radial pulse. Based on the increased use of health-related Apps, we hypothesized that there were no systematic differences between the two systems compared with peripheral radial pulse palpation. Our main goal was to demonstrate strong agreement for peripheral radial pulse measurements with both the “low cost portable pulse oximeter” and mobile health App systems.

Patients and methods

We performed a pilot observational study of diagnostic accuracy, randomized in healthy volunteers, in January and February 2014. In all, 46 (19 males/27 females) healthy subjects from university staff and students volunteered to participate. Participants were free from any cardiovascular, neurological, or musculoskeletal diseases. The study was conducted after obtaining approval from the ethics committee of Colegio Oficial de Enfermeria del Principado de Asturias (Spain), and all participants provided their written informed consent.

Exclusion criteria were the following: diagnosed cardiac abnormalities, fever, alterations in the digital area of the index finger (cuts, blisters, scars, etc.), or cold hands. The patients’ demographic data and cardiac pulse were collected. Radial pulse was measured by (1) palpation of the radial artery with three fingers at the wrist over the radius,⁴ (2) a low-cost portable liquid crystal display (LCD) finger pulse oximeter (Nonin GO2^® http://www.nonin.com/PulseOximetry/Finger/GO2), and (3) a Heart Rate Plus App (https://play.google.com/store/apps/details?id=com.dungelin.heartrate) for Samsung Galaxy Note^® smartphones, which uses the sensor and the camera flash to measure heart rate as the index finger when held to the camera for 10–30 s. There is little information regarding the effectiveness of this application, thus we aimed to determine its reliability. Pulse was taken three consecutive times with each method. To avoid heart rate variations, patients remained sitting for 10 min prior to the measurements.

Descriptive statistical analysis was performed using percentages for qualitative variables. Results are presented as mean ± standard deviation (SD) of three measurements taken at the first finger on the right hand while seated. The paired t-test was used to determine systematic differences between the heart rate parameters obtained using the two systems (App and pulse oximeter) compared with the pulse measured by palpating the radial artery.

Intra-trial reliability was established by using the three measurements with each of the three systems during one session. Intra-class correlation coefficients (ICCs) using the (1, 1) model were calculated to determine reliability between trials when using each system (radial pulse, pulse oximeter, and App). The standard error of the mean (SEM) was calculated from the ICCs and SDs for each of the three measurements. In addition, SEM values were calculated from the ICCs and SDs for each of the three systems to determine the range of error attributed to each system. SEMs were calculated according to the formula $SD \times \sqrt{(1 - ICC)}$ .^5,6

The coefficient of variation (CV) and the percent error was calculated for intra-session reliability. The CV is calculated as the mean normalized to the SD. This value represents the amount of variation between trials, normalized to the mean for each variable. A higher CV value shows greater heterogeneity of variable values, and lower CV shows greater homogeneity in the values of the variable. Similarly, the percent error is calculated as the SEM divided by the mean per 100, and provides an estimate of the inherent error or variability normalized to the mean.

Accordingly, the results of pulse measurements were compared using ICC, with a value >0.85 indicating acceptable reliability.⁷ Concurrent validities between the two systems (App and pulse oximeter) compared with the pulse at radial right artery were calculated using ICCs (2, 1).^6,8

Coefficients of variation of method errors (CVME)⁹ and 95 percent limits of agreement (LOAs) were also calculated for the absolute comparison of parameters. As shown in the formula below, CVME values were converted into percentages by calculating CVME obtained using the SD of differences between the results obtained using the two systems,⁹ and 95 percent LOA values were calculated as described by Bland and Altman⁵

ME = SD / \sqrt{2}

CVME = 2 ME / (X 1 + X 2) \times 100 %

Repeatability coefficients (RCs) were used to evaluate the level of agreement between systems as follows: radial pulse versus pulse oximeter, radial pulse versus App, and pulse oximeter versus App.

The RC was calculated according to Bland and Altman⁵ as 1.96 times the SD of the differences between the paired measurements. The difference between the two measurement systems is expected to be less than this coefficient with a probability of 95 percent. The RC was also calculated as a percentage of the average values of the each two measurement systems (RC%).

Pearson’s correlation and linear regression analysis were also performed. Pearson’s correlation coefficient (r) was used to measure the strength of association between systems of measurement (radial pulse, pulse oximeter, and App). Linear regression analysis was used to predict the heart rate values from each system. These statistical methods are generally accepted for evaluating the agreement of two systems of clinical measurements irrespective of the distribution of variables and residuals.^10,11 A p value <0.05 was considered statistically significant (SPSS for Windows, version 20.0; SPSS Inc., Chicago, IL).

Results

Patient demographic characteristics are shown in Table 1. The intra-session reliability data (represented by ICC, SEM, CV, and percent error), and normative data (represented by mean ± SD) for the variable heart rate reliability trials using three systems are presented in Table 2. The results of the trial intra-session reliability produced an excellent ICC, low SEM, low percent errors, and low coefficients of variation. The results represent a small error that may occur within trials when using any of the systems tested.

Table 1.

Patient information.

	Mean ± SD	Range (min–max)	95% CI (lim. inferior–lim. superior)
N (male/female)	46 (19/27)
Age (years)	39.30 ± 7.35	27–53	38.20–40.50
Height (cm)	172.41 ± 8.02	159–181	171.10–173.72
Weight (kg)	68.30 ± 11.69	49–92	66.4–70.2
BMI	22.8 ± 2.3

SD: standard deviation; BMI: body mass index; CI: confidence interval; lim.: limit; max: maximum; min: minimum.

Table 2.

Intra-session reliability of heart rate measurements using different methods.

Method	Heart rate beat				SEM	CV	E%
	Mean ± SD	Range (min–max)	95% CI	ICC (95% CI)
Radial pulse	76.79 ± 8.74	60.00–94.33	74.2779.31	0.995 (0.992–0.997)	0.61	0.1138	0.79
Pulsioximeter	76.91 ± 8.81	60.33–93.33	74.37–79.43	0.995 (0.992–0.997)	0.62	0.1145	0.80
App	73.67 ± 8.77	60–90	71.14–76.20	0.972 (0.952–0.984)	1.46	0.1190	1.98

SD: standard deviation; CI: confidence interval; CV: coefficient of variation; E%: percent error; ICC: intra-class correlation coefficient; SEM: standard error of the mean.

The mean and SDs of the variable using the three methods are presented in Table 3. The paired t-test was used to determine systematic differences between the heart rate values obtained by the three methods, and we determined that values were similar when comparing the radial pulse and pulse oximeter methods with no significant differences. However, when comparing the App method to the radial pulse and pulse oximeter, there were significantly lower differences. Concurrent validities between radial pulse versus pulse oximeter, radial pulse versus App, and pulse oximeter versus App were calculated using ICCs (2, 1). With regard to the heart rate parameter, the concurrent validity between all comparisons resulted in ICC values that were considered in the “excellent” range (Table 3).

Table 3.

Mean values ± SDs for heart rate and concurrent validity measured with radial palpation pulse and two oximeter methods.

Radial pulse, mean ± SD	Oximeter, mean ± SD	App, mean ± SD	p value	ICC (95%CI)	Pearson (p value)	R²	95% LOA	CV%	RC	RC%
76.7 ± 8.74	76.91 ± 8.81		0.592	0.993 (0.988–0.996)	0.986 (p < 0.001)	0.9127 (p < 0.001)	−2.74/2.97	1.34	1.46	1.90
76.79 ± 8.74		73.67 ± 8.77	0.001	0.947 (0.452–0.984)	0.955 (p < 0.001)	0.911 (p < 0.001)	−2.00/8.26	2.46	4.11	5.46
	76.91 ± 8.81	73.67 ± 8.77	0.001	0.944 (0.410–0.984)	0.954 (p < 0.001)		−1.96/8.45	2.49	4.22	5.61

SD: standard deviation; CI: confidence interval; CV: coefficient of variation; E%: percent error; ICC: intra-class correlation coefficient; RC: repeatability coefficient; RC%: repeatability coefficient calculated as a percentage; SEM: standard error of the mean; LOA: limit of agreement.

p value obtained from paired t-test.

The reliability data (represented by ICC, SEM, percent error, and CV) and normative data (represented by mean ± SD) for heart rate reliability trials measured with the three methods are presented in Table 2. The ICCs for three trial intra-session reliability for radial pulse, pulse oximeter, and App methods were 0.995, 0.995, and 0.972, respectively. The results of the average trial intra-session reliability produced almost perfect ICCs and low SEMs. The coefficients of variation for the variable were 0.1138, 0.1145, and 0.1190, respectively. This represents a small error that may occur within trials, indicating strong and absolute reliability. RCs were also small in magnitude, further indicating a close agreement between each paired system. For example, the coefficient of 4.11 heart beats/min in Table 3 indicates that the expected maximal difference between the two measurement systems, radial pulse palpation with App in this case, would be 4.11 heart beats/min on 95 percent of occasions.

We used linear regression analyses to evaluate the relationship between radial pulse palpation and App and between pulse oximeter and App (Figures 1 and 2). The App data had a statistically significant positive correlation with radial pulse palpation and pulse oximeter with a R² = 0.911 (p < 0.01) and R² = 0.9127 (p < 0.01), for radial pulse palpation/App and pulse oximeter/App variables, respectively (Table 3). The significant association suggests that a linear regression model is optimal for prediction using the App approach.

Figure 1.

Linear regression graph and mathematical formula for radial pulse palpation and App.

Figure 2.

Linear regression graph and mathematical formula for pulse oximeter and App.

Discussion

This study was conducted to investigate the concurrent validity between the standard radial pulse palpation, and a portable pulse oximeter or a smartphone App, with respect to the measurement of heart rate as well as to examine the intra-session reliability of each method in healthy young adults. This type of technology would be useful in certain situations or patient populations where it is difficult to measure pulse by traditional methods, such as palpation. For example, in athletes, it can be difficult to measure radial pulse because of its high frequency. However, with the use of the App, one can assess heart rate and monitor performance. In a home setting, it could be difficult for an older person to palpate the radial pulse, so using the App could be beneficial in assessing heart rate and potential influences from stress or nervousness.

Our study is unique because we examined a healthy adult population. It is essential that the validity of heart rate measurement systems be established in the populations for their intended use. The three systems provided consistent intra-session results between trials with a low variability. All ICC values were greater than 0.972, and nearly all the percent errors were below 1.98 percent. These findings suggest that heart rate measurements with an App and/or pulse oximeter are appropriate for use to evaluate differences between participant groups. The SEM and percent error values are important variables that should be considered when formulating research protocols that use the low-cost pulse oximeter and App. The sample size to determine significant changes can be based directly on these measures of intra-session reliability.

The ICCs are a mathematical determination of the replication between multiple numerical sets and are commonly used for scientific measurements to represent the repeatability of the measurement.^8,12 It has been suggested that ICCs greater than 0.75 are considered good reliability, and values of 0.75 or less are considered moderate-to-poor reliability.⁸

Although ICCs provide a numeric value for the reliability of a measurement device, they do not describe the amount of error or inherent variability that is expected each time the measurement is performed. Assessing the error or variation each time a trial is performed is extremely important when capturing physiologic data where small differences between trials are expected. The SEM is another mathematical formula that uses the ICC and SD values to calculate the amount of expected error for the measurement device or individual.⁸

The SEMs and percent errors for all of the variables in this study were very low, suggesting that the variables are acceptable to use when assessing change before and after intervention, or when measuring differences between participant groups. Absolute reliability is as important as relative reliability. SEM is a quantitative expression of the range of errors that can occur whenever the same participant repeats certain tests.⁶ In this study, the calculated SEM for intra-session was very low level, indicating strong absolute reliability. The SEM values provided in this analysis will allow future researchers to make clinical judgments regarding what degree of change is due to factors beyond errors associated with the normal variability of measuring between trials or between sessions. Heart rate measurements by the App method were slightly lower than those by radial pulse palpation and pulse oximeter.

Despite systematic differences between App and other methods, a high level of correlation was determined between the systems when comparing with radial pulse palpation with either the pulse oximeter or the App method. Menz et al.¹³ suggested that although ICC is a more appropriate indicator of reliability than simple correlation coefficients (Pearson’s r, Spearman’s rho), a higher ICC does not necessarily ensure a high reliability. If the values of a sample are distributed over a wide range, a relatively high ICC can be achieved, even though score differences between the two measurements may be widely distributed. Thus, it has been asserted that both coefficients of variation and LOA⁵ must be used simultaneously in order to reduce the effects of such intrinsic limitations, and to ensure absolute reliability. CVME expresses the differences between values obtained using the two systems as a percentage. By doing so, CVME can be used as a clinically useful indicator of consistency, since it is unaffected by sample heterogeneity.¹³

This study demonstrated high reliability and consistency between systems with respect to the heart rate parameter of healthy adults using three systems. For all parameters, ICC was >0.93, indicating excellent reliability. Moreover, CVME values for all parameters were between 1.66 percent to 4.06 percent, and 95 percent LOA values, including zero, were within a narrow range with a symmetric distribution. These findings indicate slight changes between repeated measures using three systems and systematic bias was rarely observed. We found significant correlation coefficients and no systematic differences between radial pulse palpation and pulse oximeter, and its precision was very high. However, the precision of prediction between radial pulse palpation and App and between pulse oximeter and App are strongly consistent. The distribution of residuals also indicates a significant variation in the prediction of radial pulse or pulse oximeter values from App trials samples. The findings suggest that simple linear models may represent the association between radial pulse and pulse oximeter and App values appropriately. Accordingly, using the App-accepted methods, radial pulse and pulse oximeter may be a reliable proxy radial pulse.

The fact that heart rate was evaluated in healthy adults, and not in adults with systemic or peripheral disease, may represent a study limitation. While a measure may be valid in a young healthy subject, the same may not be true for an older person with an abnormal gait pattern. Future work with low-cost pulse oximeter and Apps should evaluate the repeatability and normative values for various ages, pathologies, or anatomical regions as fingers or toes that are known to be susceptible to measurement in high-risk patients, such as diabetics.

Conclusion

In this study, we demonstrated that the low-cost pulse oximeter and App Systems are valid instruments for the assessment of heart rate in healthy adults, and this is the first study of its kind to examine the intra-session reliability of each system compared to the standard peripheral radial palpation. The pulse oximeter shows tight correlation with radial pulse and the App needs to apply the regression formula as corrective factor to correlate with peripheral pulse. Both systems have the benefit of being quick to implement, simple to use, and inexpensive, making it attractive to clinicians who require accuracy in the clinical setting. Such features also make this system a viable option for use in a sports environment.

Our findings suggest that simple linear models may represent the association between radial pulse and pulse oximeter and App values appropriately. Accordingly, using the App-accepted methods, radial pulse and pulse oximeter may show a reliable proxy radial pulse. Therefore, future investigations should investigate patients with documented peripheral disease or vascular injuries to determine the suitability and validity of these equipment modalities in such clinical settings. Clinicians could implement the assessment of heart rate parameters using an App in a clinical setting, or by monitoring healthy adults’ progress during sports or training interventions. Researchers, clinicians, and trainers should feel confident incorporating this testing device into trials where repeatable measures of peripheral pulse are required for clinical decision making, such as emergency situations, evaluating comparative efficacy, or examining differences between pathologic groups.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

World Health Organization (WHO). Obesity and overweight (Internet). WHO, http://www.who.int/mediacentre/factsheets/fs311/en/ (2006, accessed 26 April 2014).

Tsuji

Larson

Venditti

. Impact of reduced heart rate variability on risk for cardiac events. The Framingham Heart Study. Circulation 1996; 94(11): 2850–2855.

Carneiro

. Smartphones e Tablets para Profissionais de Saúde. Brasil: TI Medicina, 2013.

Hill

Smith

. Examination of the extremities: pulses, bruits, and phlebitis. In: Walker

Hall

Hurst

(eds) Clinical methods: the history, physical, and laboratory examinations. 3rd ed. Boston, MA: Butterworths, 1990. pp. 148–152.

Bland

Altman

. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1(8476): 307–310.

Stratford

Goldsmith

. Use of the standard error as a reliability index of interest: an applied example using elbow flexor strength data. Phys Ther 1997; 77(7): 745–750.

Landis

Koch

. The measurement of observer agreement for categorical data. Biometrics 1977; 33(1): 159–174.

Shrout

Fleiss

. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979; 86(2): 420–428.

Portney

Watkins

. Foundations of clinical research: applications to practice. 3rd ed. Upper Saddle River, NJ: Pearson/Prentice Hall, 2009.

10.

Yildizdaş

Yapicioğlu

Yilmaz

. Correlation of simultaneously obtained capillary, venous, and arterial blood gases of patients in a paediatric intensive care unit. Arch Dis Child 2004; 89(2): 176–180.

11.

Treger

Pirouz

Kamangar

. Agreement between central venous and arterial blood gas measurements in the intensive care unit. Clin J Am Soc Nephrol 2010; 5(3): 390–394.

12.

Denegar

Ball

. Assessing reliability and precision of measurement: an introduction to intraclass correlation and standard error of measurement. J Sport Rehabil 1993; 2(1): 35–40.

13.

Menz

Latt

Tiedemann

. Reliability of the GAITRite walkway system for the quantification of temporo-spatial parameters of gait in young and older people. Gait Posture 2004; 20(1): 20–25.