Abstract
Background:
In an era of increasing technology and telehealth utilization, three-dimensional (3D) wound cameras promise reliable, rapid, and touch-free ulceration measurements. However, reliability data for commercially available devices in the diabetes foot service setting is lacking. We aimed to evaluate the reliability of diabetes-related foot ulceration measurement using a 3D wound camera in comparison to the routinely used ruler and probe.
Method:
Participants were prospectively recruited from a tertiary interdisciplinary diabetes foot service. Ulcerations were measured at each visit by two blinded observers, first by ruler and probe, and then using a 3D wound camera twice. Reliability was evaluated using intraclass correlation coefficients (ICC). Measurement methods were compared by Pearson correlation.
Results:
Sixty-three ulcerations affecting 38 participants were measured over 122 visits. Interobserver reliability of ruler measurement was excellent for estimated area (ICC 0.98, 95% CI 0.97-0.98) and depth (ICC 0.93, 95% CI 0.90-0.95). Intraobserver and interobserver reliability of the 3D wound camera area was excellent (ICC 0.96, 95%CI 0.95-0.97 and 0.97 95% CI 0.96-0.98, respectively). Depth was unrecordable in over half of 3D wound camera measurements, and reliability was inferior to probe measurement. Area correlation between methods was good (
Conclusions:
3D wound cameras offer practical advantages over ruler-based measurement. In diabetes-related foot ulceration, the reliability and comparability of area measurement was excellent across both methods, although depth was more reliably obtained by the probe. These limitations, together with cost, are important considerations if implementing this technology in diabetes foot care.
Keywords
Introduction
International guidelines on diabetes-related foot complications emphasize the importance of ulceration area and depth with respect to optimizing communication and referral and also in monitoring treatment success. 1 This rudimentary component of ulceration assessment is still largely performed using disposable rulers and metal probes, with potential disadvantages including observer error or variation, measurement time, pathogen transmission, and generation of clinical waste. Further, rapid analysis and evaluation of important clinical parameters, such as ulceration area reduction, requires translation of data into a digital format.
New technologies show promise in overcoming these issues. While recent studies have tested novel three-dimensional (3D) wound cameras for diabetes-related foot ulceration,2-4 the use of these devices is technically complex and they are not yet widely accessible. Commercially available devices are increasingly utilized in wound care; however, reliability data specific to diabetes-related foot ulceration is lacking. As interdisciplinary diabetes foot services attempt to address the ever-increasing prevalence of diabetes-related foot complications in our population, 5 greater utilization of telehealth is inevitable. Integration of technologies that aid reproducible and efficient clinical assessment and enable remote specialist input will be an important component of this transition in healthcare.
This study aimed to evaluate the intraobserver and interobserver reliability of diabetes-related foot ulceration area and depth measurement using a commercially available 3D wound camera. Further, we aimed to compare this method of ulceration measurement to the routinely used ruler and probe method, both with respect to interobserver reliability and correlation of measurements.
Methods
Participants were ambulant attendees of the St Vincent’s Hospital Sydney interdisciplinary diabetes foot service, located in Sydney, Australia. Recruitment occurred between August 2019 and February 2020. Inclusion criteria included age over 18 years, diabetes mellitus diagnosis, active foot ulceration, and ability to provide informed consent. Active foot ulceration was defined as a break in the epidermis and part of the dermis distal to the malleoli. 6 There were no exclusion criteria.
Study data were collected and managed by investigators using REDCap electronic data capture tools hosted at St Vincent’s Hospital Sydney. Core details with respect to both participant and ulceration characteristics were included. 7 Participant data collected encompassed demographics, body mass index (BMI), diabetes mellitus history, glycated hemoglobin (HbA1c), prevalent macrovascular risk factors, microvascular and macrovascular complications, and previous foot complications. Ulceration was characterized according to location and aspect, duration at the time of recruitment, presence of neuropathy, ischemia or both, the precipitating event, whether it was recurrent (ie, previously healed ulceration in the same location), and WIfI clinical stage. 8
At routine care visits, two observers independently performed area and depth ulceration measurements with a disposable paper ruler and metal probe, and then twice with the 3D wound camera (eKare inSight, VA, USA). Ruler measurements included the ulcerations maximum diameter, perpendicular diameter, and maximum probe depth. Maximum probe depth was chosen to allow comparison with the 3D wound camera, which also provides a maximum depth measurement. One outlier for depth was excluded from subsequent analysis as maximum depth measurement relied on a deep probe of a sinus.
The 3D wound camera ulceration volume and “tissue classification” were also recorded, given these measures are automatically reported with area and depth. Volume is automatically calculated by the device, which creates a depth map of the ulceration bed with depth values assigned at each pixel position. 9 Tissue classification uses an inbuilt algorithm to estimate the proportion of the ulceration area that is red, yellow or black in color, signifying healthy granulation, slough, or necrosis, respectively.
The observers were senior clinicians of the interdisciplinary diabetes foot service, qualified in podiatry and endocrinology. Observers followed the manufacturer’s instructions and had access to wound models to gain familiarity with the device prior to study commencement. All 3D wound camera measurements were made under the light of a medical examination lamp, the camera was held perpendicular to the ulceration surface, and the inbuilt software was used to indicate when the camera-to-ulceration distance was adequate for 3D image capture. Observers were blinded to each other’s measurements. Rare visits where a second observer was not available were excluded from the interobserver reliability analysis.
Participant and ulceration characteristics were summarized using number with percentage for categorical variables, median with interquartile range (IQR) for nonparametric continuous variables, and mean with SD for parametric continuous variables.
Estimated elliptical area based on ruler measurement was calculated using the equation:10,11
This comparator for the area was chosen as ruler measurements are routinely obtained in the clinical setting, there is no need for specialty equipment, and the calculation is rapidly performed. We did not seek to establish validity in this study, as the most accurate and precise measurement methods are technically challenging and time consuming and, therefore, not in routine clinical use. For example, planimetry requires the wound edge to be traced on transparent film, which is then superimposed on a scale grid to calculate the area. Digital planimetry applies the same concept. Studies that attempt to establish validity typically use wound models to optimize the accuracy of measurement; however, here we opted to assess performance in the real-world clinical setting.12,13 We used the ellipse formula as opposed to the rectangle formula, given the latter is known to grossly overestimate ulcer size. 14 Previous studies have demonstrated a close correlation between the estimated area using the ellipse formula and planimetric methods,10,11 even for areas up to 40 cm2. 11 Diabetes-related foot ulceration with an area above this threshold would be atypical, and ulceration of this size was not seen in this study. Further, both the ellipse formula and planimetric methods predict healing. 11
Intraclass correlation coefficients (ICC) were calculated to assess the reliability and are presented with 95% CI. Intraobserver ICC for the 3D wound camera measurements used a two-way mixed-effects model with absolute agreement. Interobserver ICC for both methods of measurement used a two-way random-effects model with absolute agreement. Interobserver ICC for the 3D wound camera used each observer’s first measurement. Pairwise deletion was used for missing data and is presented with each corresponding analysis. Based on the 95% CI of the ICC estimate, values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 were reported as poor, moderate, good, and excellent reliability, respectively. 15
Pearson correlation (
Unexpectedly, the 3D wound camera reported a maximum depth of zero in over half of the measurements. The manufacturer advised during the study that zero millimeters is the default value assigned for depth when the device deems measurement to be unreliable. In response to this, we performed an off-protocol exploratory analysis re-evaluating the Pearson correlation in two different ulceration subsets. First, where the 3D wound camera maximum depth was recorded as greater than zero millimeters. Second, where the 3D wound camera area was recorded as less than or equal to 1.7 cm2, an arbitrary value above which only two zero millimeter depth measurements were returned across both observers.
Statistical analysis was conducted using SPSS Statistics for Macintosh, Version 25 (IBM Corp, Armonk, NY, USA). Ethics approval for the study was obtained from the St Vincent’s Hospital Sydney Human Research Ethics Committee (reference number 2019/ETH00611).
Results
Participant characteristics are summarized in Table 1. There were 38 participants with a total of 63 ulcerations. The mean participant age was 65 years, 79% were male, and most were of Caucasian ethnicity (87%). The majority had type 2 diabetes mellitus (95%) with median diabetes mellitus duration of ten years, peripheral neuropathy was ubiquitous, and a quarter had peripheral arterial disease.
Participant Characteristics (
Data are mean (±SD), median (IQR) or
ANZ, Australia and New Zealand peoples; BMI, body mass index; eGFR, estimated glomerular filtration rate; VPT, vibration perception threshold.
Ulceration characteristics are presented in Table 2. Most were neuropathic (84%) and precipitated by poorly fitting footwear (21%) or deformity (32%). The location and aspect are of importance when evaluating 3D wound camera technology, given these devices typically require surrounding nonulcerated skin as a reference. Most commonly ulcerations involved the forefoot (38%) or midfoot (29%), and there were no extensive ulcerations. Approximately half were plantar, and only 9% were on obscure aspects for 3D wound camera measurement, such as the apex of toes and interdigital sites.
Ulceration Characteristics (
Data are median (IQR) or
Based on estimate risk of amputation at one year.
Interobserver reliability of standard ruler measurements (
Intraobserver Reliability of Three-Dimensional Wound Camera Measurements.
ICC, intraclass correlation coefficients
Interobserver Reliability of Three-Dimensional Wound Camera Measurements.
ICC, intraclass correlation coefficients
Correlation coefficients comparing the estimated area using a ruler and 3D wound camera area were 0.88 and 0.94 for each observer (see Figure 1). Correlation between maximum depth using a ruler and 3D wound camera maximum depth produced coefficients of 0.49 and 0.65 for each observer. As presented in Table 5, the strength of correlation was not improved after excluding ulcerations where the 3D wound camera recorded a zero millimeter depth or smaller ulcerations with area less than or equal to 1.7 cm2.

Correlation between ruler-based estimated area and three-dimensional wound camera area measurements.
Correlations Between Ruler and Three-Dimensional Wound Camera Depth Measurements Under Different Conditions.
Discussion
There is increasing uptake of commercially available 3D wound cameras in interdisciplinary diabetes foot services. However, this is one of the first diabetes-related foot ulceration studies to evaluate the reliability of one such device and compare its measurements to those obtained using traditional techniques in a real-world setting. The participants included were representative of our general outpatient cohort, 16 as well as reflective of other diabetes-related foot ulceration cohorts, 17 with the possible exception of our relatively low prevalence of peripheral arterial disease. Overall, we found the ulceration area could be reliably measured either by ruler-based estimation or the 3D wound camera. There was a good correlation between both methods for area, with the ruler-based method likely overestimating by approximately 20%.
We found the greatest limitation of this technology to be depth perception. The importance of depth measurement should not be undervalued as this is one of the key parameters in predicting ulceration outcome. 18 Maximum depth was registered by the 3D wound camera as zero in over half of ulcerations measured, with corresponding ruler measurements ranging from one millimeter to five millimeters. Additionally, the outlier excluded from the analysis was an ulceration with sinus probing up to 20 mm deep. The manufacturer advises that depth discrimination is limited at values lower than two millimeters to three millimeters, at which point recorded depth defaults to zero millimeters. Consequently, ulceration volume typically also returns zero. It may therefore be prudent, particularly if implementing this technology in the telehealth setting, that clinicians are trained to manually record depth if unmeasurable by the device. Further, there is still an important role for probing in ulceration assessment, as sinuses and communication with underlying structures may otherwise be missed.
Finally, the ulceration tissue classification should be interpreted with caution given the low reliability. This is particularly relevant for black coloration, as this may indicate the presence of necrotic or gangrenous tissue, which has the potential to alter clinical management.
A handful of other studies have evaluated the reliability of this 3D wound camera including depth; however, none were dedicated to diabetes-related foot ulceration.9,12,13,19,20 Anghel et al also found the reliability of area to be excellent, while for maximum depth intraobserver ICC was 0.36 and interobserver ICC was 0.65. 9 Users of the camera noted difficulties in ensuring perpendicular sensor orientation, adequacy of lighting, and particular inaccuracy with wounds less than four square centimeters in area. A rodent study noted area reliability to be excellent, with results comparable but more efficiently obtained when compared with planimetry. 13 Two studies compared the device with a competing laser-based technology, although used wound models and ulcerations that were mostly larger than typical for diabetes-related foot ulceration.12,19 Overall the 3D wound camera produced comparable results, albeit with less measurement error and clinician time required. Finally, a study of pressure ulcers found the area to be reliable but advised cautious use of depth and volume measurement in the clinical setting. 20
Other promising devices are appearing in the literature, although they generally are not as compact and efficient as those commercially available. For instance, the user is required to upload wound data to separate computer software and then manually outline the wound edge. In 2015, a prototype was developed utilizing three cameras and a projector. 21 Reliability in diabetes-related foot ulceration was excellent, even for volume measurement, although the mean area of wounds was nine square centimeters. 2 The device was incapable of measuring volume for shallow and some digital wounds. Interestingly, for ulcerations greater than five square centimeters, 3D wound surface area measurement was found to be more sensitive in monitoring wound healing than two-dimensional wound area typically measured by other devices. 22 Another prototype uses two infrared cameras, an infrared projector, and two LED lights, and was found to have excellent reliability for area, maximum depth, and volume in diabetes-related foot ulceration. 4 Malone et al tested a further device against ruler-based measurements, and while area reliability was again excellent, volume reliability was moderate. 3 Area measurement, based on either method, was the best predictor of wound healing.
In our experience, there were definite benefits of using the 3D wound camera, particularly with respect to the digitalization of data, accessibility of records on a user-friendly secure web-based dashboard, ease of data extraction for research and auditing purposes, and patient engagement as ulceration progress was graphed in real time. Fortunately, as technology continues to adapt and evolve, 3D wound cameras are likely to see significant advances and will inevitably play an important role in interdisciplinary diabetes foot services. Improvements, such as greater image resolution, a dedicated light source, and incorporated data fields relevant to diabetes-related foot ulceration assessment, could greatly enhance the application of this technology. Circumferential wound measurement capability has recently been added. 23
The major strength of this study was the broad inclusion of diabetes-related foot ulcerations representative of those typically seen in interdisciplinary diabetes foot services. We also acquired a relatively large number of measurement sets. Generalizability may be a limitation, as this study was performed by highly trained tertiary hospital diabetes foot service clinicians. This is of greater relevance for ruler and probe measurements, as there is subjectivity in user discrimination of the ulceration edge, maximum and perpendicular diameters, and maximum probe depth. Therefore, our reliability findings for the ulcer and probe method should be cautiously applied to nonspecialist services. Repeating the study in remote services where similar expertise is not available may highlight a reliability advantage of the 3D wound camera technology, which would support its potential application in telehealth. The location of ulcerations in this study was another limitation, as there were no circumferential ulcerations, and a minority was located in areas known to be challenging for wound edge delineation (eg, the apex of the toe). Lighting may have been further optimized; however, the results are more applicable having subjected the device to standard clinical conditions. Finally, the area comparator was estimated by ruler measurement, and so, we cannot ascertain validity. Again, this was chosen to reflect the tools currently employed in the majority of services.
Wound imaging technology is rapidly advancing and there are several areas for future research. As new devices are marketed, comparative studies are needed to evaluate technical performance in the same ulceration cohort. 24 Applications for smartphones are also being developed, either using a single camera with multiple viewing angles 25 or particular devices with multiple cameras. 26 Removing the need for dedicated hardware would reduce costs and increase the opportunity for telehealth applications. However, it seems unlikely that the reliability of depth measurement would be improved. Of note, with the currently available technology, telehealth outcomes have been inferior to face-to-face consultation.27,28 There has been interest in using area projections in predicting ulceration healing time,3,22,29,30 and it is feasible that volume may deliver even greater predictive power. Devices may also increasingly incorporate other technologies that provide tissue health and healing information,24,31 and machine learning will play an increasing role in the delineation of tissue type. 24
Based on our findings and review of the literature, the benefit of implementing the current technology in services managing diabetes-related foot ulceration would depend on the clinical setting. In rural or remote areas where there is a greater rotation of staff and lesser familiarity with diabetes-related foot ulceration assessment, the use of the 3D wound camera is intuitive and requires minimal training. The web-based dashboard aids care continuity between clinicians; tertiary centers can be readily involved virtually, and, if required, the wound area can even be remeasured remotely. By contrast, the cost of this technology may not be justifiable for specialized services managing diabetes-related foot ulceration. We have shown that highly trained clinicians are able to reliably obtain an estimated area and depth using a ruler and probe. Further, probe assessment provides a more reliable assessment of depth than the 3D wound camera, as well as allowing detection of sinuses and deeper structures. Specialist services are often already equipped with electronic medical records and high-resolution digital cameras, and so, the practical advantages of 3D wound cameras are less relevant.
Conclusion
Ulceration imaging will undoubtedly become increasingly digitalized in interdisciplinary diabetes foot services, given the benefits of enhanced documentation and improved communication between remote sites and specialist services. However, it is critical that all technologies are carefully evaluated before being implemented in the routine care of people with diabetes-related foot ulceration. This study found a commercially available 3D wound camera delivered reliable area measurement, comparable to estimation based on ruler measurement, although depth was more reliably obtained by probe. Together with rapid ulceration area measurement, there are practical advantages of 3D wound cameras; however, this needs to be considered in light of device limitations and cost.
Footnotes
Acknowledgements
The authors acknowledge the contribution of Miss Darcie Jonas, Senior Podiatrist, St Vincent’s Hospital Sydney, and Dr Shivani Patel, Endocrinology Advanced Trainee, St Vincent’s Hospital Sydney, in participant recruitment and data collection.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was funded by a competitive research grant awarded by the Agency for Clinical Innovation, New South Wales Government.
