Validation of the Diabetes Technology Society Error Grid

Versions of the glucose error grid have been around since the Clarke glucose meter error grid, published in 1987. ¹ The recently published Diabetes Technology Society Error Grid is meant for both glucose meters and continuous glucose monitors (CGMs) ² and describes how error (difference between monitor glucose from reference) affects patients by categorizing the effect of error into various zones (see Figure 1). The zone locations were decided by a survey of 89 clinicians using their opinions based on their knowledge and evidence. This is unlike the performance requirements for most other assays which are decided by regulatory bodies such as CLIA and only have one zone.

Figure 1.

Malfunction error grid.

Yet, the locations of the zones for the error grid have never been validated with actual data. That is, how much error will cause diabetes users to complain or to be injured.

This was explored using the adverse event database (also called MAUDE, Manufacturer and User Facility Device Experience) which are real-world data—they are not method evaluations. When a CGM user has a problem and reports it to the manufacturer, the manufacturer must follow the guidance of the U.S. medical device reporting law (21CFR 803) and if appropriate submit an adverse event using FDA form 3500A. An FDA website ³ has all adverse events grouped by year as a series of text files that anyone can download. These files were downloaded for the year 2021 into a Microsoft Sequel Server database, and queries were performed to separate CGMs from other medical devices. Two key fields in the database are EVENT_TYPE (malfunction, injury, or death) and FOI_TEXT, a free-form text description of the event. In 2021, the CGM events included 348,067 malfunctions, 8,178 injuries, and 3 deaths. A subset of these data was obtained by querying the FOI_TEXT field for any form of the word inaccurate. Thus, in these records, CGM users complained about CGM inaccuracy. This yielded 18,755 malfunctions and 692 injuries, and these records were transferred to Excel.

Finally, a visual basic for applications program was written which extracted 4,638 pairs of glucose versus. reference where reference was a glucose meter result. The Excel spreadsheet had a column where the event type was either malfunction (n = 4452) or injury (n = 186). The event-type injury meant that the CGM user required medical intervention. The spreadsheet was analyzed by the glucose meter error grid software ⁴ with results shown in Figure 1 for malfunction data, the larger dataset.

The figure provides evidence that validates the clinician opinion that zone A has no risk. However, note that Zone B has the highest percentage of adverse events (also true for injury data and not shown). Data from the error grid publication ² showed about 10% of results in Zone B from method comparisons for both glucose meters and CGMs. Thus, the guidance in ISO 15197 is questionable since it requires 99% of the results to be in either Zone A or Zone B of a glucose meter error grid. With the number of complaints and injury in zone B, this is a low bar for the ISO 15197 requirement.

Previously, it was reported that clinical trial publications often omit error grids. ⁵ Yet, error grids estimate clinical accuracy and are more important than system accuracy measures. Thus, the percentage of B zone and higher zone results is more important than an estimate of MARD.