Reducing Glucose Meter Adverse Events by Using Reliability Growth With the FDA MAUDE Database

Abstract

Glucose meter evaluations are common and provide important information about glucose meter performance versus standards. Although some meters meet guidelines and others fall short in these evaluations, most results are within the A and B zones of a glucose meter error grid. Another data source that is seldom used is the FDA adverse event database (MAUDE). This database describes glucose meter malfunctions and injury as reported by actual users and returned 10 837 adverse events across all meters for the first 7 months of 2018. Reliability growth management is an established tool to reduce failure rates. A reliability growth example is presented followed by a discussion of how this tool could be applied to reduce glucose meter failure events using the MAUDE database.

Glucose meter evaluations are commonly performed to determine whether a meter meets performance standards. These evaluations are performed as part of regulatory approval and also with meters after release for sale.^1-2 The studies, performed in tightly controlled settings, show that some meters meet standards, while others fall short. It has previously been discussed why meters that have been cleared might fail performance standards after they are marketed. ³ Yet, even for the meters that don’t meet standards, most results are in either the A or B zones of a glucose meter error grid.

A glucose meter is unique compared to other diagnostic tests due to the volume of glucose meter tests. Given that 6 million people in the United States use insulin, ⁴ and assuming testing three times daily, over 6 billion glucose meter tests are performed yearly just in the United States. Note that a rare event such as a glucose meter failing to produce a result is often mentioned in passing in a laboratory evaluation but with 6 billion tests, even a small percentage of meters failing to produce a result in an evaluation can translate into thousands of such instances in the glucose meter user population when an emergent value is needed but unavailable. The same is true for the small percentage of results that occur in zones higher than A or B in a glucose meter error grid. And even for a meter with 100% of results within the A zone and no failures to obtain a result, the 95% confidence level for the percentage of adverse events for a 200-sample study could be as high as 1.8%. ⁵

The MAUDE Database

Besides evaluations, a different data source of glucose meter performance is the FDA adverse event database called Manufacturer and User Facility Device Experience (MAUDE). ⁶ A primer on working with these data ⁷ and previous adverse events for diabetes from the MAUDE database have appeared. ⁸ Although some other publications mention the MAUDE database, ⁹ there has been little analysis. There are several possible reasons for this. Although there is a query form on the FDA site, it is not as powerful as downloading files into a database, joining tables (containing 135 fields), and performing queries, which require knowledge of databases. Many of the fields in MAUDE are demographic and of limited interest. Key fields are MDR_REPORT_KEY (used to join tables), EVENT_TYPE (death, injury, or malfunction), BRAND_NAME (the meter brand), GENERIC_NAME (used in an SQL query to select glucose meters from other medical devices), and TEXT (a description of the adverse event).

In the 7-month period from January through July 2018, of the 579 357 adverse events across all medical devices there were 10 837 adverse events across all glucose meters (see Table 1). Whereas this could be considered a small number relative to the yearly number of glucose tests run, if one considers this number as people (assuming different people for most events), the number is not so small. Moreover, events that are recorded in this database differ considerably from those in published evaluations since in MAUDE, many glucose differences from a comparison method are in regions of a glucose meter error grid higher than A or B zones. Another common entry is that the meter failed to provide a result. Table 2 shows events for 10 records across different brands.

Table 1.

Glucose Meter Events for the First 7 Months of 2018.

Event type	Events months 1-7	Events per year (projected)
Injury	888	1522 (8.2%)
Malfunction	9949	17 055 (91.8%)
Total	10 837	18 577

Table 2.

Records From the MAUDE Database.

Record number	EVENT_TYPE	Text
1	IN	Complaint meter read 60 comparison meter 140
2	M	Complaint meter read 120 comparison meter 40
3	M	Meter broken
4	M	Meter read in mg/dL used to read in mmol/L
5	IN	No result due to error message
6	M	Meter failed to store result in memory
7	IN	Meter read 170 mg/dL higher than hospital meter
8	M	Meter read “LO” retest on a different meter was 138
9	M	Defective display
10	M	No result—meter would not turn on

Note: IN, injury; M, malfunction.

Whereas it would be of interest to get adverse event rates either overall or by glucose meter brand, this requires knowing how many tests are performed, which is not generally available. Without rates, one cannot meaningfully interpret trends. Unless one works for the manufacturer of a glucose meter brand, an additional challenge is reading through and classifying thousands of event records, where the text for each event can be lengthy. A simple, exported document of glucose meter adverse events for the 7-month period of 2018 is 2388 pages!

Yet, undoubtedly, each manufacturer does review these events for their meters. Reducing error rates for processes that exhibit significant numbers of failure events is well known in the reliability world and has been applied to medical products.^10-11 The purpose of this paper is to review one such reliability technique and to suggest that it could be used by manufacturers to reduce glucose meter adverse events.

Review of the Reliability Growth Management (RGM) Example

RGM is an established technique in the aerospace and automotive industry. ¹² To briefly review RGM, Duane, while at General Electric, observed that reliability improvement for a variety of different systems followed a similar pattern. ¹³ Improvement was proportional to the cumulative time that the systems were under test (equation 1).

$$\mathrm{\lambda }\left(t\right)=\left(1-\mathrm{\alpha }\right)K{T}^{-\mathrm{\alpha }}$$

(1)

where

λ is the failure rate at time t (the end of the cumulative test time T)

α is the growth rate estimated as the regression slope of the Duane plot

K is the intercept estimated as the regression intercept of the Duane plot

T is the cumulative time that the system or process is under test

This observation is based on learning theory and basically means that through observing and fixing problems, reliability will improve. The notion of “testing in quality” flies in the face of quality gurus who insist on designing it right the first time. Yet, designing in quality is only possible when one has a high state of knowledge. When this is not the case, testing in quality may be the fastest way to achieve a quality goal.

RGM was used during the development of an automated immunoassay analyzer and after it was released for sale. ¹¹ It involved several steps:

1. Selecting a metric and its goal

2. Creating a model to ensure that all relevant events are counted

3. Recording and classifying events as to their frequency and severity

4. Ranking events in a Pareto chart

5. Working on solving the top Pareto items

6. Modeling the failure rate through Duane analysis

In the case of the automated immunoassay analyzer the metric chosen was the unscheduled service call rate (call rate) as unscheduled service calls were very expensive. Using past instrument service records, a model was developed as to how events could lead to service calls. Certain events such as failure of the instrument to turn on would always result in a service call, whereas other events such as a paper jam would only result in a service call if they happened repeatedly. As the immunoassay analyzer was being developed, technicians running the analyzer were given forms to fill out so that all relevant events would be captured. An additional data source was obtained by downloading data from the instruments. After release for sale, failure data were obtained as customer complaints through the service department and again data downloaded from the instruments via the Internet.

Classifying failure events was a key part of the process and relied on the previously mentioned model. A database was used to store all records. Besides demographic information, all events were classified as to severity and frequency. As a severity example, a paper jam was weighted 0.3 versus 1.0 for when the instrument would not turn on. With events classified, a Pareto chart was prepared, which is a decreasing list of failures ranked by severity times frequency. Engineers were directed to work on the problems most likely to cause service calls—the events at the top of the Pareto chart. Without the guidance of the Pareto chart, engineers might work on problems in order of their appearance, “interesting” problems, or other such criteria. Progress was measured with Duane analysis, which also allowed prediction of when the goal would be reached.

RGM was successful in this project and used subsequently in other projects. Its success depended on faithful data collection and classification. Management appreciated the accuracy in predicting when goals would be met and also that the project was accomplished without adding more people. A challenge in undertaking RGM was the reluctance of the engineering staff to adopt tools from outside their department. Hence, upper management support was an important success factor.

Discussion

Published glucose meter evaluations and the MAUDE database are like parallel universes. Published glucose meter evaluations are controlled studies usually conducted by health care professionals where results that could be called adverse events occur rarely if at all. On the other hand, the MAUDE database contains only adverse events which are unverified and often unverifiable, with results usually conducted by lay people with a range of proficiencies.

In reliability, two commonly used tools are FMEA (failure mode effects analysis) designed to reduce risk of adverse events that might occur and FRACAS (failure reporting and corrective action system) designed to reduce rates of adverse events that have occurred. Clearly, the latter technique is applicable here as there are a significant number of adverse events for glucose meters. The source of most MAUDE data is each manufacturer as reported by their customers. Manufacturers have procedures to deal with events reported to them by customers. CAPA (corrective action preventive action) is one example. The benefit of RGM is classifying events to create a Pareto chart and estimating rates of improvement. Another difference is that CAPA typically gets its data from customer complaints, whereas RGM also receives data from field instruments through an Internet connection. ¹¹ Devices such as IoT (Internet of things) could facilitate obtaining data from meters in the field. A possible metric would be the MAUDE injury rate. This could be modeled from past MAUDE data.

For each glucose meter brand the questions are what is the rate of glucose meter adverse events over time? Is the rate increasing or decreasing or remaining constant? Is there a Pareto chart? Are there active teams to prevent problems from recurring through design changes versus a recovery policy of simply replacing meters?

There are important differences between the immunoassay instrument and glucose meters. The immunoassay instrument is a much more complex instrument with the possibility for improvement much greater than a glucose meter, which is a simpler device. Design changes for glucose meters might include items beyond hardware, such as software and training.

Finally, one must realize that there will be an underlying failure rate for glucose meters which can’t be improved, given the existing technology. An additional challenge is the previously mentioned problem that the glucose meter adverse events are unverified. So one can also ask, how close is the current adverse event rate to the rate for which further improvement is unlikely? This question is best answered by manufacturers.

Conclusions

The number of glucose meter events reported in MAUDE is probably a subset of all glucose meter failure events. These events differ considerably from the results presented in glucose meter evaluations. RGM is an established way to reduce failure rates. It could complement the results of glucose meter evaluations. Perhaps it’s time to try this technique.