Calculation of limits for significant bidirectional changes in two or more serial results of a biomarker based on a computer simulation model

Abstract

Background

Reference change values provide objective tools to assess the significance of a change in two consecutive results of a biomarker from an individual. However, in practice, more results are usually available and using the reference change value concept on more than two results will increase the number of false positive results.

Methods

A computer simulation model was developed using Excel. Based on 10,000 simulated measurements among healthy individuals, a series of up to 20 results of a biomarker from each individual was generated using different values for the within-subject biological variation plus the analytical variation. Each new result in this series was compared to the initial result. These successive serial differences were computed to give limits for significant bidirectional changes with constant cumulated maximum probabilities of 95% (p < 0.05) and 99% (p < 0.01).

Results

From an individual factors used to multiply the first result were calculated to create limits for constant cumulated significant changes. The factors were shown to become a function of the number of results included and the total coefficient of variation.

Conclusions

The first result should be multiplied by the appropriate factors for increase and decrease to give the limits for a significant bidirectional change in several consecutive measurements.

Introduction

A method to calculate the significance of change in serial results was introduced by Harris and Yasaka. ¹ The basis for this method in monitoring serial results from an individual is that, for a change to be significant, the difference in two consecutive results must exceed the inherent variation, originally termed the critical difference but now more often the reference change value (RCV). However, before any RCV calculation, further major decisions have to be made. If the clinical decision-making is based on either a significant decrease or increase, then the RCV is unidirectional (one-tailed). A bidirectional change for the RCV calculation is used when both increases and decreases of results are being considered (two-tailed). In clinical practice, bidirectional change commonly needs to be considered. Then, the probability of detecting changes has to be decided, and thereby also how many false positive results should be accepted. Usually, 95% probability (p < 0.05) is regarded as significant and 99% probability (p < 0.01) as highly significant. ² In consequence, generally 1.96 and 2.58 are the appropriate Z-scores to use for these RCV significance calculations. All laboratories know the random analytical variation (CV_A) of each of their analyses from internal quality control activities. Data on the within-subject biological variation (CV_I) are available for many quantities. ³ The total coefficient of variation (CV_T,) can be calculated from estimates of CV_A and CV_I [ ${\text{CV}}_{\text{T}}={({\text{CV}}_{\text{A}}2+{\text{CV}}_{\text{I}}2)}^{\frac{1}{2}}$ ] and RCV significance limits may also be calculated [RCV = ±(Z · 2^½ · CV_T)]. ² This RCV calculation assumes that CV_I has a random fluctuation around a homeostatic setting point and is normally (Gaussianly) distributed. However, many quantities have a CV_T in individuals over time that is not normally distributed and it has been recognized that a ln-normal approach is a better model for biological data distributions when assessing components of biological variation.^4–8 If a dataset is ln-normally distributed, then the natural logarithm of that dataset is normally distributed. In practice, however, often more than two serial results are available for an individual and it is possible to calculate the significance of changes between each of the two consecutive measurements. But, when the RCV concept is used on more than two results, the number of false positive result increases. ¹ Recently, a computer-based simulation model has been described for eliminating this growing number of false positive events by adjusting the significant limits according to the increasing number of results. ⁹ This simulation model however has only been applied on unidirectional differences. Therefore, application of the simulation model to calculate significant limits for bidirectional changes is needed.

The aim of this study was to apply the simulation model to calculate limits for significant bidirectional changes in two or more serial results with constant cumulated probabilities of both 95% (p < 0.05) and 99% (p < 0.01). The calculations were based on both normal and ln-normal computer simulated data from ‘healthy individuals’.

Materials and methods

All data for the simulations were generated using Microsoft Excel version 2010.

The method has been described in detail in a previous publication associated with significant unidirectional changes in serial results. ⁹ Briefly, the method is described here with the modifications associated with significant bidirectional changes in serial results.

Reference change up factor (RCF_up) is defined as a factor to multiply first results by when calculating the upper limit for a significant increment in concentration. When two consecutive results, X1 and X2, are considered an increase in concentration is significant if X2 > RCF_up · X1. Similarly, reference change down factor (RCF_down) factor is defined as a factor to multiply first result by when calculating the downward limit for a significant decrement in concentration. A decrease in concentration is considered significant if X2 < RCF_down · X1. The RCF_up and RCF_down factors were determined based on 10,000 simulated normally distributed results and 10,000 ln-normally distributed results. The RCF factors were calculated for different sets of results (consisting of 2, 3, 4, 5, 6, 7, 8, 9, 10, 15 and 20 serial results) and for different percentages of CV_T (10.0, 20.0, 30.0 and 40.0%). When significant limits for bidirectional changes are calculated with a probability of 95% (p < 0.05), the total false positive rate is 5%. In bidirectional changes, both significant increases and decreases are considered, i.e. 2.5% false positive > upper significant limit plus 2.5% false positive < downward significant limit. The RCF_up factor was then determined when 2.5% of the tested sets were recorded as having a significant increase at a bidirectional probability of 95%, i.e. when

$\text{X}2(\text{falsepositiveresults}\hspace{1em}(2.5\%))\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}}>\text{uppersignificantlimit}={\text{RCF}}_{\text{up}}·\text{X}1$

Similarly, the RCF_down factor was determined when 2.5% of the tested sets were recorded as having a significant decrease at a bidirectional probability of 95%, i.e. when

$\text{X}2(\text{falsepositiveresults}\hspace{1em}(2.5\%))\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}}<\text{downwardsignificant}{\text{limit}=\text{RCF}}_{\text{down}}·\text{X}1$

Similarly, the RCF_up and RCF_down factors were determined as having significant changes for a bidirectional probability of 99%, i.e. a total of 1% false positive results, when X2 (0.5% false positive) > RCF_up · X1 plus X2 (0.5% false positive) < RCF_down · X1.

Results

The factors for calculating the upper significant limits for bidirectional changes, RCF_up, are listed in Table 1 with the variables examined (CV_T, number of results, probability of 95 and 99%) for normally distributed as well as ln-normally distributed simulated data. Similarly, the factors for calculation of the lower significant limits, RCF_down, are presented in Table 2.

Table 1.

RCF_up ^a factors for calculating the upper limits for a significant increase at constant cumulated bidirectional probabilities of 95 and 99% for both normal and ln-normal distributions.

	RCFup a (95% probability)
	CVTb = 10.0%		CVTb = 20.0%		CVTb = 30.0%		CVTb = 40.0%
Number of results c	Normal d	Ln-normal e	Normal d	Ln-normal e	Normal d	Ln-normal e	Normal d	Ln-normal e
2	1.32	1.32	1.80	1.72	2.65	2.24	4.87	2.88
3	1.37	1.37	1.93	1.85	2.95	2.49	5.60	3.30
4	1.40	1.39	2.01	1.92	3.14	2.63	6.20	3.57
5	1.42	1.41	2.07	1.98	3.26	2.75	6.52	3.75
6	1.43	1.43	2.11	2.04	3.37	2.87	6.70	3.98
7	1.44	1.44	2.13	2.06	3.45	2.92	6.92	4.10
8	1.45	1.45	2.15	2.09	3.46	2.97	6.97	4.17
9	1.46	1.45	2.16	2.10	3.53	3.00	7.26	4.24
10	1.46	1.46	2.18	2.12	3.58	3.06	7.35	4.32
15	1.49	1.49	2.26	2.21	3.72	3.25	7.75	4.70
20	1.51	1.52	2.32	2.29	3.86	3.41	8.09	5.01
	RCFup a (99% probability)
2	1.45	1.44	2.30	2.04	4.95	2.88	>100	4.00
3	1.50	1.48	2.42	2.16	5.77	3.15	>100	4.50
4	1.53	1.52	2.52	2.28	5.90	3.39	>100	4.97
5	1.55	1.53	2.56	2.33	6.10	3.50	>100	5.18
6	1.56	1.55	2.65	2.39	6.30	3.60	>100	5.40
7	1.58	1.56	2.66	2.41	6.52	3.68	>100	5.52
8	1.58	1.56	2.68	2.44	6.53	3.72	>100	5.61
9	1.58	1.57	2.69	2.45	6.54	3.80	>100	5.80
10	1.58	1.57	2.73	2.47	6.55	3.83	>100	5.84
15	1.60	1.61	2.84	2.56	7.00	4.04	>100	6.25
20	1.65	1.67	2.96	2.75	7.18	4.49	>100	7.15

a

RCF_up, factor to multiply the first result by when calculating the upper limit for a significant increase in concentrations.

b

CV_T, total coefficient of variation.

c

Simulated concentration data from healthy individuals.

d

Data based on normal (Gaussian) distributions.

e

Data based on ln-normal (ln-Gaussian) distributions.

Table 2.

RCF_down ^a factors for calculating the downward limits for a significant decrement with constant cumulated bidirectional probabilities of 95 and 99% for both normal and ln-normal distributions.

	RCFdown a (95% probability)
	CVTb = 10.0%		CVTb = 20.0%		CVTb = 30.0%		CVTb = 40.0%
Number of results c	Normal d	Ln-normal e	Normal d	Ln-normal e	Normal d	Ln-normal e	Normal d	Ln-normal e
2	0.68	0.76	0.20	0.58	0 f	0.45	0 f	0.35
3	0.63	0.73	0.07	0.54	0 f	0.40	0 f	0.30
4	0.60	0.72	0 f	0.52	0 f	0.38	0 f	0.28
5	0.58	0.71	0 f	0.51	0 f	0.36	0 f	0.27
6	0.57	0.70	0 f	0.49	0 f	0.35	0 f	0.25
7	0.56	0.69	0 f	0.49	0 f	0.34	0 f	0.24
8	0.55	0.69	0 f	0.48	0 f	0.34	0 f	0.24
9	0.54	0.69	0 f	0.48	0 f	0.33	0 f	0.24
10	0.54	0.68	0 f	0.47	0 f	0.33	0 f	0.23
15	0.51	0.67	0 f	0.45	0 f	0.31	0 f	0.21
20	0.49	0.66	0 f	0.44	0 f	0.29	0 f	0.20
	RCFdown a (99% probability)
2	0.55	0.69	0 f	0.49	0 f	0.35	0 f	0.25
3	0.50	0.68	0 f	0.46	0 f	0.32	0 f	0.22
4	0.47	0.66	0 f	0.44	0 f	0.29	0 f	0.20
5	0.45	0.65	0 f	0.43	0 f	0.29	0 f	0.19
6	0.44	0.66	0 f	0.42	0 f	0.28	0 f	0.19
7	0.42	0.64	0 f	0.41	0 f	0.27	0 f	0.18
8	0.42	0.64	0 f	0.41	0 f	0.27	0 f	0.18
9	0.42	0.64	0 f	0.41	0 f	0.26	0 f	0.17
10	0.42	0.64	0 f	0.40	0 f	0.26	0 f	0.17
15	0.40	0.62	0 f	0.39	0 f	0.25	0 f	0.16
20	0.35	0.60	0 f	0.36	0 f	0.22	0 f	0.14

a

RCF_down, factor to multiply the first result by when calculating the downward limit for a significant decrement in concentrations.

b

CV_T, total coefficient of variation.

c

Simulated concentration data from healthy individuals.

d

Data based on normal (Gaussian) distributions.

e

Data based on ln-normal (ln-Gaussian) distributions.

f

The RCF_down factor is < 0 (negative) and therefore the significant limit is zero (0) because a negative concentration makes no sense.

The RCF_up factors calculated in Table 1, based on simulated data sets, exceed the theoretically RCF_up values based on RCF_up = (1 + (RCV/100)). ³ For example, for two serial results and a bidirectional probability of 95% (Z = 1.96), the RCF_up results based on the theoretically RCF_up calculations were 1.28, 1.55, 1.83 and 2.11 for CV_T of 10.0, 20.0, 30.0 and 40.0%, respectively; the corresponding RCF_up results based on the simulations (Table 1) were 1.32, 1.80, 2.65 and 4.87. These RCF_up results based on the simulations were calculated to give 2.5% false positive results. However, the theoretically RCF_up calculations applied on the normally simulated data gave 4.0, 6.7, 9.0 and 11.5% false positive results for CV_T of 10.0, 20.0, 30.0 and 40.0% respectively. These differences clearly increase with higher CV_T. For example, for CV_T = 40% and probability of 99%, the RCF_up factors became so high that calculations were impossible (RCF_up > 100, Table 1). RCF_down factors are negative (<0) for CV_T = 20.0% and up to CV_T = 40.0% (Table 2) for normal distributions and therefore these downward limits are zero because negative concentrations cannot occur. In Table 3, the Z_n-scores are calculated for different percentages of CV_T and numbers of results (n) based on ln-normal distributions. For two results, the Z₂-scores for the probability of 95 and 99%, respectively, are almost constant and indistinguishable from the Z-scores 1.96 and 2.58 used in the formula: RCF_up = exp(Z · 2^½ · σ) where σ is the standard deviation of the underlying normal distribution. ⁴ Similarly, the Z_n-scores calculated for three and up to 20 results also remain approximately constant. This demonstrates that the formula is also valid for more than two serial results with Z_n-scores increasing with number of results. In addition, the results of RCF_down factor calculations based on the simulation data (Table 2) are the same as those calculated using the formula RCF_down = exp(–Z · 2^½ · σ). ⁴ From Tables 1 and 2 it can be shown that RCF_up = 1/RCF_down, and RCF_down = 1/RCF_up. Furthermore, the equation σ = [ln((CV_T/100)²+ 1)]^½ was confirmed due to CV_T calculations of all the ln-normal data (data not shown). ⁴ From the same relationship, it can be shown that σ approximates to CV_T, i.e. σ and CV_T are almost identical up to about CV_T = 30%. Therefore, in practice, the following equation is useful to calculate the RCF_up for CV_T < 30% and the relevant number of results, n: RCF_up = exp(Z_n · 2^½ · CV_T/100).

Table 3.

Z_n-scores ^a appropriate for the number of serial results (n) at probabilities of 95 and 99% for ln-normal distributions.

Number of results	Zn-scores a (95% probability)
(n)	CVT = 10.0%	CVT = 20.0%	CVT = 30.0%	CVT = 40.0%
2	1.97 b	1.94 b	1.94 b	1.94 b
3	2.23	2.20	2.20	2.19
4	2.33	2.33	2.33	2.34
5	2.43	2.45	2.44	2.43
6	2.53	2.55	2.54	2.54
7	2.58	2.58	2.58	2.59
8	2.63	2.63	2.62	2.62
9	2.63	2.65	2.65	2.65
10	2.68	2.68	2.69	2.69
15	2.83	2.83	2.84	2.84
20	2.97	2.96	2.96	2.96
	Zn-scores a (99% probability)
2	2.58 c	2.55 c	2.55 c	2.54 c
3	2.78	2.75	2.76	2.76
4	2.97	2.94	2.94	2.94
5	3.01	3.02	3.02	3.02
6	3.11	3.11	3.09	3.10
7	3.15	3.14	3.14	3.14
8	3.15	3.19	3.17	3.17
9	3.20	3.20	3.22	3.23
10	3.20	3.23	3.24	3.24
15	3.37	3.36	3.36	3.36
20	3.63	3.61	3.62	3.61

a

Z_n-scores were used to calculate RCF_up factors for bidirectional changes during serial monitoring. Z_n-scores were calculated using the formula Z_n = ln(RCF_up)/(2^½ · σ). RCF_up results are used from Table 1 giving a constant cumulated bidirectional probability of 95 and 99% and is based on ln-normal distributions from healthy individuals. σ = [ln((CV_T/100)²+ 1)]^½ and is calculated based on different CV_T. ⁴ CV_T, denotes total coefficient of variation.

b

Z₂ results estimated from simulations should be equal to Z = 1.96 used for traditional RCV.

c

Z₂ results estimated from simulations should be equal to Z = 2.58 used for traditional RCV.

Discussion

As also previously found ⁹ we have shown that the practical use of traditional RCV calculations applied on normally distributed data will produce too many false positive results especially for high values of CV_T, compared to theoretical RCV calculations. On the other hand, the calculated RCF_up factors are comparable for both normal and ln-normal distributions up to CV_T = 20.0% (Table 1), but not for RCF_down factors (Table 2). At higher CV_T, the significant upper limit calculated using the RCF_up factors for normal distributions will exceed those determined using the RCF_up factors for ln-normal distributions. In the same way, RCF_down factors for normal distributions are always smaller than the corresponding RCF_down factors for ln-normal distributions. Accordingly, the ranges of limits for significant changes (up and down) are all wider for normal distributions as compared with ln-normal distributions (Tables 1 and 2). Table 2 shows that RCF_down values are negative (<0) for CV_T = 20% and more than three results and up to CV_T = 40.0% for normal distributions and, therefore, these decrease limits are zero because negative concentrations are impossible. Consequently, we propose that all calculations of limits for significant changes should be based on ln-normal distributions. Compared with traditional RCV calculation [RCV = (Z · 2^½ · CV_T)], the proposed significance calculations based on ln-normal distributions are not more complicated. The only difference of variables compared with traditional RCV calculations is the Z_n-score obtained from Table 3 for more than two serial results.

In this study we have used the simulation model on bidirectional changes. Recently we have also examined the model on unidirectional differences. ⁹ In both of our studies, bidirectional and unidirectional, we have used probabilities of 95% (p < 0.05) and 99% (p < 0.01). Consequently, all the calculated ranges of limits for significant changes (up and down) are wider for bidirectional changes compared with unidirectional differences.

An example of calculation of limits for significant bidirectional changes for up to six ln-normally distributed serial results

A typical anaemic female patient had a blood haemoglobin concentration (6.0 mmol/L) which is below the population-based reference interval (7.1–9.3 mmol/L). The clinical symptoms indicated iron-deficiency anaemia and oral iron therapy was initiated. Monitoring of blood haemoglobin concentrations was started and a bidirectional change for calculation of significant limits was used. A decrease in results indicates that the anaemia may be due to cause other than iron deficiency and an increase indicates that the treatment is effective. Calculation of upper limits for significant increments in blood haemoglobin concentration with a probability of both 95 and 99% compared with the initial concentration of 6.0 mmol/L is listed in Table 4, and the corresponding downward significant limits are listed in Table 5. These significant limits are also illustrated in Figure 1 as a function of sample number.

Figure 1.

The significant limits for bidirectional changes for a patient with an initial blood haemoglobin concentration of 6.0 mmol/L and probabilities of both 95% (p < 0.05) and 99% (p < 0.01) illustrated as a function of sample number.

Table 4.

Calculation of upper limits for significant increase in blood haemoglobin concentrations compared with an initial concentration of 6.0 mmol/L. The results are from a typical anaemic patient.

Number of results (n)	RCFup a calculated as exp(Zn b  · 2½ · CVT c /100)	Significant limit d RCFup 6 (mmol/L)	RCVup a calculated as exp(Zn e  · 2½ · CVT c /100)	Significant limit f RCFup 6 (mmol/L)
2	Exp(1.97 · 2½ · 0.0302) = 1.09	6.5	Exp(2.58 · 2½ · 0.0302) = 1.12	6.7
3	Exp(2.23 · 2½ · 0.0302) = 1.10	6.6	Exp(2.78 · 2½ · 0.0302) = 1.13	6.8
4	Exp(2.33 · 2½ · 0.0302) = 1.10	6.6	Exp(2.97 · 2½ · 0.0302) = 1.14	6.8
5	Exp(2.43 · 2½ · 0.0302) = 1.11	6.7	Exp(3.01 · 2½ · 0.0302) = 1.14	6.8
6	Exp(2.53 · 2½ · 0.0302) = 1.11	6.7	Exp(3.11 · 2½ · 0.0302) = 1.14	6.9

a

RCF_up, factor to multiply the first result by when calculating the upper limit for a significant increase in concentrations.

b

Z_n-scores are found from Table 3 using a bidirectional probability of 95%.

c

CV_T = (CV_A²+ CV_I²)^½= 3.02% where analytical variation, CV_A = 1.0% and within-subject biological variation, CV_I = 2.85%. ³

d

Using a bidirectional probability of 95%.

e

Z_n-scores are found from Table 3 using a bidirectional probability of 99%.

f

Using a bidirectional probability of 99%.

Table 5.

Calculation of downward limits for significant decrease in blood haemoglobin concentrations compared with an initial concentration of 6.0 mmol/L. The results are from a typical anaemic patient.

Number of results (n)	RCFdowna = 1/RCFup b	Significant limit c RCFdown 6 (mmol/L)	RCFdowna = 1/RCFup b	Significant limit d RCFdown 6 (mmol/L)
2	0.92	5.5	0.90	5.4
3	0.91	5.5	0.89	5.3
4	0.91	5.4	0.88	5.3
5	0.90	5.4	0.88	5.3
6	0.90	5.4	0.88	5.3

a

RCF_down, factor to multiply the first result by when calculating the downward limit for a significant decrease in concentrations.

b

The corresponding RCF_up factors are calculated in Table 4.

c

Using a bidirectional probability of 95% (p < 0.05).

d

Using a bidirectional probability of 99% (p < 0.01).

Conclusions

A simple calculation of limits for significant bidirectional changes in two or more serial results is presented here. The upper limit for significant increment of serial results (ln-normally distributed) is calculated by a factor RCF_up. If X1 is the first result, then upper limit is found with X1 multiplied by RCF_up

$\text{upperlimit}=\text{X}1·{\text{RCF}}_{\text{up}}$

The RCF_up factor is calculated from the equation

${\text{RCF}}_{\text{up}}=\text{exp}({\text{Z}}_{\text{n}}·2\frac{1}{2}·{\text{CV}}_{\text{T}}/100)$

The Z_n-score, which varies with the sampling number (n), is provided in Table 3 for ln-normal distributions. The CV_T is calculated using estimates of analytical variation (CV_A) and within-subject biological variation (CV_I) [ ${\text{CV}}_{\text{T}}={({\text{CV}}_{\text{A}}2+{\text{CV}}_{\text{I}}2)}^{\frac{1}{2}}$ ]. ² The corresponding downward limit for a significant decrement of serial results is calculated by the factor RCF_down (downward limit = X1·RCF_down). The RCF_down factor is determined by the equation

${\text{RCF}}_{\text{down}}=1/{\text{RCF}}_{\text{up}}$