A statistical framework for planning and analysing test–retest studies of repeatability

Abstract

There is an increasing number of potential quantitative biomarkers that could allow for early assessment of treatment response or disease progression. However, measurements of such biomarkers are subject to random variability. Hence, differences of a biomarker in longitudinal measurements do not necessarily represent real change but might be caused by this random measurement variability. Before utilizing a quantitative biomarker in longitudinal studies, it is therefore essential to assess the measurement repeatability. Measurement repeatability obtained from test–retest studies can be quantified by the repeatability coefficient, which is then used in the subsequent longitudinal study to determine if a measured difference represents real change or is within the range of expected random measurement variability. The quality of the point estimate of the repeatability coefficient, therefore, directly governs the assessment quality of the longitudinal study. Repeatability coefficient estimation accuracy depends on the case number in the test–retest study, but despite its pivotal role, no comprehensive framework for sample size calculation of test–retest studies exists. To address this issue, we have established such a framework, which allows for flexible sample size calculation of test–retest studies, based upon newly introduced criteria concerning assessment quality in the longitudinal study. This also permits retrospective assessment of prior test–retest studies.

1. Introduction

A biomarker is a characteristic objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes, or response to a therapeutic intervention. ¹ Biomarkers used as indicators of response to a therapeutic intervention, or disease progression, are called treatment response biomarkers. One prime, established treatment response biomarker is lesion size change in cross-sectional imaging. For clinical trials concerning solid tumors, the measurement of lesion size is formalized in the so-called response evaluation criteria in solid tumors (RECIST), ² that categorize treatment response. With the rapid advancement in medical sciences, there is an increasing number of new potential treatment response biomarkers that could possibly allow for early and objective assessment of treatment response or disease progression in clinical trials and clinical practice. ³

However, using a biomarker in practice requires some basic research into the reliability of its measurement. In addition to a fixed systematic measurement error (bias), which can be investigated by comparing measurements with a known target value (e.g. phantom studies), it is important to take into account that measurements of quantitative biomarkers are subject to random variability. Hence, changes in a biomarker in longitudinal measurements made under the same conditions do not necessarily represent real change but might be caused by exactly this random measurement variability. Before testing or even utilizing a quantitative biomarker in longitudinal studies, it is therefore of principal importance to assess the measurement repeatability. ⁴

The repeatability of measurement is determined by test–retest studies, which then are also referred to as repeatability studies. In such studies, replicate measurements are made on a sample of subjects under conditions that are as constant as possible. ⁵ Measurement repeatability can be quantified by the within-subject standard deviation ( $w_{\mathrm{SD}}$ ). Using $w_{\mathrm{SD}}$ , the repeatability coefficient ( $RC$ ) can be calculated.^4,6,7 $RC$ is then used in the longitudinal study to determine if a difference in the biomarker represents presumed real change or is within the range of random measurement variability. It is defined in such a way that a desired specificity to detect changes—usually 95%—is targeted.

The $w_{\mathrm{SD}}$ and the $RC$ , as determined by the test–retest study, are point estimates, and hence suffer from random error. As we will show, the targeted specificity is therefore generally not achieved in practice. Following standard statistical results, the more subjects and the more repeated measurements are included in the test–retest study, the more reliable the estimates of $w_{\mathrm{SD}}$ and $RC$ will be. Accordingly, the probability of a relevant deviation of the actually achieved value from the targeted specificity will decrease. The quality of assessments in the longitudinal study and consequently the validity of its results is directly governed by the precision of the estimates of $w_{\mathrm{SD}}$ and $RC$ .

Of course, exact knowledge of measurement repeatability is not only crucial for biomarkers. For example, excellent measurement repeatability of scales and other laboratory instruments is mandatory. The reliability of a scale can be checked using weights with a known mass and it is possible to perform many repeated measurements. In contrast, many biomarkers are measured in-vivo, rendering attainment of large sample sizes difficult. Also, it might be necessary from an ethical point of view to keep sample sizes as low as possible, since the measurement in question might be inconvenient, invasive, or even harmful for the patient or the healthy test person. For example, a biomarker might be derived from computed tomography, which involves ionizing radiation. Yet, if the sample size in the test–retest study is small, there is a high chance of obtaining suboptimal estimates of $RC$ with associated detrimental effects on sensitivity and specificity in the longitudinal study.

Related to but different from repeatability is reproducibility. While repeatability represents the measurement precision under constant conditions, that is, same measurement procedure, same operators, same measuring system, etc., reproducibility is, in contrast, measurement precision under differing conditions as various operators, measuring systems, etc. ⁸ In the following, we will focus solely on repeatability.

Statistical literature concerning requirements for test–retest studies is scarce. One notable study investigating sample size requirements is by Obuchowski and Bullen. ⁶ Their work includes a critical examination of several statistical issues that arise in the use of quantitative imaging biomarkers in clinical applications. The technical performance of these markers is investigated by multiple simulation studies. The repeatability and reproducibility of measurements as well as their bias are analysed. In addition to the standard models, heteroscedasticity and non-linear relationships are also considered. One aspect of their work concerns the relation between the sample size in the test–retest study and the difference between a fixed targeted specificity of 95% and the expected value of the specificity actually achieved in a following longitudinal study. In order to limit this difference to 1 percentage point, the authors give a blanket recommendation for the sample size of test–retest studies with two repeated measurements based on their results from a fixed set of simulation parameters.

Our goal is to expand upon the results of Obuchowski and Bullen ⁶ for the particular problem of specificity. First, we want to introduce new quality criteria for the planning of test–retest studies. For this purpose, random variables are introduced to distinguish between the targeted specificity and the specificity actually achieved in a subsequent longitudinal assessment that is based upon the results of a test–retest study. Furthermore, we will expand the considerations to include sensitivity, which has not been investigated in the literature so far. Finally, we aim to provide analytical solutions. In contrast to simulation studies, this allows for flexible calculation of sample size requirements without restrictions to fixed parameters and also the retrospective assessment of test–retest studies, as we will show. In doing so, we establish a comprehensive framework in which the notions introduced above are precisely defined.

In what follows, we will introduce the model used for our framework and study the aspects of specificity and sensitivity in separate sections. Afterwards, we demonstrate the application of our concepts in a practical example and discuss our results.

2. Definitions

One possible approach to distinguish true change from random variation in the longitudinal study is to estimate measurement variability in a test–retest study. For this purpose, $n$ patients are measured $m$ times within a short period of time, in which their true value presumably does not change. For our considerations we assume independent subjects, for example, measurement of one target per patient. In addition, independent replicate measurements are necessary, that is measurements on a subject need to be made independent of the knowledge of its previous value(s). ⁹ Consequently, we establish the following model for the $j$ -th measurement of the $i$ -th patient $Y_{ij}$ of the test–retest study:

$$Y_{ij}={\mu }_i+{\epsilon }_{ij}$$

(1)

where ${\mu }_i$ is the true value for the $i$ -th patient and ${\epsilon }_{ij}$ is the random error. For our following considerations, it is irrelevant whether the values $({\mu }_i{)}_{i=1,\dots ,n}$ stem from a fixed effects or a random effects model. We assume the random errors to be independent and normally distributed with mean $0$ and variance $w_{\mathrm{SD}}^2$ . ⁶ In particular, it follows that $Y_{ij}\sim \mathcal{N}({\mu }_i,w_{\mathrm{SD}}^2)$ for any $i\in \{1,\dots ,n\}$ and $j\in \{1,\dots ,m\}$ . This model is appropriate when true replicates are studied and a learning effect can be ruled out. As we are only addressing measurement repeatability, a fixed bias does not need to be considered since it cancels out. We also assume that measurement error is independent from the magnitude of ${\mu }_i$ . From this data, we can estimate the within-patient standard deviation $w_{\mathrm{SD}}$ ⁷ by

$${\hat{w}}_{\mathrm{SD}}:=\sqrt{\frac{1}{n}\sum _{i=1}^n\frac{1}{m-1}\sum _{j=1}^m(Y_{ij}-{\overline{Y}}_{i\cdot }{)}^2}$$

where ${\overline{Y}}_{i\cdot }:=1/m\sum _{j=1}^m{Y}_{ij}$ denotes the mean value of the measurements of patient $i$ . In this formula, the patient-specific means $({\mu }_i{)}_{i=1,\dots ,n}$ directly cancel each other out. Following Cochran’s theorem, ¹⁰ the distribution of this entity can be derived from

$$n(m-1)\frac{{\hat{w}}_{\mathrm{SD}}^2}{w_{\mathrm{SD}}^2}\sim {\chi }_{n(m-1)}^2$$

(2)

A two-sided confidence interval for $w_{\mathrm{SD}}$ at level $\alpha$ is given by

$$[\sqrt{\frac{n(m-1)\cdot {\hat{w}}_{\mathrm{SD}}^2}{{\chi }_{n(m-1),1-\frac{\alpha }{2}}^2}},\sqrt{\frac{n(m-1)\cdot {\hat{w}}_{\mathrm{SD}}^2}{{\chi }_{n(m-1),\frac{\alpha }{2}}^2}}]$$

(3)

where ${\chi }_{n(m-1),\alpha }^2$ denotes the $\alpha$ -quantile of the ${\chi }^2$ -distribution with $n(m-1)$ degrees of freedom.

For any fixed $m$ , a central limit theorem can be applied to ${\hat{w}}_{\mathrm{SD}}^2$ for $n\to \mathrm{\infty }$ . A subsequent application of the delta method ¹¹ using the square root function yields the convergence

$$\frac{{\hat{w}}_{\mathrm{SD}}-w_{\mathrm{SD}}}{\frac{w_{\mathrm{SD}}}{\sqrt{2n(m-1)}}}\underset{n\to \mathrm{\infty }}{\stackrel{\mathcal{D}}{\to }}Z$$

(4)

for any fixed $m$ , where $Z$ is a standard normally distributed random variable. Furthermore, this convergence also holds for any fixed $n$ when $m$ goes to infinity. This is a consequence of two subsequent applications of the delta method ¹¹ and the commutativity of addition and convergence in distribution for independent random variables.

If the number of repeated measurements differs between subjects, that is, the $i$ -th subject is measured $m_i$ times, the value $n(m-1)$ needs to be replaced by $\sum _{i=1}^n(m_i-1)$ in all formulas. For the sake of simplicity, we restrict ourselves to the case of an equal number of repetitions $m$ per subject.

In order to assess changes in the measurements of a single patient in the subsequent longitudinal study, the $RC$ is computed. ⁷ It indicates the range in which two repeated measurements are expected to fall with a certain probability. In what follows, we restrict ourselves to the assessment of changes in both directions. We want to keep our decision rules flexible, that is, we establish a target specificity $p_{\mathrm{sp}}\in (0,1)$ which shall be reached for patients with no change in their true biomarker value. Hence $RC$ is a function of $p_{\mathrm{sp}}$ and is given by

$$RC(p_{\mathrm{sp}}):={\mathrm{\Phi }}^{-1}(1-(1-p_{\mathrm{sp}})/2)\cdot \sqrt{2}\cdot w_{\mathrm{SD}}$$

(5)

In most literature, the $RC$ is only considered for a fixed targeted specificity of 95%, that is, $RC(0.95)$ .^4,12

In practice, $w_{\mathrm{SD}}$ is unknown and hence replaced by its consistent estimator ${\hat{w}}_{\mathrm{SD}}$ to obtain the estimated RC

$$\widehat{RC}(p_{\mathrm{sp}}):={\mathrm{\Phi }}^{-1}(1-(1-p_{\mathrm{sp}})/2)\cdot \sqrt{2}\cdot {\hat{w}}_{\mathrm{SD}}$$

A confidence interval for the $RC$ at level $\alpha$ is given by multiplying the limits of the corresponding confidence interval for $w_{\mathrm{SD}}$ in (3) with ${\mathrm{\Phi }}^{-1}(1-(1-p_{\mathrm{sp}})/2)\cdot \sqrt{2}$ .

The point estimate $\widehat{RC}(p_{\mathrm{sp}})$ can be applied as cutpoint in a longitudinal study to determine whether there has been change between two consecutive measurements $Y_{\mathrm{pre}}$ and $Y_{\mathrm{post}}$ . Here, we also assume, that the measured values have independent errors, but the true levels ${\mu }_{\mathrm{pre}}$ and ${\mu }_{\mathrm{post}}$ might actually be different, that is, we have $Y_{\mathrm{pre}}={\mu }_{\mathrm{pre}}+{\epsilon }_{\mathrm{pre}}$ and $Y_{\mathrm{post}}={\mu }_{\mathrm{post}}+{\epsilon }_{\mathrm{post}}$ with ${\epsilon }_{\mathrm{pre}}$ and ${\epsilon }_{\mathrm{post}}$ being independent and normally distributed with mean $0$ and variance $w_{\mathrm{SD}}^2$ .

In case the true values have not changed, that is, ${\mu }_{\mathrm{pre}}={\mu }_{\mathrm{post}}$ , the difference $Y_{\mathrm{post}}-Y_{\mathrm{pre}}$ is normally distributed with mean $0$ and variance $2w_{\mathrm{SD}}^2$ . Hence, following the definition of $RC$ from (5), we obtain in this case

$$p_{\mathrm{sp}}=\mathbb{P}[Y_{\mathrm{post}}-Y_{\mathrm{pre}}\in [-RC(p_{\mathrm{sp}}),RC(p_{\mathrm{sp}})]]$$

The rule to decide whether there is a change in a patient with the two measured values $Y_{\mathrm{pre}}$ and $Y_{\mathrm{post}}$ should thus be whether their difference lies outside or inside the interval $[-RC(p_{\mathrm{sp}}),RC(p_{\mathrm{sp}})]$ . As the bounds are unknown in practice, this decision rule is replaced by the decision rule based on the estimated interval $[-\widehat{RC}(p_{\mathrm{sp}}),\widehat{RC}(p_{\mathrm{sp}})]$ . Consequently, the targeted specificity $p_{\mathrm{sp}}$ will never be exactly met. This applies analogously to considerations for the sensitivity of this procedure.

3. Effective specificity as a criterion for sample size estimation

Our goal is to quantify the uncertainty introduced by the replacement of $w_{\mathrm{SD}}$ by its estimator ${\hat{w}}_{\mathrm{SD}}$ . As mentioned, the targeted specificity ( $p_{\mathrm{sp}}$ ) is not met in practice. To assess this problem, we introduce the effective specificity $P_{\mathrm{esp}}$ which is the specificity actually achieved if a realisation of the estimate ${\hat{w}}_{\mathrm{SD}}$ is plugged in. Hence, $P_{\mathrm{esp}}$ is a random quantity as it depends on the value of ${\hat{w}}_{\mathrm{SD}}$ . We use a capital letter to emphasize that it is indeed a random variable. It can be implicitly defined via

$$RC(P_{\mathrm{esp}})=\widehat{RC}(p_{\mathrm{sp}})$$

(6)

Although this quantity is unknown in practice, we can nevertheless analyse its distribution. Firstly, we can compute the expected value $\mathbb{E}[P_{\mathrm{esp}}]$ and the bias, that is, the difference $\mathbb{E}[P_{\mathrm{esp}}]-p_{\mathrm{sp}}$ . This is also the quantity targeted by Obuchowski and Bullen. ⁶ Their quality criterion requires $|\mathbb{E}[P_{\mathrm{esp}}]-p_{\mathrm{sp}}|$ to be at most 0.01, that is, they want the mean effective specificity to differ by no more than 1 percentage point from the target specificity, which they set to 95%. But what is even more important, from our point of view, is that we can compute quantiles of the distribution of $P_{\mathrm{esp}}$ which will enable us to establish quality guarantees on the effective specificity of the longitudinal studies based on the design parameters $n$ and $m$ of the test–retest study.

3.1. Expected value and bias

According to (6), $P_{\mathrm{esp}}$ is given by

$$\begin{array}{rl}{P}_{\mathrm{esp}}& =R{C}^{-1}(\widehat{RC}(p_{\mathrm{sp}}))\\ & =1-2\cdot (1-\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})\cdot \frac{{\hat{w}}_{\mathrm{SD}}}{w_{\mathrm{SD}}}))\end{array}$$

The function $RC$ can be inverted as it is a continuous, monotonically increasing function on $(0,1)$ . The expectation of this random quantity can be computed exactly using (2) or approximately using the central limit theorem (4), according to which the distribution of ${\hat{w}}_{\mathrm{SD}}/w_{\mathrm{SD}}$ can be approximated with a normal distribution with expectation $1$ and variance $1/(2n(m-1))$ . Hence, we get

$$\begin{array}{rl}\mathbb{E}[P_{\mathrm{esp}}]& =1-2\cdot (1-{\int }_0^{\mathrm{\infty }}\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})\cdot w)f_{{\chi }_{n(m-1)}^2}(n(m-1)w^2)2wn(m-1)dw)\end{array}$$

(7)

$$\begin{array}{rl}& \approx 1-2\cdot (1-{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})\cdot w)\sqrt{\frac{n(m-1)}{\pi }}\exp (-n(m-1)(w-1{)}^2)dw)\end{array}$$

(8)

where $f_{{\chi }_{n(m-1)}^2}$ denotes the probability density function (PDF) of a ${\chi }^2$ -distributed random variable with $n(m-1)$ degrees of freedom. By numerical evaluation of the terms in (7) and (8), the bias can be computed.

3.2. Quantiles of the distribution of $P_{\mathrm{esp}}$

We need to be aware that even if $\mathbb{E}[P_{\mathrm{esp}}]$ is close to $p_{\mathrm{sp}}$ , that is, the bias is low, the probability for a substantial deviation of the actually realized specificity from the targeted specificity might be large (Figure 1).

Figure 1.

(A) Expected value of the effective specificity ( $\mathbb{E}[P_{\mathrm{esp}}]$ ) as a function of $n$ in the test–retest study for a target specificity $p_{\mathrm{sp}}$ of 95% and $m=2$ . Already for a small case number of 10 (solid vertical line), the expected value is comparably high. For a case number of 30 (dotted vertical line), the bias ( $\mathbb{E}[P_{\mathrm{esp}}]-p_{\mathrm{sp}}$ ) is below 1 percentage point and does not change substantially with an increase of the case number to 60 (dashed vertical line). (B) However, while $\mathbb{E}[P_{\mathrm{esp}}]$ is already relatively high for a case number of 10, the tails of the corresponding PDF (solid line) are prominent, resulting in a high chance of obtaining a low $P_{\mathrm{esp}}$ in practice. The dotted and dashed lines represent the PDF of the effective specificity for $n=30$ and $n=60$ , respectively.

Therefore, we want to know with which confidence $p_{\mathrm{conf}}$ we can say that the effective specificity is larger than some lower bound $p_{\mathrm{esp},\mathrm{lb}}$ . This is expressed by the formula

$$\begin{array}{rl}& \mathbb{P}[P_{\mathrm{esp}}\ge p_{\mathrm{esp},\mathrm{lb}}]=p_{\mathrm{conf}}\\ \iff & \mathbb{P}[\widehat{RC}(p_{\mathrm{sp}})\ge RC(p_{\mathrm{esp},\mathrm{lb}})]=p_{\mathrm{conf}}\end{array}$$

We want to introduce a new quality criterion based on this concept.

The quantity $p_{\mathrm{conf}}$ is a function of $p_{\mathrm{esp},\mathrm{lb}}$ and of course also depends on $p_{\mathrm{sp}},n$ , and $m$ . For notational convenience, however, we omit those arguments. After some calculations, one obtains

$$\begin{array}{rl}{p}_{\mathrm{conf}}& =1-F_{{\chi }_{n(m-1)}^2}(\frac{{\mathrm{\Phi }}^{-1}{(1-\frac{1-p_{\mathrm{esp},\mathrm{lb}}}{2})}^2}{{\mathrm{\Phi }}^{-1}{(1-\frac{1-p_{\mathrm{sp}}}{2})}^2}n(m-1))\end{array}$$

(9)

$$\begin{array}{rl}& \approx 1-\mathrm{\Phi }\left((\frac{{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{esp},\mathrm{lb}}}{2})}{{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}-1)\sqrt{2n(m-1)}\right)\end{array}$$

(10)

where $F_{{\chi }_{n(m-1)}^2}$ denotes the cumulative distribution function of a ${\chi }^2$ -distributed random variable with $n(m-1)$ degrees of freedom. These formulas can now be used in different ways. In the above form, one can determine the confidence with which the effective specificity exceeds a fixed bound $p_{\mathrm{esp},\mathrm{lb}}$ with given design parameters $n$ and $m$ of the test–retest study. Analogous considerations can be made for upper bounds by computing the probability of the complementary event. The Supplemental Material contains a simulation study that confirms this key calculation.

In the planning stage of the test–retest study, it could be beneficial to choose the sample size $n$ in such a way that a desired lower bound $p_{\mathrm{esp},\mathrm{lb}}$ is achieved with a prespecified confidence $p_{\mathrm{conf}}$ . To this end, the asymptotic formula (10) can be solved explicitly for $n$ :

$$n\ge \frac{1}{2(m-1)}{\left(\frac{{\mathrm{\Phi }}^{-1}(1-p_{\mathrm{conf}}){\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}{{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{esp},\mathrm{lb}}}{2})-{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}\right)}^2$$

(11)

The exact formula (9) cannot be explicitly solved for $n$ . However, one can numerically solve

$$min\{n\in \mathbb{N}:1-F_{{\chi }_{n(m-1)}^2}(\frac{{\mathrm{\Phi }}^{-1}{(1-\frac{1-p_{\mathrm{esp},\mathrm{lb}}}{2})}^2}{{\mathrm{\Phi }}^{-1}{(1-\frac{1-p_{\mathrm{sp}}}{2})}^2}n(m-1))\ge p_{\mathrm{conf}}\}$$

(12)

In our application example, we will apply these formulas in the planning stage of a hypothetical test–retest study.

The need for a number of measurements to ensure compliance with the quality criteria proposed here cannot only be formulated based on the number of cases $n$ for an arbitrary fixed $m$ . Although this is common practice, the formulas given here show that the product $n(m-1)$ is decisive. Hence, if the number of replicated measurements $m$ can also be chosen freely, the total number of measurements $N=mn$ can be reduced by increasing $m$ . In this sense, setting $n=1$ and $m=N$ is the most efficient configuration. However, such a design of a test–retest study might be regarded inadvisable for several reasons as laid down in the discussion.

If one wants to identify the worst possible cases for given $n$ and $m$ , one could compute the lower bound of the effective specificity which is reached with confidence $p_{\mathrm{conf}}$ :

$$\begin{array}{rl}{p}_{\mathrm{esp},\mathrm{lb}}& =1-2(1-\mathrm{\Phi }\left(\sqrt{\frac{F_{{\chi }_{n(m-1)}^2}^{-1}(1-p_{\mathrm{conf}})}{n(m-1)}}{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})\right))\end{array}$$

(13)

$$\begin{array}{rl}& \approx 1-2(1-\mathrm{\Phi }(\frac{{\mathrm{\Phi }}^{-1}(1-p_{\mathrm{conf}}){\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}{\sqrt{2n(m-1)}}+{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})))\end{array}$$

(14)

Accordingly, in $(1-p_{\mathrm{conf}})\cdot 100\mathrm{\%}$ of all cases, the effective specificity will be even lower than the obtained $p_{\mathrm{esp},\mathrm{lb}}$ .

From our point of view, the probability of exceeding a lower bound $p_{\mathrm{esp},\mathrm{lb}}$ is a valid criterion for evaluating the quality of assessment in a longitudinal study. Different from the expected value of $P_{\mathrm{esp}}$ which has been previously proposed as a quality criterion, ⁶ our criterion considers the tails of the distribution of $P_{\mathrm{esp}}$ . This allows to bound the probability of strongly deviating from the desired specificity.

4. Consideration of effective sensitivity

Concerning the sensitivity, that is, the ability to detect real change between two measurements of one patient in the longitudinal study, we can make similar considerations. Before coming back to the problem of the uncertainty caused from the estimation of $w_{\mathrm{SD}}$ , we first assume, that $w_{\mathrm{SD}}$ and hence also $RC(p_{\mathrm{sp}})$ is known. Of course, the sensitivity strongly depends on the difference between ${\mu }_{\mathrm{pre}}$ and ${\mu }_{\mathrm{post}}$ . Also, such differences are more difficult to detect if $w_{\mathrm{SD}}$ is large and a large target specificity is chosen. To be more precise, the sensitivity $p_{\mathrm{se}}$ to detect a difference can be written as a function of ${\mu }_{\mathrm{\Delta }}:={\mu }_{\mathrm{post}}-{\mu }_{\mathrm{pre}}$ , $w_{\mathrm{SD}}$ and the chosen specificity $p_{\mathrm{sp}}$ . If the intercepts ${\mu }_{\mathrm{post}}$ and ${\mu }_{\mathrm{pre}}$ are considered random, all the following probability statements are conditional on ${\mu }_{\mathrm{\Delta }}$ . Then, this relationship is given by

$$\begin{array}{rl}{p}_{\mathrm{se}}({\mu }_{\mathrm{\Delta }},w_{\mathrm{SD}})& :=\mathbb{P}[Y_{\mathrm{post}}-Y_{\mathrm{pre}}\notin [-RC(p_{\mathrm{sp}}),RC(p_{\mathrm{sp}})]]\\ & =1-(\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}})-\mathrm{\Phi }({\mathrm{\Phi }}^{-1}\left(\frac{1-p_{\mathrm{sp}}}{2}\right)-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}}))\end{array}$$

In this form, the function can also be seen as a function of the effect size $\delta :={\mu }_{\mathrm{\Delta }}/w_{\mathrm{SD}}$ , that is,

$$\begin{array}{rl}{p}_{\mathrm{se}}(\delta )& :=1-(\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})-\frac{\delta }{\sqrt{2}})-\mathrm{\Phi }({\mathrm{\Phi }}^{-1}\left(\frac{1-p_{\mathrm{sp}}}{2}\right)-\frac{\delta }{\sqrt{2}}))\end{array}$$

(15)

This dependence of the sensitivity on the effect size $\delta$ is visualized in Figure 2.

Figure 2.

Sensitivity as a function of effect size $\delta :={\mu }_{\mathrm{\Delta }}/w_{\mathrm{SD}}$ for $p_{\mathrm{sp}}$ =0.95. Note that we assume $w_{\mathrm{SD}}$ to be known here.

As $w_{\mathrm{SD}}$ is unknown and needs to be estimated by ${\hat{w}}_{\mathrm{SD}}$ which will then be plugged in to compute $\widehat{RC}(p_{\mathrm{sp}})$ , the sensitivity computed in (15) will not be reached. Analogously to our considerations for the specificity, we introduce the effective sensitivity $P_{\mathrm{ese}}$ which is the sensitivity that is actually achieved if a realization of the estimate ${\hat{w}}_{\mathrm{SD}}$ is plugged in. Of course, it is also a random variable and does depend again on ${\mu }_{\mathrm{\Delta }}$ , $w_{\mathrm{SD}}$ and $p_{\mathrm{sp}}$ . It can be defined by the equation

$$\begin{array}{rl}{P}_{\mathrm{ese}}({\mu }_{\mathrm{\Delta }},w_{\mathrm{SD}})& :=\mathbb{P}[Y_{\mathrm{post}}-Y_{\mathrm{pre}}\notin [-\widehat{RC}(p_{\mathrm{sp}}),\widehat{RC}(p_{\mathrm{sp}})]|{\hat{w}}_{\mathrm{SD}}]\\ & =1-(\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})\frac{{\hat{w}}_{\mathrm{SD}}}{w_{\mathrm{SD}}}-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}})-\mathrm{\Phi }({\mathrm{\Phi }}^{-1}\left(\frac{1-p_{\mathrm{sp}}}{2}\right)\frac{{\hat{w}}_{\mathrm{SD}}}{w_{\mathrm{SD}}}-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}}))\end{array}$$

(16)

With this expression and the exact distribution of ${\hat{w}}_{\mathrm{SD}}$ given as in (2) resp. the approximation of the distribution of $\frac{{\hat{w}}_{\mathrm{SD}}}{w_{\mathrm{SD}}}$ by a normal distribution from (4) we can now quantify the bias caused by the replacement of $w_{\mathrm{SD}}$ by ${\hat{w}}_{\mathrm{SD}}$ and compute quantiles of the distribution of $P_{\mathrm{ese}}$ which will enable us to also give quality guarantees on the effective sensitivity. Unlike our considerations for the specificity, these values will also depend from the actual $w_{\mathrm{SD}}$ and the difference ${\mu }_{\mathrm{\Delta }}$ of the longitudinal study and hence will be regarded as functions thereof.

4.1. Expected value and bias

To compute the bias in dependence from $p_{\mathrm{sp}}$ , ${\mu }_{\mathrm{\Delta }}$ , and $w_{\mathrm{SD}}$ , we can take the expectation of the right hand side of (16) and use the exact distribution (2) and the central limit theorem (4) to obtain the result

$$\begin{array}{rl}\mathbb{E}[P_{\mathrm{ese}}(p_{\mathrm{sp}},{\mu }_{\mathrm{\Delta }},w_{\mathrm{SD}})]& =1-{\int }_0^{\mathrm{\infty }}\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})\cdot w-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}})f_{{\chi }_{n(m-1)}^2}(n(m-1)w^2)2wn(m-1)dw\\ & +{\int }_0^{\mathrm{\infty }}\mathrm{\Phi }({\mathrm{\Phi }}^{-1}\left(\frac{1-p_{\mathrm{sp}}}{2}\right)\cdot w-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}})f_{{\chi }_{n(m-1)}^2}(n(m-1)w^2)2wn(m-1)dw\\ & \approx 1-{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})\cdot w-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}})\sqrt{\frac{n(m-1)}{\pi }}\exp (-n(m-1)(w-1{)}^2)dw\\ & +{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\mathrm{\Phi }({\mathrm{\Phi }}^{-1}\left(\frac{1-p_{\mathrm{sp}}}{2}\right)\cdot w-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}})\sqrt{\frac{n(m-1)}{\pi }}\exp (-n(m-1)(w-1{)}^2)dw\end{array}$$

(17)

Please note that this can essentially be seen as a function of $\delta$ . Following (16), the bias of the effective sensitivity can be considered a function of $\delta$ for any given $p_{\mathrm{sp}}$ , i.e. $\mathbb{E}[P_{\mathrm{ese}}(p_{\mathrm{sp}},\delta )]-p_{\mathrm{se}}(p_{\mathrm{sp}},\delta )$ .

4.2. Quantiles of the distribution of $P_{\mathrm{ese}}$

For the most accurate examination of the distribution of $P_{\mathrm{ese}}$ , we would need to consider both events

$$\begin{array}{r}\begin{array}{rl}& Y_{\mathrm{post}}-Y_{\mathrm{pre}}>\widehat{RC}(p_{\mathrm{sp}})\mathrm{and}\\ & Y_{\mathrm{post}}-Y_{\mathrm{pre}}<-\widehat{RC}(p_{\mathrm{sp}})\end{array}\end{array}$$

(18)

However, this leads to expressions that are difficult to handle analytically. Actually, the two probabilities

$$\begin{array}{rl}& \mathbb{P}[Y_{\mathrm{post}}-Y_{\mathrm{pre}}>\widehat{RC}(p_{\mathrm{sp}})|{\hat{w}}_{\mathrm{SD}}]\text{ and }\end{array}$$

(19)

$$\begin{array}{rl}& \mathbb{P}[Y_{\mathrm{post}}-Y_{\mathrm{pre}}<-\widehat{RC}(p_{\mathrm{sp}})|{\hat{w}}_{\mathrm{SD}}]\end{array}$$

(20)

sum up to the effective sensitivity. However, in the presence of an effect, one of them will be much larger than the other. In the case $\delta >0$ , the probability in (19) is larger than that from (20) which is bounded from above by 0.025 and quickly converges to 0 as $\delta$ increases. To enable the derivation of analytical formulas, we will therefore restrict ourselves to the consideration of $\delta >0$ and the event (18). It is nevertheless possible to circumvent this simplification by numerical inversion of the relationship given in (16). But here, we will approximate

$$\begin{array}{rl}{P}_{\mathrm{ese}}(p_{\mathrm{sp}},{\mu }_{\mathrm{\Delta }},w_{\mathrm{SD}})& \approx \mathbb{P}[Y_{\mathrm{post}}-Y_{\mathrm{pre}}>\widehat{RC}(p_{\mathrm{sp}})|{\hat{w}}_{\mathrm{SD}}]\\ & =1-\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})\frac{{\hat{w}}_{\mathrm{SD}}}{w_{\mathrm{SD}}}-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}})\end{array}$$

In analogy to the previous section, we can provide confidence levels $p_{\mathrm{conf}}$ which indicate the probability that the effective sensitivity for some effect $\delta$ exceeds the lower bound $p_{\mathrm{ese},\mathrm{lb}}$ :

$$\begin{array}{rl}{p}_{\mathrm{conf}}& =\mathbb{P}[P_{\mathrm{ese}}(p_{\mathrm{sp}},\delta )\ge p_{\mathrm{ese},\mathrm{lb}}]\\ & \approx F_{{\chi }_{n(m-1)}^2}({\left(\frac{{\mathrm{\Phi }}^{-1}(1-p_{\mathrm{ese},\mathrm{lb}})+\delta /\sqrt{2}}{{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}\right)}^{2}n(m-1))\\ & \approx \mathrm{\Phi }\left((\frac{{\mathrm{\Phi }}^{-1}(1-p_{\mathrm{ese},\mathrm{lb}})+\delta /\sqrt{2}}{{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}-1)\sqrt{2n(m-1)}\right)\end{array}$$

(21)

Of course, such considerations only make sense if $p_{\mathrm{se}}>p_{\mathrm{ese},\mathrm{lb}}$ for the chosen effect size $\delta$ . As above, analogous considerations can be made for upper bounds by computing the probability of the complementary event.

While (21) allows to compute the confidence of reaching a certain lower bound of the sensitivity for an effect $\delta$ , this formula may also be transformed to be used in the planning stage of the test–retest study. If one wants to achieve a fixed confidence with which the effective sensitivity for an effect size $\delta$ exceeds some lower bound, one can use the exact results from above or the approximations made thereafter to determine the sample size $n$ of the test–retest study in which each patient is measured $m$ times. It shall be chosen such that

$$\begin{array}{rl}n& =min\{n\in \mathbb{N}:\mathbb{P}[1-(\mathrm{\Phi }({\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})\frac{{\hat{w}}_{\mathrm{SD}}}{w_{\mathrm{SD}}}-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}})\\ & -\mathrm{\Phi }({\mathrm{\Phi }}^{-1}\left(\frac{1-p_{\mathrm{sp}}}{2}\right)\frac{{\hat{w}}_{\mathrm{SD}}}{w_{\mathrm{SD}}}-\frac{{\mu }_{\mathrm{\Delta }}}{\sqrt{2}{w}_{\mathrm{SD}}}))\ge p_{\mathrm{ese},\mathrm{lb}}]\ge p_{\mathrm{conf}}\}\end{array}$$

(22)

$$\begin{array}{rl}& \approx min\{n\in \mathbb{N}:F_{{\chi }_{n(m-1)}^2}({\left(\frac{{\mathrm{\Phi }}^{-1}(1-p_{\mathrm{ese},\mathrm{lb}})+\delta /\sqrt{2}}{{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}\right)}^{2}n(m-1))\ge p_{\mathrm{conf}}\}\end{array}$$

(23)

$$\begin{array}{rl}& \approx \frac{1}{2(m-1)}{\left(\frac{{\mathrm{\Phi }}^{-1}(p_{\mathrm{conf}}){\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}{{\mathrm{\Phi }}^{-1}(1-p_{\mathrm{ese},\mathrm{lb}})+\delta /\sqrt{2}-{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}\right)}^2\end{array}$$

(24)

Analogous to the preceding section, we can use these results in the planning stage of a test–retest study, as we will demonstrate in the following application example. Even if a study is planned based on considerations of the specificity, the formulas allow to assess the distribution of the effective sensitivity for any given effect size of interest.

Calculations for $\delta <0$ follow analogously to the considerations for $\delta >0$ .

5. Application example

To illustrate our considerations, we will discuss a hypothetical application for early treatment response assessment in recurrent or metastatic nasopharyngeal carcinoma. While some patients with recurrent nasopharyngeal carcinoma show response or stable disease after systemic treatment, many patients will have progressive disease, which is invariably lethal.^13,14 Nevertheless, futile treatments should be avoided due to associated toxicity.^13,15 To suspend futile treatment as soon as possible an imaging biomarker is desirable which accurately classifies treatment response earlier than change in morphologic lesion size, the current standard. A promising biomarker in this context is diffusion weighted magnetic resonance imaging (DWI). ¹⁶ DWI depends on the differences in the movement of water molecules based on Brownian motion, which can be quantified by the apparent diffusion coefficient (ADC). An exemplary measurement of ADC is shown in Figure 3. Change in ADC has shown promise as an early treatment response marker in various tumors, including nasopharyngeal carcinoma.^16–19

Figure 3.

Example case of a tumor in the right nose showing restricted diffusion (A) with a mean apparent diffusion coefficient (ADC) of $610\times {10}^{-6}$ mm²/s. (B) Region of interest outlined in yellow.

5.1. Prospective planning of test–retest studies

As laid out above, before conducting a longitudinal study in which a biomarker is applied to assess treatment response, a test–retest study should be conducted to assess repeatability. In our example, we will set $p_{\text{sp}}$ to 95% and $m=2$ , as these are the usual values in the literature. We imagine the researcher would want to obtain a specificity of at least 90% ( $p_{\mathrm{esp},\mathrm{lb}}$ ) with 95% certainty ( $p_{\mathrm{conf}}$ ) in the longitudinal study. What sample size ( $n$ ) is necessary in the test–retest study? This question can be answered using the asymptotic formula (11):

$$\begin{array}{rl}& n\ge \frac{1}{2(m-1)}{\left(\frac{{\mathrm{\Phi }}^{-1}(1-p_{\mathrm{conf}}){\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}{{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{esp},\mathrm{lb}}}{2})-{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}\right)}^2\\ \iff & n\ge \frac{1}{2(2-1)}{\left(\frac{{\mathrm{\Phi }}^{-1}(0.05){\mathrm{\Phi }}^{-1}\left(0.975\right)}{{\mathrm{\Phi }}^{-1}\left(0.95\right)-{\mathrm{\Phi }}^{-1}\left(0.975\right)}\right)}^2\\ \iff & n\ge 52.3\end{array}$$

This can also be concluded from Figure 4(A). Numerical solution of the exact formula (12) yields a sample size of 54. Resulting sample sizes for other values of $p_{\mathrm{esp},\mathrm{lb}}$ can be taken from Figure 4(B). The resulting scenario in terms of the distribution of the relative error in the estimation of ${\hat{w}}_{\mathrm{SD}}$ and its effect on $P_{\mathrm{esp}}$ is displayed in Figure 5(A). As already noted in Section 3.2, it can be seen in Figure 4(B) that the overall number of measurements $N=mn$ can be reduced by increasing $m$ . Here, for $m=3$ , a total number of $81=3\times 27$ measurements is necessary, while for $n=2$ , $108=2\times 54$ are required. Taking this to the extreme, $55$ measurements of only one patient would yield the same result in terms of the distribution of $P_{\mathrm{esp}}$ , although this may not be advisable.

Figure 4.

(A) Lower bound of effective specificity reached with confidence of 95% as a function of sample size ( $n$ ) and number of repeated measurements ( $m$ ) of the test–retest study. (B) Sample size resulting from (12) for different values of the desired lower bound $p_{\mathrm{esp},\mathrm{lb}}$ that shall be exceeded with a fixed confidence of 95%.

Figure 5.

Visualization of the application example. The x-axis denotes the relative error between ${\hat{w}}_{\mathrm{SD}}$ and $w_{\mathrm{SD}}$ . The solid black line represents the asymptotic probability density function (PDF) of the relative error. The dashed line shows the effective specificity. Analogously, the dotted line shows the effective sensitivity for an underlying effect size of $\delta =4$ . (A) For $n=53$ and $m=2$ : The area shaded in light gray represents 95% of the area under the normal curve. That is, there is a 95% chance of obtaining a ${\hat{w}}_{\mathrm{SD}}$ from the test–retest study that will result in an effective specificity of >90%. (B) For $n=139$ and $m=2$ : The area shaded in light gray represents 95% of the area under the normal curve. That is, there is a 95% chance of obtaining a ${\hat{w}}_{\mathrm{SD}}$ from the test–retest study that will result in a specificity of >75%. In this case, an effective specificity of 92.27% will be reached with a certainty of 95%.

Analogous considerations can be made for the effective sensitivity. We consider the sensitivity for an underlying true effect size of $\delta =4$ in a study with $p_{\mathrm{sp}}=0.95$ and $m=2$ . According to formula (15), a sensitivity of 80.74% was achieved if $w_{\mathrm{SD}}$ was a known quantity. However, this will not be met in practice. What is the minimum sample size ( $n$ ) of the test–retest study such that we can be 95% ( $p_{\mathrm{conf}}$ ) sure to achieve at least a sensitivity of 75% ( $p_{\mathrm{ese},\mathrm{lb}}$ ) for that effect size? This question can be answered using the approximate formula (24).

$$\begin{array}{rl}& n\approx \frac{1}{2(m-1)}{\left(\frac{{\mathrm{\Phi }}^{-1}(p_{\mathrm{conf}}){\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}{{\mathrm{\Phi }}^{-1}(1-p_{\mathrm{se},\mathrm{lb}})+\delta /\sqrt{2}-{\mathrm{\Phi }}^{-1}(1-\frac{1-p_{\mathrm{sp}}}{2})}\right)}^2\\ \Rightarrow & n\approx {\left(\frac{{\mathrm{\Phi }}^{-1}(0.95){\mathrm{\Phi }}^{-1}\left(0.975\right)}{{\mathrm{\Phi }}^{-1}\left(0.25\right)+4/\sqrt{2}-{\mathrm{\Phi }}^{-1}\left(0.975\right)}\right)}^2\\ \iff & n\approx 138.1\end{array}$$

Accordingly, a sample size of 139 patients in the test–retest study would be recommended to achieve the set targets. Using the exact formula (13) or the asymptotic formula (14), an effective specificity of at least 92.25% resp. 92.27% is reached with a certainty of 95%, in this scenario. This is also depicted by Figure 5(B).

5.2. Retrospective assessment of test–retest studies

Conducting a test–retest study prior to a longitudinal study may not always be necessary. The $RC$ used in the longitudinal study might be adopted from already published test–retest studies. If one intends to use the point estimate of the $RC$ obtained in a previous study, it is advisable to retrospectively assess the resulting distribution of the effective specificity and sensitivity. This allows to evaluate the impact of the sample size of the used test–retest study on quality criteria of the longitudinal study, especially the probability of exceeding a given $p_{\mathrm{esp},\mathrm{lb}}$ .

Common sample sizes in test–retest studies are around 10 and 20.^20–25 If the point estimator of $RC(0.95)$ resulting from a test–retest study with a sample size of 10 and two repeated measurements is used, the distribution of the effective specificity will have prominent tails as illustrated in Figure 1. According to (13), for sample sizes of 10 and 20, the lower bounds of the effective specificity obtained with 95% confidence are 0.7814 and 0.8512, respectively (Figure 4). This might be insufficient. Note that for the recommendation by Obuchowski and Bullen ⁶ of a sample size of 35 for test–retest studies with $m=2$ the probability of achieving an effective specificity below 94% is 39.74%.

Such considerations are also possible for the effective sensitivity.

6. Discussion

We have established a comprehensive framework for planning of test–retest studies concerning repeatability. It enables flexible calculation of sample size requirements and retrospective assessment of such studies with regard to different quality criteria.

To better discuss planning of test–retest studies we have introduced the notions of effective specificity ( $P_{\mathrm{esp}}$ ) and effective sensitivity ( $P_{\mathrm{ese}}$ ), allowing for clearer differentiation of the targeted specificity $p_{\mathrm{sp}}$ and sensitivity $p_{\mathrm{se}}$ from the values actually achieved in the longitudinal study. Both $P_{\mathrm{esp}}$ and $P_{\mathrm{ese}}$ are random quantities and their actual values are unknown in practical application. However, we can determine their distribution and thus can compute different characteristics which properly reflect the uncertainty caused by the estimation process.

Expanding on the work of Obuchowski and Bullen, ⁶ we have introduced a new quality criterion for sample size calculation of test–retest studies. In their work, Obuchowski and Bullen ⁶ demand that the mean effective specificity ( $\mathbb{E}[P_{\mathrm{esp}}]$ ) deviates at most by 0.01 from the fixed targeted specificity ( $p_{\mathrm{sp}}$ ) of 0.95. However, using the mean effective specificity as sole quality criterion has limitations, since the whole distribution of the effective specificity is not properly taken into account. As illustrated in Figure 1, there is a high probability that the actually achieved effective specificity deviates strongly from its target even if the mean effective specificity may be close to the targeted specificity. Therefore, we propose a quality criterion for sample size calculations based on the probability that the effective specificity exceeds a chosen lower bound, taking into account the tails of the distribution of $P_{\mathrm{esp}}$ .

Typically, the number of repeated measurements per patient is set to $m=2$ . Nevertheless, efficiency gains in the total number of measurements of the test–retest study can be attained by increasing $m$ , as the distribution of $P_{\mathrm{esp}}$ only depends from the product $n(m-1)$ . However, caution is advised here as a very small number of subjects with a large number of repeated measurements may come with some disadvantages. From an ethical point of view, it can be unjustified to subject a patient to a high number of potentially harmful examinations. But also from a statistical point of view this can be unfavorable. Assumptions of the underlying model, such as homoskedasticity, cannot be investigated properly in such a trial and habituation effects after a large number of measurements could bias the estimate of $w_{\mathrm{SD}}$ .

In contrast to previous works, we expand our consideration also to issues of sensitivity. Here, of course, it must also be taken into account that the sensitivity depends on the underlying effect size. Nevertheless, we can determine the distribution of the effective sensitivity for any effect size and provide analogous sample size formulas as for the specificity.

Finally, our study is the first to provide analytical rather than simulation results. This provides greater flexibility as the targeted specificity $p_{\mathrm{sp}}$ and number of repeated measurements $m$ may be chosen freely. Hence, it allows the readers to avoid conducting time-consuming simulation studies themselves. While our formulas enable flexible calculations for all scenarios, for convenience of the reader we also provide a table with sample sizes for some exemplary scenarios in Figure 4(B). Sample sizes resulting from other choices of the parameters $p_{\mathrm{conf}}$ , $p_{\mathrm{esp},\mathrm{lb}}$ , $p_{\mathrm{sp}}$ , and $m$ can be found in Supplemental Tables S1 to S4.

A larger sample size than often found in the literature for test–retest studies is required if a high quality of the estimate ${\hat{w}}_{\mathrm{SD}}^2$ is desired according to the choices of $p_{\mathrm{conf}}$ , $p_{\mathrm{esp},\mathrm{lb}}$ , and $p_{\mathrm{ese},\mathrm{lb}}$ . From a purely statistical perspective, multiple estimates ${\hat{w}}_{\mathrm{SD},1},\dots ,{\hat{w}}_{\mathrm{SD},\mathrm{k}}$ of the within-patient standard deviation can be combined (e.g. from different studies) in a meta-analytic way to meet this increased sample size requirement by defining

$${\hat{w}}_{\mathrm{SD}}^2:=\sum _{i=1}^k\frac{n_i(m_i-1)}{d}{\hat{w}}_{\mathrm{SD},\mathrm{i}}^2\text{with}d:=\sum _{i=1}^k{n}_i(m_i-1)$$

The properties of this aggregated estimate can be investigated using all the tools introduced in our work as

$$d\frac{{\hat{w}}_{\mathrm{SD}}^2}{w_{\mathrm{SD}}^2}\sim {\chi }_d^2$$

However, for this combination to be sensible, it must be ensured that all of these estimators estimate the same quantity.

Our study has some limitations. The field of application is restricted to test–retest studies in which true replicates of measurements are possible, for example, in quantitative imaging markers. Our considerations are not valid if the measurement process itself results in a change of the measurand (learning/practice effect) as has been described for some psychological assessments.^26,27

Our standard model (1) assumes independent and identically normally distributed errors. It is, therefore, advisable to examine whether there is a relationship between the within-subject variation and the level of the measured value before applying our approach. ²⁸ If the variability of the measurement error increases with the magnitude of the measured value, a log transformation might resolve the issue.^29,30,28 Beyond that, non-normally distributed error terms are not covered so far. We also have not specifically considered the scenario of clustered data, for example, measuring multiple lesions per subject. However, if a hierarchical model structure with independent errors can be assumed, this does not pose a restriction to application of our approach.

It should be noted that exact solutions based on the ${\chi }^2$ -distribution for all our considerations are available. In some cases, when an analytic solution is not possible, these exact solutions require the application of numerical methods. In order to give completely analytic solutions, some of our formulas rely on asymptotic results and approximations. The differences between exact and approximate results are most severe for small sample sizes and small effect sizes. Applying both exact and approximate formulas in our application example, it can be seen that these differences are negligible in practically relevant scenarios. However, we recommend using the exact calculations for a final planning of a test-retest study. Implementations of exact and approximate solutions can be found in our Supplemental R Code. ³¹

So far, our considerations are limited to repeatability, that is, assuming same measurement conditions for the repeated measurements. However, for real world application of biomarkers, consideration of reproducibility is also important since longitudinal measurements are often performed under different measuring conditions, for example, varying readers or scanners. Therefore, our model should be enhanced to include aspects of reproducibility such as a fixed bias as, for example, in some models considered by Obuchowski and Bullen ⁶ or multiple variance components that take into account different sources of variability in such settings. Nevertheless, since repeatability limits reproducibility, a good knowledge of the former is useful in order to interpret reproducibility studies properly. ⁹

Test–retest studies of repeatability should be well planned to guarantee for a sufficient quality of dependent longitudinal studies. Our framework allows the derivation of analytical solutions for quality criteria that can be used to assess implications of the test–retest study design on subsequent longitudinal studies.

Supplemental Material

sj-r-1-smm-10.1177_09622802241227959 - Supplemental material for A statistical framework for planning and analysing test–retest studies of repeatability

Supplemental material, sj-r-1-smm-10.1177_09622802241227959 for A statistical framework for planning and analysing test–retest studies of repeatability by Moritz Fabian Danzer, Maria Eveslage, Dennis Görlich and Benjamin Noto in Statistical Methods in Medical Research

Supplemental Material

sj-pdf-2-smm-10.1177_09622802241227959 - Supplemental material for A statistical framework for planning and analysing test–retest studies of repeatability

Supplemental material, sj-pdf-2-smm-10.1177_09622802241227959 for A statistical framework for planning and analysing test–retest studies of repeatability by Moritz Fabian Danzer, Maria Eveslage, Dennis Görlich and Benjamin Noto in Statistical Methods in Medical Research