Stata tip 154: Computing power and sample size for prospective diagnostic accuracy studies using Stata’s official power commands

1 Introduction

A major issue that researchers face when planning prospective studies to evaluate the accuracy of diagnostic tests is that they do not know beforehand how many participants will have the disease or not. A complicating factor is that the required sample size needed to produce a given power is highly dependent on the prevalence of the disease in the population under study. When the prevalence is low, an immense sample size (possibly in the tens of thousands) may be required to net sufficiently many diseased participants to ensure that adequate power is achieved (Li and Fine 2004; Obuchowski and Zhou 2002).

Stata’s official power commands for proportions (power oneproportion, power twoproportions, and power pairedproportions) can be easily modified to compute sample size in prospective studies for the two most common binary measures of diagnostic accuracy—sensitivity and specificity. Sensitivity is the proportion of true positive cases that are correctly classified by the test as having the disease, and specificity is the proportion of true negative cases that are correctly classified by the test as not having the disease (Linden 2006). High sensitivity indicates that few cases of true disease are missed, while high specificity implies that there are few false-positive results. It is generally desirable for diagnostic tests to have both high sensitivity and high specificity.

To account for the known prevalence of disease in studies evaluating sensitivity, one uses power to estimate the sample size for the diseased group (n ₁), which is then divided by the prevalence of disease in the population under study (p) to derive the total sample size n_T = n ₁ /p. When evaluating specificity, one uses power to estimate the sample size for the nondiseased group (n ₀), which is then divided by 1-prevalence of disease in the population to derive the total sample size n_T = n ₀ /(1 − p) (Obuchowski and Zhou 2002).

In the following sections, I describe how to use Stata’s official power commands to estimate sample size and power for sensitivity and specificity in prospective studies, accounting for the known prevalence of disease in the population.

2 One-sample proportion test

During development of a new diagnostic test, the usual goal is simply to determine whether the diagnostic test can discriminate between those with and without the disease. Commonly, this is evaluated using a one-sample proportion test (see [R] prtest ) or a binomial probability test (see [R] bitest ) in which the null hypothesis states that the expected sensitivity (specificity) of the new test is equal to the sensitivity (specificity) of an existing test, while the alternative hypothesis is that the sensitivities (specificities) differ.

2.1 Sensitivity

Let’s assume there is a rare disease (3% prevalence in the population) that currently requires an expensive lab test for diagnosis and achieves a sensitivity of 70%. We developed a new diagnostic test that can be administered by the patient themselves at a fraction of the cost of the lab test and is expected to have a better sensitivity and specificity profile. For our prospective study of the new diagnostic test, we will determine sample size for a sensitivity of 90% at an alpha of 0.05 with 80% power using a one-sided test. Using power oneproportion, we plug in these initial values for sensitivity. We specify the nfractional option to provide a fractional sample size because in the next step (when we compute total sample size n_T ), we will round up to the nearest integer using the ceil() function.

The estimated sample size of 25.3 refers to the number of diseased participants needed for the study. We derive the total sample size by dividing the sample size for the diseased group by the prevalence of disease in the population, n_T = n ₁ /p:

The total estimated sample size is 867. Subtracting the expected sample size for the diseased group from the total sample size gives us the expected sample size of the nondiseased group, n ₀ = n_T − n ₁ = 841.

We can also easily evaluate the actual power of this test by specifying the option n(26), which is the rounded-up value produced in the initial sample-size computation above:

As the output indicates, the true power of the test is 81.24%.

2.2 Specificity

Following from the above example, let’s assume that the currently available test has a specificity of 85%. Using power oneproportion, we will determine sample size for a specificity of 92%, at an alpha of 0.05 and 80% power using a one-sided test:

Here the estimated sample size of 135.77 refers to the number of nondiseased participants (n ₀) needed for the study. We derive the total expected sample size as n_T = n ₀ /(1 − p) and the sample size of the diseased group as n ₁ = n_T − n ₀:

As displayed, the total estimated sample size needed for the study is 140, where we expect that 4 participants will have the disease and 136 will not have the disease. The power of the test can be computed by specifying n(136) in the power oneproportion syntax in lieu of specifying power:

3 Two-sample proportion test for independent samples

In many studies, investigators are interested in comparing the accuracies of two diagnostic tests. While a paired-sample design is preferred because it requires fewer patients than the independent two-sample design, in some circumstances a paired design is not possible; for example, the tests are mutually exclusive because of their invasiveness, or it is unethical to expose patients to both tests’ risk of complications or discomfort, or there are time constraints on treatment that do not allow more than one test to be performed (Zhou, Obuchowski, and McClish 2011). Commonly, these tests are evaluated using a two-sample proportion test (see [R] prtest ) or a χ ² test (see [R] tabulate twoway ) where the null hypothesis states that the sensitivities (specificities) of the two tests are equal, while the alternative hypothesis is that the two sensitivities (specificities) differ.

3.1 Sensitivity

Let’s assume there is an existing diagnostic test to detect a disease with 30% prevalence in the clinical population with 81% sensitivity. We have a new test that we believe can diagnose the disease with 90% sensitivity. For our prospective study, we will compare the sensitivity of the new test with the existing test using an independent two-sample approach, where half the participants will be assigned to diagnostic test 1 and half will be assigned to diagnostic test 2. We compute sample size using power twoproportions, set alpha at 0.05, and set power to 80%:

The estimated sample size of 478.16 refers to the number of participants expected to have the disease—with half of them assigned to diagnostic test 1 and half of them assigned to diagnostic test 2. As before, n_T = n ₁ /p and n ₀ = n_T − n ₁:

The output shows that the required total sample size is 1,597 (half assigned to diagnostic test 1 and half assigned to diagnostic test 2) with the expectation that 1,118 of the participants will be nondiseased.

3.2 Specificity

Following with the example above for sensitivity, let’s assume that the specificity of the existing diagnostic test is 82% and we believe that the new test will elicit a specificity of 92%. All the other specifications remain the same:

The results show an estimated total sample size of 504, with an expectation that 353 participants will be nondiseased and 151 will have the disease.

4 Paired two-sample test

In a paired design, the same patients undergo all the tests being evaluated in the study. As with the independent two-sample test, the null hypothesis states that the sensitivities (specificities) of the two tests are equal, while the alternative hypothesis is that the two sensitivities (specificities) differ. Commonly, these tests are evaluated using McNemar’s χ ² test (see [R] Epitab [specifically, mcc and mcci]), where the null hypothesis states that the marginal proportions (which in this case are synonymous with sensitivity and specificity) of the two tests are equal, while the alternative hypothesis is that the two marginal proportions differ.

4.1 Sensitivity

Let’s assume there is an existing diagnostic test that has a sensitivity of 85% to detect a disease with 20% prevalence in the population under study. We developed a new diagnostic test that is less expensive than the existing test, and we believe it may achieve a sensitivity of 95%. We want to design a paired, prospective study in which each participant will be given both the old and the new diagnostic tests. To assess the sample size needed for this study, we require 90% power and a two-tailed alpha level of 0.01. To compute the minimum sample size using power pairedproportions, we specify the sensitivities as marginal proportions and specify the hypothesized correlation between the two tests using the corr() option. Here we will assume that the correlation is 0.50:

The output shows that the expected number of diseased participants is 154.45. We now compute n_T = n ₁ /p and n ₀ = n_T − n ₁:

In summary, to achieve the desired 90% power at an alpha of 0.01, we need to enroll at least 775 participants, of which we expect that 155 will have the disease and 620 will be disease free.

4.2 Specificity

In our study described above, we assume that the existing diagnostic test has a specificity of 88% and the new diagnostic test will achieve a specificity of 92%:

The output indicates we should expect 856.56 nondiseased participants in our sample. We compute n_T = 1071 and n ₁ = 214. (I leave it to the reader to perform the computations!)

5 Conclusion

In prospective diagnostic accuracy studies, the total estimated sample size must be adjusted by the prevalence of disease to account for the uncertainty in the number and mix of diseased and nondiseased participants likely to be recruited. In this Stata tip, I described in detail how to use Stata’s official power commands to assist in deriving the necessary estimates. While performing the computations manually is straightforward, some users may prefer the convenience of the community-contributed power diagnostic package (Linden 2022), which produces comparable results with those illustrated here.