Comparing means and variances of two simulations

Abstract

Two systems are simulated using the same random- number stream. To compare their means one can use standard statistical techniques, e.g., the t-test for paired comparisons. To compare their variances, however, the standard F-test fails. Three approaches are suggested. Approach 1 uses the traditional F- statistic and is conservative. Approach 2 splits the observations into two groups. For each group the F- statistic is exact. It is shown how to combine the two F-statistics in a very simple way. This approach is also conservative. Approach 3 was developed in 1946 by Wilks. It is slightly more complicated but exact.