The IDEAL-CT (InDirect tEnsile AsphaLt Cracking Test) has rapidly become a popular cracking test for asphalt mixture acceptance. There have been several studies conducted to ensure this test is sufficient and acceptable for mixture acceptance purposes with respect to testing variability and the test’s ability to detect mixture quality improvements. A potential source of testing variability yet to be comprehensively studied is the variability introduced between testing devices. This paper provides a case study for the potential CTIndex variability, which can be attributed to the test device and the impact this has on determining whether devices are equivalent. The two one-sided tests (TOST) equivalence test, applied to this study, is a more appropriate statistical test to determine device equivalence when compared with typical hypothesis tests like analysis of variance. The TOST requires the user to have an understanding of an acceptable limit of deviation between two devices to determine both the statistical and practical equivalence between devices. This paper details the equivalence testing of 100 unique device-to-device comparisons and provides an example of the importance of using an appropriate number of samples to set achievable limits for accepting device equivalency. A relatively simple approach to assessing equivalence is recommended to IDEAL-CT users in situations in which device-to-device comparisons have real-world consequences. Finally, guidance is provided for selecting a minimum number of samples required to adequately test for device-to-device equivalence.