Rater agreement at lung scintigraphy after consensus training

Abstract

Purpose: Using the PIOPED criteria in the interpretation of lung scintigraphy, perfect agreement between experienced raters with training in consensus interpretation has been demonstrated. In a previous paper we found a kappa value of 0.54 (0.45–0.63) between experienced raters with no experience of interpretation in consensus; this is significantly lower than the perfect agreement described between consensus-trained raters. The purpose of this study was to reinvestigate the same study population with the same raters, but this time after they had undergone consensus training.

Material and Methods: Lung scintigrams of 192 patients were reviewed and categorized according to the PIOPED criteria by 2 raters after consensus training. The 2 raters reviewed a training material of 59 cases, which was not part of the study population, in order to reach a full consensus interpretation.

Results: After consensus training the kappa value was 0.57 (0.48–0.66). Three cases had a discrepancy of 2 categories (1.6%), and the remaining discrepancies were of one category.

Conclusion: Consensus training did not significantly improve the agreement rate between experienced raters from different hospitals. A substantial improvement in agreement rates between hospitals may make heavy demands on resources.