Invited Commentary on “Investigating the Reliability and Factor Structure of Kalichman’s ‘Survey 2: Research Misconduct’ Questionnaire”

Holm and Hoffman (2017) present an analysis of the reliability and factor structure of Kalichman’s “Survey 2: research misconduct” questionnaire based on a secondary analysis of survey data among biomedical doctoral students in Scandinavia. Its content is relevant for investigators interested in assessing attitudes toward research misconduct, although the recommendation to use the evaluated instrument seems a bit premature as this is only a partial “validation.” Other important measurement properties, such as content and construct validity, and the interpretability of scores were not studied (Mokkink et al., 2010). In addition, the responsiveness of the instrument is not yet known, which would be especially informative given the methodological limitations of studies that have evaluated the impact of interventions to prevent research misconduct so far (Marusic, Wager, Utrobicic, Rothstein, & Sambunjak, 2016). Another limitation is that the instrument only concerns research misconduct in the strict sense (fabrication, falsification, and plagiarism) and ignores the probably much more important—due to their high frequency of occurrence—questionable research practices or sloppy science (Bouter, Tijdink, Axelsen, Martinson, & ter Riet, 2016).

The authors used the 14-item instrument prior to teaching a research methods course. They report an explanatory factor analysis of data from Oslo doctoral students which indicated a four-factor solution. The confirmatory factor analysis among the non-Oslo participants yielded similar results. One item was removed which improved the internal consistency of the instrument. As an afterthought, the authors performed a confirmatory factor analysis for the original five factors proposed by Kalichman, which yielded a somewhat better percentage of explained variance. But one of these subscales had a relatively low Cronbach’s alpha, which presumably led to the decision to stick to the four-factor solution. The fact that the data set was not randomly split into two subsets is somewhat puzzling. Furthermore, using the Cronbach’s alphas from the combined Oslo and non-Oslo data sets to draw conclusions on the reliability of the four subscales and the overall scale is a concern. Reliability should be based solely on the non-Oslo data that were used for the confirmatory factor analysis.

It is a pity and a missed opportunity that the authors did not follow the more logical sequence: (a) do a confirmatory factor analysis of the full data set with the five-factor structure proposed by Kalichman, (b) draw the conclusion that the results are not good enough and explain why, (c) perform an exploratory factor analysis on the random half of their data, and (d) do a confirmatory factor analysis in the other half. Presentation of the results and formatting the tables in that logical sequence would have been more informative to readers, of course explaining that most of this was designed after the data were collected.

Taken together, this article offers interesting data presented in a slightly suboptimal form. But when the data set is made publicly available, others can make use of it and improve the instrument further or use the evidence to design a better alternative.