Problematic research practices in psychology: Misconceptions about data collection entail serious fallacies in data analysis

Abstract

Given persistent problems (e.g., replicability), psychological research is increasingly scrutinised. Arocha (2021) critically analyses epistemological problems of positivism and the common population-level statistics, which follow Galtonian instead of Wundtian nomothetic methodologies and therefore cannot explore individual-level structures and processes. Like most critics, however, he focuses on only data analyses. But the challenges of psychological data generation are still hardly explored—especially the necessity to distinguish the study phenomena from the means to explore them (e.g., concepts, terms, methods). Widespread fallacies and insufficient consideration of the epistemological, theoretical, and methodological foundations of data generation—institutionalised in psychological jargon and the popular rating scale methods—entail serious problems in data analysis that are still largely overlooked, even in most proposals for improvements.

Epistemological problems of established research practices

Arocha (2021) rightfully highlights epistemological problems of positivism, especially empiricism, operationism, and neglected theory development. Contrary to empiricists’ beliefs, facts and observations cannot be pure elements of truth because it is scientists who decide what constitutes facts and what to observe in their field—and this presupposes theory (Weber, 1949). Operationist beliefs that only concrete operations could provide empirical meaning for concepts, although long refuted, still guide psychological practices, such as when constructs and models are derived from empirically associated item assessments (e.g., Big Five personality factors), which conflates theoretical with operational concept definition (e.g., “intelligence” with test performances; Uher, 2018a, 2020). Positivism led psychologists to plunge into practical (empirical) activities (e.g., statistism; Lamiell, 2019), ignoring the necessity to develop their philosophical and theoretical foundations as well—including those underlying approaches and methods for data generation and analysis. This can entail mismatches between methods and research questions and hamper explorations of the actual phenomena of interest (Toomela & Valsiner, 2010; Uher, 2016a, 2019).

Furthermore, Arocha (2021) points out limitations of deterministic and reductionist approaches. Common psychological models and analytical designs implicitly build on the image of humans not as living beings but, instead, as deterministic machines featuring immutable components with simple (often linear) interrelations (e.g., with stimuli). Accordingly, complex psychical phenomena are reduced to manageable chunks, entified as “constructs,” and their workings explored by simply “manipulating” input “variables” and observing individuals’ changes in response (Valsiner, 2012). In psychology, however, clear directions and simplistic causal relations between “variables” (e.g., IV→DV) exist only in statistical models. Decisions about which “variables” to analyse as independent, dependent, moderating, and so forth are often arbitrary—as is the selection of theories for justifying these decisions (given many opposing theories). As living beings, individuals function as integrated wholes on different levels of organisation in which dynamic nonlinear and recursive processes occur that feed back to and change the components and processes from which they emerge (retro-actions; Morin, 2008). These complex patterns of upward and downward causation, resulting in irreversible changes and processes of development, are incompatible with the reductionist and deterministic assumptions underlying common psychological models and approaches (Trofimova et al., 2018; Uher, 2018b).

Exploring these complex processes requires analyses of variability. But in many psychologists’ views, variability derives from uncontrolled random factors, masking “true” relations, and must therefore be cancelled out, such as by using inferential statistics on aggregations of individual cases. This Galtonian nomothetic methodology is suited to explore population-level phenomena (e.g., in epidemiology) but not individual-level processes and structures because it restricts analyses to sample-level testing and theory development to inductive generalisation across individuals. It also entails fallacies, such as the widespread assumption that between-individual and within-individual structures could be isomorphic, which ignores nonergodicity (Chirkov & Anderson, 2018; Molenaar & Campbell, 2009; Uher, 2015b). Instead, individual variability must be analysed, for which Arocha (2021) presents approaches from perceptual control theory and observation-oriented modelling—yet without specifying their relations to the Wundtian nomothetic methodology, which is needed to develop generalised knowledge about psychical processes and functioning from case-by-case based analyses (Salvatore & Valsiner, 2010; Uher, 2021b).

Psychological jargon often blurs the vital distinction between the study phenomena and the means used for their exploration

A key challenge for psychologists is to distinguish the phenomena under study from the means used to explore them (e.g., concepts, methods, data), as reflected in the terms psychical versus psychological ¹ in many non-English languages (similarly, we get viral and not virological infections but we do virological research). This distinction is intricate because the means by which science is made (e.g., concepts, abstractions) are psychical (mental) phenomena in themselves and many psychical phenomena are accessible only through language. Language, as all semiotic systems, inherently involves psychical phenomena (e.g., meaning) in itself and is therefore inseparable from its users’ minds (Valsiner, 2012). That is, the means of psychological investigation do not exist outside of the empirical phenomena under study (Uher, 2021a).

Common psychological jargon often blurs this vital distinction—prominently reflected in the undifferentiated use of the term “psychological,” widespread in English-language psychology. Similarly, as in Arocha’s (2021) article, the term “variable” often denotes both the study phenomena in themselves (e.g., psychical phenomena) and the sign systems (e.g., statistical variables) used to encode information about them for the purpose of analysis (variable-referent conflation). Many psychological key terms have such disparate meanings. Their frequent conflation entails numerous fallacies, such as when concepts describing the study phenomena are reified as real entities and erroneously equated with the phenomena underlying those described (e.g., “traits”), thus confusing description with explanation. Therefore, by using common jargon, even critics implicitly build on misconceptions that they rightfully criticise and that will persist unless psychologists establish a more elaborated research terminology (Uher, 2021b).

Data “collection”: A misleading term masking the still underdeveloped theoretical foundation of a key scientific activity

Distinguishing the study phenomena from the means of exploration is fundamental for data generation. The common notion of “collecting” data implies an empiricist ease, unburdened by the necessity of prior theorising. But data are not out there ready to be collected like mushrooms in the forest (Uher, 2019). Even for the latter, one needs to know what mushrooms actually are, which ones are edible, and how they look. Lacking well-developed philosophical-theoretical foundations, psychologists still struggle to define even their basic phenomena (Uher, 2021a; Zagaria et al., 2020). Why does Arocha’s (2021) definition of “behaviour” involve only sensations and stimulus perceptions, ignoring other experiences (e.g., conceiving, reasoning) that are largely accessible only through language? Are external changes and activities (e.g., movements) not behaviours (Uher, 2016b)? Where is the mind in Arocha’s reality if there are no general or abstract things and properties?

Although data are elementary to empirical research, many psychologists are not very familiar with the theoretical foundations of sign systems like language and data, especially with their inherently symbolic and composite nature (Uher, 2015a). The ease of using language, entrenched in our everyday thinking, lets us often overlook that the written symbols and verbal utterances we use for objects typically bear no inherent relations to the objects themselves (e.g., no resemblance). ² These relations are established only through conventions; what is written down or spoken (signifier) can signify information about the study phenomena (referent) only through the meanings (signified) that the sign-using persons attribute to both. From the tight interrelations between signifier, referent and meaning established through conventions, emerges the functionality of this tripartite ensemble as a sign.

The inherently symbolic and composite nature of sign systems entails that any data-generation system requires specification of (a) the empirical phenomena and properties under study (referent), (b) the symbolic system devised to represent information about them for the purposes of analysis (signifier), and (c) systematic assignment relations between them (meaning). This applies in particular to measurement, which requires establishment of documented, unbroken measurand–result connections that allow tracing the results and their generation back to the properties and phenomena studied (Uher, 2018a, 2020). But positivist beliefs and widespread misconceptions about sign systems lead many psychologists to ignore the inherently representational function of data, thereby conflating the study phenomena with the means used for their exploration (Uher, 2021b, 2021c).

Widespread fallacies and misconceptions implemented in rating methods

With the establishment of rating scales as one of psychology’s standard methods of data generation, the conceptual problems above were firmly anchored in psychological practices. Rating scales build on operationist concepts and Galtonian nomothetic methodology (sample-level associations of item assessments used to define constructs); deterministic input–output models (respondents react to item stimuli, ignoring respondents’ context-dependent construction of meaning for these stimuli); and neglect of variability (ratings require overall, thus abstracted and generalised judgements). Masked by the ease of use of language, rating scales invoke common misconceptions about sign systems. They lead psychologists to mistake verbal descriptions for the described phenomena in themselves, and therefore to use rating scales both as description of the empirical study system (referent) and as symbolic system (signifier), overlooking the necessity to first define each system and to specify the assignment relations between them (meaning). This violates elementary principles of data generation (e.g., traceability from symbolic to empirical study system), rendering inferences from data analyses to the actual study phenomena invalid (Uher, 2018a, 2020). The widespread use of rating scales for construct operationalisation also often leads psychologists to reify their own linguistic and statistical abstractions (e.g., personality factors) into real entities internal to the individuals studied.

Conclusion

Many problematic research practices in psychology are rooted in the failed distinction between the study phenomena and the means of their exploration (e.g., concepts, methods, data). Its institutionalisation in common jargon and in the popular rating scale methods highlights that improvements of only data analyses are insufficient for overcoming these problems. Psychologists must scrutinise the entirety of their research practices and must develop their philosophical–theoretical foundations in holistic ways.