Psychological measurement is highly questionable but the details remain controversial: A response to Tafreshi,Michell,and Trendler

Abstract

The profound and stimulating comments by Michell (2022), Tafreshi (2022), and Trendler (2022) push the measurement debate forward. Tafreshi provides arguments that are in line with my own position. While Michell is right in claiming that conceptualizations of psychological measurement are not necessarily Cartesian, I argue that the continuing influence of Cartesian thinking on conceptualizations of psychological measurement is evident in current definitions of psychological constructs. Moreover, I show that, contrary to Michell, it is meaningless to ask whether intelligence is quantitative. Michell and Trendler could counter this claim, but only at the cost of contradicting their stance that conceptual questions are not primary to the empirical questions surrounding psychological measurement. Beyond that, I contend that Trendler’s epistemological distinction of extensive versus intensive quantities does not compensate for the conceptual confusions inherent in his and Michell’s examples. Finally, I emphasize that all disputants agree that psychological measurement is a highly questionable idea.

It is an honor that three distinguished experts in the measurement debate have provided commentaries on my article. I thank Donna Tafreshi (2022), Joel Michell (2022), and Günter Trendler (2022) for their thought-provoking impulses.

Tafreshi’s (2022) article is a valuable contribution to the growing literature on the conceptually confused foundations of quantitative psychology. I agree with her that the “primary takeaway” of my article “is that the question, ‘are psychological attributes quantitative?’ is nonsensical when referring to nontechnical psychological phenomena” (p. 159). Beyond that, Tafreshi provides several interesting arguments, which lead her to conclude that “asking whether psychological attributes are quantitative is, in many cases, a nonsensical question to ask” (p. 158) Since Tafreshi and I seem to be largely in agreement, I will not go into the details of her argumentation.

It is impossible to do justice to all the thoughtful counterarguments provided by Michell (2022) and Trendler (2022). For example, both Michell (2022, p. 154) and Trendler (2022, p. 167) emphasize the possibility that our daily psychological concepts might change one day in light of scientific evidence, which is a complex issue that has already been discussed at length by many other authors (e.g., Baker & Hacker, 1982; Bennett & Hacker, 2003; Maraun, 1998a, 1998b; Tafreshi, 2022). What I can try to do in the following, however, is to aim once again at more conceptual clarity in the measurement debate.

Psychological measurement is still haunted by Descartes’ ghost

With reference to the historical development of psychological measurement, Michell (2022) claims that “psychometrics is not inherently Cartesian” (p. 51). Since Michell’s historical expertise certainly exceeds my own, I am not going to disagree with him. I admit that my initial claim that psychological measurement is inherently Cartesian was overblown.

Nevertheless, Cartesian presumptions still have a considerable influence on conceptualizations of psychological measurement, because a Cartesian understanding of psychological concepts makes measurement seem a necessary step toward unbiased psychological evidence. If psychological concepts denote distinct “inner” phenomena that are “hidden inside” people, and if we can only gain knowledge about them indirectly via complex inferences, then measurement might help us to make these inferences more reliable. This close link between a Cartesian view on psychological phenomena and the attempt for measurement in psychology is evident, for instance, in the fact that constructs—the “things” that, according to widespread terminology, are usually measured in psychology—are oftentimes characterized as “the real but unobservable objects of study in psychological research” (Slaney & Racine, 2013, p. 7). Take the following textbook example: “The construct is latent, that is, it can never be known directly, it can only be inferred from its associated measurement operations” (Barker et al., 2016, p. 52). If constructs are latent in a Cartesian sense because they belong to an inaccessible “inner” realm and if, therefore, there can only be indirect knowledge about them, it might appear reasonable to develop sophisticated measurement methods to investigate constructs.

Nevertheless, Michell’s (2022) reference to Spearman might suggest a non-Cartesian understanding of “latent construct,” according to which this term is defined either with reference to a biological phenomenon, such as the “level of the brain’s energy” (p. 152), or simply with a technical term, such as Spearman’s g (p. 152). While it might be justifiable for psychologists to opt for one of these two interpretations in some instances, understanding “psychological constructs” exclusively in this manner is not possible.

First, speaking of psychological measurement and simultaneously conceiving psychological phenomena exclusively as biological entities presupposes the philosophical thesis that the mental can be reduced to the biological. If every psychological phenomenon is conceived to be mere biological activity, psychology essentially becomes neuroscience. Then, it remains unclear, however, we why should still call the respective research “psychological” and, more importantly, why there is a need at all for a special psychological measurement besides the measurement of biological phenomena. Identifying all psychological constructs with biological phenomena makes a specific psychological measurement superfluous.

Second, defining psychological constructs exclusively with technical terms, such as Spearman’s g, only postpones the conceptual issues that surround psychological measurement. Research results about g are only informative if we can relate them at some point to psychological phenomena denoted by our common-or-garden concepts. What are the consequences, for instance, if someone scores high or low on g? This question can only be answered if we can relate g to phenomena denoted by common-or-garden concepts (e.g., life satisfaction, creativity, communicative skills). Without establishing such relations, the statement “Timmy has a g of 112” is completely uninformative. Consequently, at some point, the psychometrician cannot avoid defining constructs using common-or-garden concepts. For, in order to provide quantitative evidence for associations between g and, for example, life satisfaction, creativity, or communicative skills, the psychometrician needs to measure the latter constructs. Thereby, the psychometrician inevitably arrives at a nontechnical understanding of psychological construct, which, in conjunction with the aforementioned textbook characterization of constructs as real but unobservable research objects, brings us back to the Cartesian conception of psychological measurement outlined above.

It does not make sense to ask whether intelligence is quantitative

I understand some of Michell’s (2022) arguments as being in line with my own. For example, when he argues that some psychological phenomena might not be quantitative because of inherent qualitative differences (p. 153), I understand this, at least partly, as a conceptual argument against psychological measurement (e.g., describing pain as stabbing and burning is a qualitatively different description than describing pain as dull and throbbing). However, according to Michell, my argumentation only “promises results . . . when applied to attempts to measure currently occurring mental states, such as anxiety” (p. 153), which he sees in opposition to “merely dispositional concepts, including intellectual abilities, personality traits, and social attitudes” (p. 51).

With the example of Gilbert Ryle’s analysis of the intelligence concept, I will show that, contrary to Michell (2022), dispositional concepts are also not compatible with measurement. Ryle (1949/2000) remarked that “there are many activities which directly display qualities of the mind, yet are neither themselves intellectual operations nor yet effects of intellectual operations. Intelligent practice is not a step-child of theory” (p. 27). Ryle argued that the concept “intelligence” (and related concepts, such as cleverness, prudence, etc.) does not refer to a single attribute, mental faculty, or operation that causes intelligent behavior. Rather, intelligent “performance has a special procedure or manner, not special antecedents” (p. 32). According to Ryle, it is an important part of the meaning of “intelligence” that this concept denotes a special way of acting:

Cleverness at fighting is exhibited in the giving and parrying of blows, not in the acceptance or rejection of propositions about blows, just as ability in reasoning is exhibited in the construction of valid arguments and the detection of fallacies, not in the avowal of logicians’ formulae. (p. 48)

What we can learn from Ryle is that “intelligence” does not, in many instances, refer to a special reasoning process or intellectual capacity, but rather to a particular style of performing. Behaving in an intelligent manner means being intelligent and is not a clue to an intelligence faculty causing prudent behavior.

If this highly abridged Rylean analysis is correct, we have further reason to see why the question “Are psychological attributes quantitative?” is nonsensical. If the sentence “Timmy’s intelligence was evident in his presentation” refers to the manner in which Timmy delivered his speech, it does not make sense to ask whether Timmy’s intelligence is quantitative. “Timmy showed his cleverness by identifying the critical premise of argument X” is a meaningful sentence but the question “Was Timmy’s identification of the premise a quantitative phenomenon?” is not. It does not make sense to ask: Is the fact that Timmy reacted to criticism Z with the counterargument C a magnitude with equal differences between different levels in the same sense as temperature is a magnitude with equal differences between the levels of 10°C and 20°C and between the levels of 20°C and 30°C? However, it is perfectly meaningful to say that Timmy’s reaction was very intelligent. If “intelligence” denotes a special manner of doing certain things, then there is no meaning to the question whether intelligence is quantitative and, therefore, measurable.

Of course, this argumentation depends on the outlined analysis of the intelligence concept and, although I find Ryle’s position illuminating, I concede that it might have its limitations. Consequently, Michell (2022) and Trendler (2022) are free to protest by questioning the conceptual foundations of my argumentation. However, since Michell (2022) contends that “rules of language provide no basis for conceptual analysis” (p. 154) and Trendler (2022) concludes that I do “not succeed in demonstrating that conceptual quantity objections nullify empirical quantity objections” (p. 169), I read them both as denying the primacy of conceptual analyses in regard to the question of measurement.

What does it mean to claim that psychological phenomena are intensive quantities?

Trendler (2022) accuses me of “misunderstanding Hölder’s axioms of quantity” (p. 165) and argues that I erroneously treat psychological phenomena as extensive and not as nonextensive or intensive attributes. Trendler (2009) himself defined intensive quantities as follows: “With regard to intensive quantities (e.g., temperature, pressure, electrical tension, etc.), relations of additivity are not directly observable and therefore in principle not amenable to direct testing” (p. 583). While observing the quantitative nature of extensive quantities is relatively straightforward, we have no direct access to intensive quantities and, therefore, have to draw inferences about them based on complex experimental setups. Beyond that, Trendler states that “psychological attributes can be thought of only as intensive quantities” (p. 583). However, he provides no reasons for this claim.

As a first response, I want to ask Trendler what justifies the exclusive categorization of psychological phenomena as intensive quantities. Why does he think that we can access psychological phenomena only indirectly? From Ryle’s point of view, assessing other people’s intelligence is in many instances straightforward because the more or less intelligent ways in which people perform are obvious. I can literally observe that Timmy arranged his arguments on his slides in an intelligent order.

The second and more important point is, however, that the distinction of extensive and nonextensive quantities is epistemological since it describes our possibilities of gaining evidence about the quantity of certain phenomena. When Trendler (2022) states that my “neglect of the distinction: extensive versus nonextensive quantities significantly weakens” (p. 165) my conceptual argumentation, he confuses epistemological and conceptual questions. It is an epistemological question whether we can identify an attribute to be quantitative through direct observation or only indirectly with the help of sophisticated methods. In contrast, whether a phenomenon can be conceptualized as quantitative in the first place is a conceptual question. If it is not possible to conceptualize a phenomenon as quantitative, then it is pointless to ponder the epistemological question whether we have to deal with an extensive or an intensive quantity. Consequently, Trendler’s emphasis on the epistemological distinction of extensive and intensive quantities misses the central conceptual point of my article (Franz, 2022).

Third, as was also argued by Tafreshi (2022), trying to conceptualize psychological phenomena as quantitative is in many instances pointless. Whereas Trendler (2022) holds that there are “legitimate interpretations of additivity when applied to psychological (or nonextensive) attributes in many places in Joel Michell’s extensive work (see, e.g., Michell, 1999, pp. 198–204)” (p. 165), I contend that this is doubtful. For instance, in the section that Trendler refers to, Michell (1999) illustrates conjoint measurement using the following example:

Suppose persons K and L perform at exactly the same level on the test despite the fact that they differ from one another in motivation and ability. K does as well as L because of a higher level of motivation and L compensates for K’s higher level of motivation by possessing a higher level of ability. That is, K’s level of motivation (M_K) minus L’s level of motivation (M_L) equals L’s level of ability (A_L) minus K’s level of ability (A_K) in terms of effects upon performance. Putting it in quantitative terms, relative to performance, a difference in motivation equals a difference in ability, i.e., M_K − M_L = A_L − A_K. (p. 201)

This example is conceptually confused. While it does make sense to say that a person compensates for their lack of skill by being highly motivated, it does not make sense to state that a difference of two motivations is equal to a difference of two abilities. As I elaborated extensively in my original article (Franz, 2022), the rule-governed use of psychological concepts does not contain the conceptual possibility of subtracting psychological phenomena and comparing the results.

The example from Trendler’s response (2022) is also conceptually problematic. Trendler is right that “test performance depends on the ability of test participants and the difficulty of the task” (p. 166) is a meaningful statement. However, this claim is only a rather vague description of his precise “quantitative hypothesis R = D/A, where R is the time required to solve a task of difficulty D and A is ability” (p. 166). Beyond that, this hypothesis entails a technical definition of ability A = D/R. By assuming that ability can be defined as a ratio of difficulty and time, Trendler simply presupposes that ability can be conceptualized as measurable. Moreover, Trendler’s definition of ability leads into conceptual confusions. “Timmy’s reasoning ability was evident in the fact that he identified the central premise” is a meaningful sentence, whereas “Timmy’s identification of the central premise is a ratio of the difficulty of the argument and the time Timmy needed” is not.

Conclusion

In closing, I want to draw attention to the fact that Michell (2022), Tafreshi (2022), Trendler (2022), and I agree about the most important point: the doubtfulness of established measurement practices. Although the details will and should be the topic of further debate, various different theoretical perspectives speak against the widespread assumption that psychologists are routinely measuring the mind.