Back to the future: Topic modelling and beyond

I appreciate Bednarek’s research note about topic modelling (this issue) for at least two reasons. First, it effectively documents what is crucial for our discipline and others: the critical methodological engagements that foster collaboration, interdisciplinarity, and, most importantly, dialogue across disciplines, which can potentially lead to new insights, solutions and serendipities. My own experience using topic modelling (TM) stemmed from interdisciplinary collaborative research with a colleague from economics and business management, who had amassed a large collection of corporate social responsibility (CSR) reports. Although aware of the significance of texts and words, he did not know what to do with them and how an analysis might proceed – a scenario likely familiar to many linguists working with scholars from other fields. He sought to identify topics within each document and was advised by another economist to use TM as a practical tool for processing over 100 texts. Our collaboration (Jaworska and Nanda, 2018) involving a linguist sceptical of word lists alone and subsequent corroboration of identified ‘topics’ with concordance and collocation analysis, led to more refined results that made both parties happy. More importantly, this dialogue opened doors to further collaborations that extended beyond economics and business management into other social sciences and diverse research themes. In my own experience, taking (corpus) linguistics and its methods beyond our discipline – whether using TM or not – has led to novel contributions, often to the surprise of other social scientists. (Corpus) linguistics is, after all, a social science, though it is not always recognised as such. Engaging in critical methodological dialogues is important not only for raising awareness of the value of (corpus) linguistics, but more importantly, it contributes to mergers of perspectives and purposes, creating new and critical insights. This is what the German philosopher Gadamer (1989) had in mind when in his seminal work, Wahrheit und Methode (Truth and Method), he introduced the concept of Horizontenverschmelzung (‘fusion of horizons’), which he saw as essential for advancing human knowledge. This ‘fusion’ is not about reaching complete agreement (that would be the end of knowledge I suppose) but needs to involve dialogue, openness, and negotiation between different perspectives. Knowledge is not a static, linear package but a dynamic, communal process in which our preconceived notions and practices are continuously challenged and reshaped.

Second, reading the research note prompted me to consider two other issues: the clarity of how we define a topic and the sometimes challenged and critiqued notion of subjectivity in topic identification. It is striking that, although linguistic research often focuses on topics, rarely do we define what a topic actually is. Commonly, topics are assumed to be ‘whatever is being talked about’ (Brown and Yule, 1983: 62), presuming that topics are inherent in texts and can be directly read off. Both keywords analysis (a technique more commonly used in Corpus Linguistics than topic modelling) and TM seem to suggest that topics can reside in recurrent words. This approach emphasises the material dimension of texts, where frequent words and topics can be identified empirically through frequency and probability-based methods. However, texts – and thus topics – also have a reader-related dimension, which is interpretive, hermeneutic, and historical (Gadamer, 1989). Scott’s (2006) notion of ‘aboutness’ and his lesser-known concept of a ‘cline of aboutness,’ ranging from no aboutness (functional words) to great aboutness (text summary), touch upon these two dimensions. He emphasises that the cline of aboutness is fundamentally a cline of importance. But what is deemed important depends; it depends on the historical, social and cultural context and our own historicity and biases. In recent research, Mathew Gillings and I evaluated topic labelling (and by extension topic identification) using different methods, including TM, large language models (LLMs) like ChatGPT, concordance analysis, and close reading (Gillings and Jaworska, 2024). The data under study included a corpus of 10 CSR reports (just under 100k words). Human readers, who were tasked with reading the entire corpus, identified topics such as Diversity and Inclusion that other methods did not, likely because the frequencies of the relevant words were lower. This highlights that frequency does not necessarily equate to topical importance. Also, there was a degree of inconsistency amongst our human readers, which is not surprising and not necessarily a ‘bad’ thing, since novel, creative and critical insights do not always come from a clean and clear path but rather, they emerge from the ‘messiness’ of different perspectives (Gadamer, 1989). I am not suggesting that we should use just human judgement as a ‘gold’ standard; we would lose some important perspectives related to the material dimension of texts, which is of interest to linguists. In the words of Gadamer (1989: 295), the point is ‘not to develop a procedure of understanding, but to clarify the conditions in which understanding takes place’.

This brings me to a final remark, which I feel is in place: the use of LLMs, which can produce topical summaries of uploaded documents, though the capacity is limited to smaller size corpora and only available to those who pay for it. Compared to LLMs, TM may seem a little outdated, quite ‘mechanical’ but certainly much more transparent than the black box of LLMs, whose inner workings remain largely unknown even to their developers. As the tools based on LLMs are widely rolled out, it is crucial now more than ever to uphold academic principles of transparency and reflexivity and critically assess how these tools operate and what they can do for us. It is also vital not to lose sight of the critical contributions that our ‘glass box’ linguistics, with its well-developed theoretical and analytical principles, can make to our understanding of meanings and relationships in texts and society. The interdisciplinary engagements with TM and subsequent discussions such as those outlined in the research note and its associated commentaries in this issue set a valuable precedent in this regard.