Leveraging Artificial Intelligence to Linguistically Compare Test Translations: A Methodological Introduction and Demonstration

Abstract

The validity and utility of translated instruments (psychological measures) depend on the quality of their translation, and differences in key linguistic characteristics could introduce bias. Likewise, linguistic differences between instruments designed to measure analogous constructs might contribute to similar instruments possessing dissimilar psychometrics. This article introduces and demonstrates the use of natural language processing (NLP), a subfield of artificial intelligence, to linguistically analyze 13 translations of two psychological measures previously translated into numerous languages. NLP was used to generate estimates reflecting specific linguistic characteristics of test items (emotional tone/intensity, sentiment, valence, arousal, and dominance), which were then compared across translations at both the test- and item-level, as well as between the two instruments. Results revealed that key linguistic characteristics can profoundly vary both within and between tests. Following a discussion of results, the current limitations of this approach are summarized and strategies for advancing this methodology are proposed.

Keywords

artificial intelligence psychometrics natural language processing linguistics test translation

Get full access to this article

View all access options for this article.

References

Argüello-González

Aquino-Esperanza

Salvador

Bretón-Romero

Del Río-Bermudez

Tello

Menke

(2023). Negation recognition in clinical natural language processing using a combination of the NegEx algorithm and a convolutional neural network. BMC Medical Informatics and Decision Making, 23, 216.

Bach

Brown

T. A.

Mulder

R. T.

Newton-Howes

Simonsen

Sellbom

(2021). Development and initial evaluation of the ICD-11 personality disorder severity scale: PDS-ICD-11. Personality and Mental Health, 15(3), 223–236.

Bargh

J. A.

Chartrand

T. L.

(2000). The mind in the middle: A practical guide to priming and automaticity research. In Reis

H. T.

Judd

C. M.

(Eds.), Handbook of research methods in social and personality psychology (pp. 253–285). Cambridge University Press.

Bartz

J. A.

Lydon

J. E.

(2004). Close relationships and the working self-concept: Implicit and explicit effects of priming attachment on agency and communion. Personality and Social Psychology Bulletin, 2004, 1389–1401.

Combaluzier

Gouvernet

Auvage

Bourgoise

Murphy

(2023). Le modèle alternatif des troubles de la personnalité en population française : évaluation avec des outils brefs. [The alternative model of personality disorders among the French population: Assessment with brief tools]. L’Encephale, 49(5), 496–503.

Cottin

Muzard

Hernández

Mundt

Behn

. (in preparation). Validation of the Spanish translation of the LPFS-BF 2.0 in a Chilean community and clinical sample.

Crystal

(1981). Clinical linguistics. Disorders of human communication (Vol. 3). Springer.

Czarnek

Stillwell

(2022). Two is better than one: Using a single emotion lexicon can lead to unreliable conclusions. PLoS One, 17(10), e0275910.

Elgendi

Kumar

Barbic

Howard

Abbott

Cichocki

(2018). Subliminal Priming-State of the Art and Future Perspectives. Behavioral Sciences, 8(6), 54.

10.

Ghali

J. P. E.

Inuzuka

Shima

Moriyama

Mutoh

(2023). Automating lexicon generation: A comprehensive review of alternative approaches. Procedia Computer Science, 225, 1142–1150. https://doi.org/10.1016/j.procs.2023.10.102

11.

Gritti

E. S.

De Carli

Hutsebaut

Simonelli

Zimmermann

. (in preparation). Cross-cultural adaptation of the Personality Functioning Scale- Brief Form 2.0 (LPFS-BF 2.0): Temporal stability and construct validity of the Italian version.

12.

Gutiérrez

Aluja

Rodríguez

Gárriz

Peri

J. M.

Gallart

Calvo

Ferrer

Gutiérrez-Zotes

Soler

Pascual

J. C.

(2023). Severity in the ICD-11 personality disorder model: Evaluation in a Spanish mixed sample. Frontiers in Psychiatry, 13, 1015489.

13.

Jockers

M. L.

(2015). Syuzhet: Extract sentiment and plot arcs from text. https://github.com/mjockers/syuzhet

14.

Kam

C. C. S.

(2018). Why do we still have an impoverished understanding of the item wording effect? An empirical examination. Sociological Methods & Research, 47(3), 574–597.

15.

Koo

T. K.

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.

16.

Krippendorff

(2019). Content analysis. Sage. https://doi.org/10.4135/9781071878781

17.

Le Corff

Aluja

Rossi

Lapalme

Forget

García

L. F.

Rolland

J. P

. (2022). Construct Validity of the Dutch, English, French, and Spanish LPFS-BF 2.0: Measurement Invariance Across Language and Gender and Criterion Validity. Journal of personality disorders, 36(6), 662–679. https://doi.org/10.1521/pedi.2022.36.6.662

18.

Lee

K. M.

Lee

Chung

C. Y.

Ahn

Sung

K. H.

Kim

T. W.

Lee

H. J.

Park

M. S.

(2012). Pitfalls and important issues in testing reliability using intraclass correlation coefficients in orthopaedic research. Clinics in Orthopedic Surgery, 4(2), 149–155.

19.

Mahany

Khaled

Elmitwally

Aljohani

Ghoniemy

(2022). Negation and Speculation in NLP: A Survey, Corpora, Methods, and Applications. Applied Sciences, 12(10), 5209. https://doi.org/10.3390/app12105209

20.

Meyer

D. E.

Schvaneveldt

R. W.

(1971). Facilitation in recognizing pairs of words: evidence of a dependence between retrieval operations. Journal of Experimental Psychology, 90, 227.

21.

Mohammad

S. M.

(2018a). Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.

22.

Mohammad

S. M.

(2018b). Word Affect Intensities. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan.

23.

Mohammad

S. M.

(2020). Practical and ethical considerations in the effective use of emotion and sentiment lexicons. arXiv preprint. https://doi.org/10.48550/arXiv.2011.03492.

24.

Mohammad

S. M.

Turney

P. D.

(2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3), 436–465.

25.

Molden

D. C.

(2014). Understanding priming effects in social psychology: An overview and integration. Social Cognition, 32(Supplement), 243–249.

26.

Morante

Blanco

(2021). Recent advances in processing negation. Natural Language Engineering, 27(2), 121–130. https://doi.org/10.1017/S1351324920000534

27.

Mukherjee

Badr

Doppalapudi

Srinivasan

S. M.

Sangwan

R. S.

Sharma

(2021). Effect of negation in sentences on sentiment analysis and polarity detection. Procedia Computer Science, 185, 370–379. https://doi.org/10.1016/j.procs.2021.05.038

28.

Natoli

A. P.

Bach

Behn

Cottin Arredondo

Gritti

E. S.

Hutsebaut

Lamba

Le Corff

Zimmermann

Lapalme

(2022). Multi-national evaluation of the measurement invariance of the Level of Personality Functioning Scale—Brief Form 2.0: Comparison of student and community samples across seven countries. Psychological Assessment, 34(12), 1112–1125. https://doi.org/10.1037/pas0001176

29.

Natoli

A. P.

Rodriguez

C. M.

(2024). A new performance-based measure of personality functioning impairment: Development and preliminary evaluation of reliability and validity. Discover Mental Health, 4(1), 6. https://doi.org/10.1007/s44192-024-00059-4

30.

Oitsalu

M. L.

Kreegipuu

Hutsebaut

(2022). Psychometric evaluation of the Estonian version of the semi-structured interview for personality functioning DSM-5 (STiP-5.1). Borderline Personality Disorder and Emotion Dysregulation, 9(1), 28.

31.

Podsakoff

P. M.

MacKenzie

S. B.

Lee

J. Y.

Podsakoff

N. P.

(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879

32.

Portney

L.G.

Watkins

M.P.

(2000). Foundations of clinical research: Applications to practice (2nd ed.). Prentice Hall Health.

33.

Proutskova

Gruszczynski

(2007). Kripp.boot: An R Package for Performing Bootstrap Replicates of Krippendorff’s alpha on Intercoder Reliability Data. R package version 1.0.0. https://github.com/MikeGruz/kripp.boot

34.

Regmi

Naidoo

Pilkington

(2010). Understanding the Processes of Translation and Transliteration in Qualitative Research. International Journal of Qualitative Methods, 9, 16.

35.

Revelle

(2024). psych: Procedures for psychological, psychometric, and personality research. R package version 2.4.6, https://CRAN.R-project.org/package=psych

36.

Schwarz

(1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105. https://doi.org/10.1037/0003-066X.54.2.93

37.

Sorrel

M. A.

García

L. F.

Aluja

Rolland

J. P.

Rossier

Roskam

Abad

F. J.

(2021). Cross-cultural measurement invariance in the Personality Inventory for DSM-5. Psychiatry Research, 304, 114134. https://doi.org/10.1016/j.psychres.2021.114134

38.

Spitzer

Müller

Kerber

Hutsebaut

Brähler

Zimmermann

(2021). Die deutsche Version der Level of Personality Functioning Scale-Brief Form 2.0 (LPFS-BF): Faktorenstruktur, konvergente Validität und Normwerte in der Allgemeinbevölkerung. [The German version of the Level of Personality Functioning Scale-Brief Form 2.0 (LPFS-BF): Latent structure, convergent validity and norm values in the general population]. Psychotherapie, Psychosomatik, Medizinische Psychologie, 71(7), 284–293.

39.

Stapel

D. A.

Blanton

(2004). From seeing to being: Subliminal social comparisons affect implicit and explicit self-evaluations. Journal of Personality and Social Psychology, 87, 468.

40.

Stapel

D. A.

Koomen

(2000). Distinctness of others, mutability of selves: Their impact on self-evaluations. Journal of Personality and Social Psychology, 79, 1068–1087.

41.

Stapel

D. A.

Koomen

(2001). I, we, and the effects of others on me: How self-construal level moderates social comparison effects. Journal of Personality and Social Psychology, 80, 766–781.

42.

Weekers

L. C.

Hutsebaut

Kamphuis

J. H.

(2019). The level of personality functioning scale-brief form 2.0: Update of a brief instrument for assessing level of personality functioning. Personality and Mental Health, 13(1), 3–14. https://doi.org/10.1002/pmh.1434

43.

Weekers

L. C.

Sellbom

Hutsebaut

Simonsen

Bach

(2023). Normative data for the LPFS-BF 2.0 derived from the Danish general population and relationship with psychosocial impairment. Personality and Mental Health, 17(2), 157–164.

44.

Zimmermann

Müller

Bach

Hutsebaut

Hummelen

Fischer

(2020). A common metric for self-reported severity of personality disorder. Psychopathology, 53(3), 161–171. https://doi.org/10.1159/000507377

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.24 MB