The use of observational tools in psychological assessment has decreased in recent years, mainly due to its personnel and time costs, and researchers have not explored methodological innovations like adaptive algorithms in observational assessment. In the present study, we introduce the behavior-driven observation procedure to develop, test, and implement observational adaptive instruments. In Study 1, we use a preexisting observational checklist to evaluate nonverbal behaviors related to psychotic symptoms and to specify the adaptive algorithm’s model. We fit the model to observational data collected from 114 participants. The results support the model’s goodness of fit. In Study 2, we use the estimated model parameters to calibrate the adaptive procedure and test the algorithm for accuracy and efficiency in adaptively reconstructing 58 nonadaptively collected response patterns. The results show the algorithm’s good accuracy and efficiency, with a 40% average reduction in the number of administered items. In Study 3, we used real raters to test the adaptive checklist built with behavior-driven observation. The results indicate adequate intrarater agreement and good consistency of the observed response patterns. In conclusion, the results support the possibility of using behavior-driven observation to create accurate and affordable (in terms of resources) observational assessment tools.
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders–Fourth edition–Text revision. Author.
3.
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders–Fifth edition. Author.
4.
BatesD.MaechlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://doi.org/10.18637/jss.v067.i01
5.
BrüneM.SonntagC.Abdel-HamidM.LehmkämperC.JuckelG.TroisiA. (2008). Nonverbal behavior during standardized interviews in patients with schizophrenia spectrum disorders. Journal of Nervous and Mental Disease, 196(4), 282-288. https://doi.org/10.1097/NMD.0b013e31816a4922
6.
CantorN.MischelW. (1979). Prototypes in person perception. In Advances in experimental social psychology (Vol. 12, pp. 3-52). Elsevier. https://doi.org/10.1016/S0065-2601(08)60258-0
7.
ChangW.ChengJ.AllaireJ.XieY.McPhersonJ. (2018). shiny: Web application framework for R (R package version 1.1.0) [Computer software]. https://CRAN.R-project.org/package=shiny
8.
DempsterA. P.LairdN. M.RubinD. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 1-38.
DonadelloI.SpotoA.SamboF.BadaloniS.GranziolU.VidottoG. (2017). ATS-PD: An Adaptive Testing System for Psychological Disorders. Educational and Psychological Measurement, 77(5), 792-815. https://doi.org/10.1177/0013164416652188
11.
DowiaschS.BackaschB.EinhäuserW.LeubeD.KircherT.BremmerF. (2016). Eye movements of patients with schizophrenia in a natural environment. European Archives of Psychiatry and Clinical Neuroscience, 266(1), 43-54. https://doi.org/10.1007/s00406-014-0567-8
FalmagneJ.-C.DoignonJ.-P. (1988). A class of stochastic procedures for the assessment of knowledge. British Journal of Mathematical and Statistical Psychology, 41(1), 1-23. https://doi.org/10.1111/j.2044-8317.1988.tb00884.x
FliegeH.BeckerJ.WalterO.BjornerJ.KlappB.RoseM. (2005). Development of a computer-adaptive test for depression (D-CAT). Quality of Life Research, 14(10), 2277-2291. https://doi.org/10.1007/s11136-005-6651-9
GamerM.LemonJ.FellowsI. (2012). irr: Various coefficients of interrater reliability and agreement (R package version 0.84) [Computer software]. https://CRAN.R-project.org/package=irr
GeertsE.BrüneM. (2009). Ethological approaches to psychiatric disorders: Focus on depression and schizophrenia. Australian & New Zealand Journal of Psychiatry, 43(11), 1007-1015. https://doi.org/10.1080/00048670903270498
23.
GibbonsR. D.WeissD. J.PilkonisP. A.FrankE.MooreT.KimJ. B.KupferD. J. (2012). Development of a computerized adaptive test for depression. Archives of General Psychiatry, 69(11), 1104-1112. https://doi.org/10.1001/archgenpsychiatry.2012.14
24.
GoodenoughF. L. (1928). Measuring behavior traits by means of repeated short samples. Journal of Juvenile Research, 12, 230-235.
GranziolU.BottesiG.SerraF.SpotoA.VidottoG. (2017). New perspectives on the assessment of the social anxiety disorder: The formal psychological assessment. Journal of Evidence-Based Psychotherapies, 17(2), 53-68. https://doi.org/10.24193/jebp.2017.2.3
27.
GranziolU.SpotoA.VidottoG. (2018). The assessment of nonverbal behavior in schizophrenia through the formal psychological assessment. International Journal of Methods in Psychiatric Research, 27(1), Article e1595. https://doi.org/10.1002/mpr.1595
28.
Groth-MarnatG. (2009). Handbook of psychological assessment. John Wiley.
29.
HaidetK. K.TateJ.Divirgilio-ThomasD.KolanowskiA.HappM. B. (2009). Methods to improve reliability of video-recorded behavioral data. Research in Nursing & Health, 32(4), 465-474. https://doi.org/10.1002/nur.20334
30.
HawesD. J.DaddsM. R.PasalichD. (2013). Observational coding strategies. In ComerJ. S.KendallP. C. (Eds.), The Oxford handbook of research strategies for clinical psychology (pp. 120-141). https://doi.org/10.1093/oxfordhb/9780199793549.013.0008
31.
HaynesS. N.O’BrienW. H. (2000). Principles and strategies of behavioral observation. In Principles and practice of behavioral assessment (pp. 225-263). Springer. https://doi.org/10.1007/978-0-306-47469-9_12
32.
KayS. R.FiszbeinA.OplerL. A. (1987). The Positive and Negative Syndromescale (PANSS) for schizophrenia. Schizophrenia Bulletin, 13(2), 261-276. https://doi.org/10.1093/schbul/13.2.261
33.
KølbækP.BlicherA. B.BuusC. W.FellerS. G.HolmT.DinesD.O’LearyK. M.SørensenR. S.OplerM.CorrellC. U.MorsO.BechP.ǾstergaardS. D. (2018). Inter-rater reliability of ratings on the six-item Positive and Negative Syndrome Scale (PANSS-6) obtained using the Simplified Negative and Positive Symptoms Interview (SNAPSI). Nordic Journal of Psychiatry, 72(6), 431-436. https://doi.org/10.1080/08039488.2018.1492014
34.
KooT. K.LiM. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. https://doi.org/10.1016/j.jcm.2016.02.012
35.
LeeD.BarakA.UhlemannM. (1999). Forming clinical impressions during the first five minutes of the counseling interview. Psychological Reports, 85(3), 835-844. https://doi.org/10.2466/pr0.1999.85.3.835
36.
LordC.LuysterR.GothamK.GuthrieW. (2012). Autism diagnostic observation schedule second edition (ados-2) manual (Part ii): Toddler module. Western Psychological Services.
37.
LordC.RutterM.DiLavoreP.RisiS.GothamK.BishopS. (2012). Autism diagnostic observation schedule second edition (ados-2) manual (Part 1): Modules 1-4. Western Psychological Services.
38.
MartinP.BatesonP. P. G.BatesonP. (1993). Recording methods. In Measuring behaviour: An introductory guide (pp. 84-100). Cambridge University Press.
39.
MichelP.BaumstarckK.LanconC.GhattasB.LoundouA.AuquierP.BoyerL. (2018). Modernizing quality of life assessment: Development of a multidimensional computerized adaptive questionnaire for patients with schizophrenia. Quality of Life Research, 27(4), 1041-1054. https://doi.org/10.1007/s11136-017-1553-1
40.
MummaG. (2002). Effects of three types of potentially biasing information on symptom severity judgments for major depressive episode. Journal of Clinical Psychology, 58(10), 1327-1345. https://doi.org/10.1002/jclp.10046
41.
PetersenM. A.GroenvoldM.AaronsonN.FayersP.SprangersM.BjornerJ. B., & the European Organisation for Research and Treatment of Cancer Quality of Life Group. (2006). Multidimensional computerized adaptive testing of the EORTC QLQ-C30: Basic developments and evaluations. Quality of Life Research, 15(3), 315-329. https://doi.org/10.1007/s11136-005-3214-z
42.
PinoM. C.SpotoA.MarianoM.GranziolU.PerettiS.MaseduF.ValentiM.MazzaM.VidottoG. (2018). Formal psychological assessment for autism spectrum disorder diagnosis: A new methodology to build an adaptive testing system. Open Psychology Journal, 11(1), 112-122. https://doi.org/10.2174/1874350101811010112
43.
PowellJ.MartindaleB.KulpS.MartindaleA.BaumanR. (1977). Taking a closer look: time sampling and measurement error. Journal of Applied Behavior Analysis, 10(2), 325-332. https://doi.org/10.1901/jaba.1977.10-325
44.
R Core Team. (2018). R: A language and environment for statistical computing [Computer software]. The R Foundation. https://www.R-project.org/
45.
RayJ. M.RayR. D. (2008). Train-to-code: An adaptive expert system for training systematic observation and coding skills. Behavior Research Methods, 40(3), 673-693. https://doi.org/10.3758/BRM.40.3.673
46.
ReiserM.VandenBergM. (1994). Validity of the chi-square test in dichotomous variable factor analysis when expected frequencies are small. BritishJournal of Mathematical and Statistical Psychology, 47(1), 85-107. https://doi.org/10.1111/j.2044-8317.1994.tb01026.x
47.
RhineR. J.LinvilleA. K. (1980). Properties of one-zero scores in observational studies of primate social behavior: The effect of assumptions on empirical analyses. Primates, 21(1), 111-122. https://doi.org/10.1007/BF02383828
48.
SerraF.SpotoA.GhisiM.VidottoG. (2015). Formal psychological assessment in evaluating depression: A new methodology to build exhaustive and irredundant adaptive questionnaires. PLOS ONE, 10(4), Article e0122131. https://doi.org/10.1371/journal.pone.0122131
49.
SerraF.SpotoA.GhisiM.VidottoG. (2017). Improving major depressive episode assessment: A new tool developed by formal psychological assessment. Frontiers in Psychology, 8, Article 214. https://doi.org/10.3389/fpsyg.2017.00214
50.
SmithP. K. (1985). The reliability and validity of one-zero sampling: Misconceived criticisms and unacknowledged assumptions. British Educational Research Journal, 11(3), 215-220. https://doi.org/10.1080/0141192850110302
51.
SpotoA.BottesiG.SanavioE.VidottoG. (2013). Theoretical foundations and clinical implications of formal psychological assessment. Psychotherapy and Psychosomatics, 82(3), 197-199. https://doi.org/10.1159/000345317
52.
SpotoA.SerraF.DonadelloI.GranziolU.VidottoG. (2018). New perspectives in the adaptive assessment of depression: The ATS-PD version of the QuEDS. Frontiers in Psychology, 9, Article 1101. https://doi.org/10.3389/fpsyg.2018.01101
53.
SpotoA.StefanuttiL.VidottoG. (2010). Knowledge space theory, formal concept analysis, and computerized psychological assessment. Behavior Research Methods, 42(1), 342-350. https://doi.org/10.3758/BRM.42.1.342
54.
SpotoA.StefanuttiL.VidottoG. (2012). On the unidentifiability of a certain class of skill multi map based probabilistic knowledge structures. Journal of Mathematical Psychology, 56(4), 248-255. https://doi.org/10.1016/j.jmp.2012.05.001
55.
SpotoA.StefanuttiL.VidottoG. (2013). Considerations about the identification of forward-and backward-graded knowledge structures. Journal of Mathematical Psychology, 57(5), 249-254. https://doi.org/10.1016/j.jmp.2013.09.002
56.
SpotoA.StefanuttiL.VidottoG. (2016). An iterative procedure for extracting skill maps from data. Behavior Research Methods, 48(2), 729-741. https://doi.org/10.3758/s13428-015-0609-9
57.
StefanuttiL.RobustoE. (2009). Recovering a probabilistic knowledge structure by constraining its parameter space. Psychometrika, 74(1), 83-96. https://doi.org/10.1007/s11336-008-9095-7
58.
StefanuttiL.SpotoA.VidottoG. (2018). Detecting and explaining BLIM’s unidentifiability: Forward and backward parameter transformation groups. Journal of Mathematical Psychology, 82(February), 38-51. https://doi.org/10.1016/j.jmp.2017.11.001
59.
SuenH. K.AryD. (1984). Variables inuencing one-zero and instantaneous time sampling outcomes. Primates, 25(1), 89-94. https://doi.org/10.1007/BF02382298
60.
SuenH. K.AryD. (1986). A post hoc correction procedure for systematic errors in time-sampling duration estimates. Journal of Psychopathology and Behavioral Assessment, 8(1), 31-38. https://doi.org/10.1007/BF00960870
61.
TroisiA.PompiliE.BinelloL.SterponeA. (2007). Facial expressivity during the clinical interview as a predictor functional disability in schizophrenia: A pilot study. Progress in Neuro-Psychopharmacology and Biological Psychiatry, 31(2), 475-481. https://doi.org/10.1016/j.pnpbp.2006.11.016
62.
Van RossumG. (2007). Python programming language. In Usenix annual technical conference (Vol. 41, p. 36). Python Software Foundation.
WeisbergH.WeisbergH. F. (1992). Central tendency and variability. Sage.
65.
WilleR. (1982). Restructuring lattice theory: An approach based on hierarchies of concepts. In RivalI. (Ed.), Ordered sets (pp. 445-470). Reidel.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
