We outline a procedure for examining collapsibility over site in multiple-location settings that are frequently utilized in contemporary educational and behavioral research. The method is based on a test of cross-site identity of the response distributions of polytomous items in multi-component measuring instruments, which implies the possibility to pool over study location. The approach is readily applicable in empirical studies using popular and widely circulated software and is generalizable to various types of items. The described procedure is illustrated with data from a child development survey.
AgrestiA. (2002). Categorical data analysis. Wiley.
2.
AgrestiA. (2018). Statistical methods for the social sciences. Pearson.
3.
AreanP. A.KraemerH. C. (2013). High quality psychotherapy research: From conception to piloting to national trials. Oxford University Press.
4.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57, 289–300.
5.
BenjaminiY.YekutieliD. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29, 1165–1188.
6.
BornsteinM. H.YuJ.PutnickD. L. (2020). Mothers’ parenting knowledge and its sources in five societies: Specificity in and across Argentina, Belgium, Italy, South Korea, and the United States. International Journal of Behavioral Development, 44(2), 135–145.
7.
CasellaG.BergerJ. O. (2002). Statistical inference. Wadsworth.
8.
ChenL.MiočevićM.FalkC. F. (2024). Tackling challenges in data pooling: Missing data handling in latent variable models with continuous and categorical indicators. Structural Equation Modeling, 31, 1–16.
9.
CurranP. J.HussongA. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100.
10.
GoldsteinH. (2011). Multilevel modeling. Arnold.
11.
GrimesD.WarschauerM. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8, 35–47.
12.
HarrellF. E. (2015). Regression modeling strategies with applications to linear models, logistic and ordinal regression, and survival analysis. Springer.
13.
HuangG. C.LindellD.JaffeL. E.SullivanA. M. (2016). A multi-site study of strategies to teach critical thinking: Why do you think that?Medical Education, 50, 236–249.
14.
International Business Machines Statistical Package for the Social Sciences. (2023). IBM SPSS statistics for windows. International Business Machines Corporation. https://www.ibm.com/products/spss-statistics
15.
JohnsonR. A.WichernD. W. (2018). Applied multivariate statistical analysis. Prentice Hall.
16.
KushJ. M.MasynK. E.Amin-EsmaeiliM.SusukidaR.WilcoxH. C.MusciR. J. (2023). Utilizing moderated non-linear factor analysis models for integrative data analysis: A tutorial. Structural Equation Modeling, 30, 149–164.
17.
LohrS. L. (1999). Sampling: Design and analysis. Chapman Hall.
18.
MacPheeD. (2002). Manual for the knowledge of infant development inventory [Unpublished manuscript]. University of North Carolina.
19.
MarcoulidesK. M. (2023). Integration of historical data for the analysis of multiple assessment studies. Measurement: Interdisciplinary Research and Perspectives, 21, 181–193.
20.
Nobre-LimaL.da Luz Vale-DiasM.MendesT. V.MónicoL.MacPheeD. (2014). The Portuguese version of the knowledge of infant development inventory-P (KIDI-P). European Journal of Developmental Psychology, 11(6), 740–745.
21.
Rabe-HeskethS.SkrondalA. (2022). Multilevel and longitudinal modeling using Stata. Stata Press.
22.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear modeling: Applications and data analysis methods. Sage.
23.
RaykovT.MarcoulidesG. A. (2012). Basic statistics: An introduction with R. Rowman & Littlefield.
24.
RaykovT.MarcoulidesG. A.ChangC. (2016). Studying population heterogeneity in finite mixture settings using latent variable modeling. Structural Equation Modeling, 23, 726–730.
25.
RaykovT.MarcoulidesG. A.GablerS.LeeY. (2017). Testing criterion correlations with scale component measurement errors using latent variable modeling. Structural Equation Modeling, 24, 468–474.
26.
ReckaseM. D. (2009). Multidimensional item response theory. Springer.
27.
SeptemberS. J.RichE. G.RomanN. V. (2016). The role of parenting styles and socio-economic status in parents’ knowledge of child development. Early Child Development and Care, 186(7), 1060–1078.
28.
ShuD.LiX.HerQ.WongJ.LiD.WangR.TohS. (2023). Combining meta-analysis with multiple imputation for one-step, privacy-protecting estimation of causal treatment effects in multi-site studies. Research Synthesis Methods, 14(5), 742–763. https://doi.org/10.1002/jrsm.1660
29.
SinghA.UwimpuhweG.LiM.EinbeckJ.HigginsS.KasimA. (2022). Multisite educational trials: Estimating the effect size and its confidence intervals. International Journal of Research & Method in Education, 45(1), 18–38.
30.
WassermanL. W. (2004). All of statistics: A concise course in statistical inference. Springer.
