Can One Pool Over Site in a Multi-Site Study With Categorical Item Measuring Instruments? A Multiple Testing Procedure

Abstract

We outline a procedure for examining collapsibility over site in multiple-location settings that are frequently utilized in contemporary educational and behavioral research. The method is based on a test of cross-site identity of the response distributions of polytomous items in multi-component measuring instruments, which implies the possibility to pool over study location. The approach is readily applicable in empirical studies using popular and widely circulated software and is generalizable to various types of items. The described procedure is illustrated with data from a child development survey.

Keywords

Benjamini–Hochberg procedure categorical item collapsibility integrative data analysis multi-site study polytomous item

Get full access to this article

View all access options for this article.

References

Agresti

(2002). Categorical data analysis. Wiley.

Agresti

(2018). Statistical methods for the social sciences. Pearson.

Arean

P. A.

Kraemer

H. C.

(2013). High quality psychotherapy research: From conception to piloting to national trials. Oxford University Press.

Benjamini

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57, 289–300.

Benjamini

Yekutieli

(2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29, 1165–1188.

Bornstein

M. H.

Putnick

D. L.

(2020). Mothers’ parenting knowledge and its sources in five societies: Specificity in and across Argentina, Belgium, Italy, South Korea, and the United States. International Journal of Behavioral Development, 44(2), 135–145.

Casella

Berger

J. O.

(2002). Statistical inference. Wadsworth.

Chen

Miočević

Falk

C. F.

(2024). Tackling challenges in data pooling: Missing data handling in latent variable models with continuous and categorical indicators. Structural Equation Modeling, 31, 1–16.

Curran

P. J.

Hussong

A. M.

(2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100.

10.

Goldstein

(2011). Multilevel modeling. Arnold.

11.

Grimes

Warschauer

(2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8, 35–47.

12.

Harrell

F. E.

(2015). Regression modeling strategies with applications to linear models, logistic and ordinal regression, and survival analysis. Springer.

13.

Huang

G. C.

Lindell

Jaffe

L. E.

Sullivan

A. M.

(2016). A multi-site study of strategies to teach critical thinking: Why do you think that? Medical Education, 50, 236–249.

14.

International Business Machines Statistical Package for the Social Sciences. (2023). IBM SPSS statistics for windows. International Business Machines Corporation. https://www.ibm.com/products/spss-statistics

15.

Johnson

R. A.

Wichern

D. W.

(2018). Applied multivariate statistical analysis. Prentice Hall.

16.

Kush

J. M.

Masyn

K. E.

Amin-Esmaeili

Susukida

Wilcox

H. C.

Musci

R. J.

(2023). Utilizing moderated non-linear factor analysis models for integrative data analysis: A tutorial. Structural Equation Modeling, 30, 149–164.

17.

Lohr

S. L.

(1999). Sampling: Design and analysis. Chapman Hall.

18.

MacPhee

(2002). Manual for the knowledge of infant development inventory [Unpublished manuscript]. University of North Carolina.

19.

Marcoulides

K. M.

(2023). Integration of historical data for the analysis of multiple assessment studies. Measurement: Interdisciplinary Research and Perspectives, 21, 181–193.

20.

Nobre-Lima

da Luz Vale-Dias

Mendes

T. V.

Mónico

MacPhee

(2014). The Portuguese version of the knowledge of infant development inventory-P (KIDI-P). European Journal of Developmental Psychology, 11(6), 740–745.

21.

Rabe-Hesketh

Skrondal

(2022). Multilevel and longitudinal modeling using Stata. Stata Press.

22.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear modeling: Applications and data analysis methods. Sage.

23.

Raykov

Marcoulides

G. A.

(2012). Basic statistics: An introduction with R. Rowman & Littlefield.

24.

Raykov

Marcoulides

G. A.

Chang

(2016). Studying population heterogeneity in finite mixture settings using latent variable modeling. Structural Equation Modeling, 23, 726–730.

25.

Raykov

Marcoulides

G. A.

Gabler

Lee

(2017). Testing criterion correlations with scale component measurement errors using latent variable modeling. Structural Equation Modeling, 24, 468–474.

26.

Reckase

M. D.

(2009). Multidimensional item response theory. Springer.

27.

September

S. J.

Rich

E. G.

Roman

N. V.

(2016). The role of parenting styles and socio-economic status in parents’ knowledge of child development. Early Child Development and Care, 186(7), 1060–1078.

28.

Shu

Her

Wong

Wang

Toh

(2023). Combining meta-analysis with multiple imputation for one-step, privacy-protecting estimation of causal treatment effects in multi-site studies. Research Synthesis Methods, 14(5), 742–763. https://doi.org/10.1002/jrsm.1660

29.

Singh

Uwimpuhwe

Einbeck

Higgins

Kasim

(2022). Multisite educational trials: Estimating the effect size and its confidence intervals. International Journal of Research & Method in Education, 45(1), 18–38.

30.

Wasserman

L. W.

(2004). All of statistics: A concise course in statistical inference. Springer.