A Partial Simulation Study of Phantom Effects in Multilevel Analysis of School Effects: The Case of School Socioeconomic Composition

Abstract

Hierarchical linear modeling (HLM) is often used to estimate the effects of socioeconomic status (SES) on academic achievement at different levels of an educational system. However, if a prior academic achievement measure is missing in a HLM model, biased estimates may occur on the effects of student SES and school SES. Phantom effects describe the phenomenon in which the effects of student SES and school SES disappear once prior academic achievement is added to the model. In the present analysis, partial simulation (i.e., simulated data are used together with real-world data) was employed to examine the phantom effects of student SES and school SES on science achievement, using the national sample of the United States from the 2015 Programme for International Student Assessment. The results showed that the phantom effects of student SES and school SES are rather real. The stronger the correlation between prior science achievement and (present) science achievement, the greater the chance that the phantom effects occur.

Keywords

partial simulation student socioeconomic status school socioeconomic status science achievement

Get full access to this article

View all access options for this article.

References

Caldas

S. J.

Bankston

. 1997. “Effect of School Population Socioeconomic Status on Individual Academic Achievement.” Journal of Educational Research 90:269–77.

Cohen

1988. Statistical Power Analysis for the Behavioral Sciences. New York: Routledge.

Hedges

L. V.

Hedberg

E. C.

. 2007. “Intraclass Correlation Values for Planning Group-randomized Trials in Education.” Educational Evaluation and Policy Analysis 29:60–87.

Bradley

K. D.

. 2008. “Using Multilevel Modeling to Investigate School Effects.” Pp. 59–110 in Multilevel Modeling of Educational Data, edited by O’Connell

A. A.

McCoach

D. B.

. Charlotte, NC: Information Age.

Zhou

. 2018, September. Developing an Analytical Framework to Examine “Phantom Effects” of School Context on Schooling Outcomes. Paper presented at the annual meeting of the European Educational Research Association. Bolzano, Italy.

Marks

G. N.

2015. “Are School-SES Effects Statistical Artefacts? Evidence from Longitudinal Population Data.” Oxford Review of Education 41:122–44.

Marks

G. N.

2017. “Is SES Really that Important for Educational Outcomes in Australia? A Review and Some Recent Evidence.” Australian Educational Researcher 44:191–211.

Organisation for Economic Co-operation and Development. 2009. PISA Data Analysis Manual: SPSS. Paris: Author.

Organisation for Economic Co-operation and Development. 2016. PISA 2015 Results (Volume I): Excellence and Equity in Education. Paris: Author.

10.

Perry

L. B.

McConney

. 2010. “Does the SES of the School Matter? An Examination of Socioeconomic Status and Student Achievement using PISA 2003.” Teachers College Record 112:1137–62.

11.

Perry

2018. “‘Phantom’ Compositional Effects in English School Value-added Measures: The Consequences of Random Baseline Measurement Error.” Research Papers in Education 34:239–62.

12.

Pokropek

2015. “Phantom Effects in Multilevel Compositional Analysis: Problems and Solutions.” Sociological Methods and Research 44:677–705.

13.

Raudenbush

S. W.

Bryk

A. S.

. 2002. Hierarchical Linear Models: Applications and Data Analysis Methods, 2nd ed. Thousand Oaks, CA: Sage.

14.

Televantou

Marsh

H. W.

Kyriakides

Nagengast

Fletcher

Malmberg

L. E.

. 2015. “Phantom Effects in School Composition Research: Consequences of Failure to Control Biases Due to Measurement Error in Traditional Multilevel Models.” School Effectiveness and School Improvement 26:75–101.

15.

Thum

Y. M.

Bryk

A. S.

. 1997. “Value-added Productivity Indicators: The Dallas System.” Pp. 100–19 in Grading Teachers, Grading Schools: Is Student Achievement a Valid Evaluation Measure? edited by Millman

. Thousand Oaks, CA: Corwin.

16.

White

K. R.

1982. “The Relation between Socioeconomic Status and Academic Achievement.” Psychological Bulletin 91:461–81.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB