The technical complexities and sheer size of international large-scale assessment (LSA) databases often cause hesitation on the part of the applied researcher interested in analyzing them. Further, inappropriate choice or application of statistical methods is a common problem in applied research using these databases. This article serves as a primer for researchers on the issues and methods necessary for obtaining unbiased results from LSA data. The authors outline the issues surrounding the analysis and reporting of LSA data, with a particular focus on three prominent international surveys. In addition, they make recommendations targeted at applied researchers regarding best analysis and reporting practices when using these databases.
AM Statistical Software [Manual]. (2009). Washington, DC: American Institutes for Research.
2.
AsparouhovT.MuthénB. (2006). Multilevel modelling of complex survey data. Proceedings of the Survey Research Methods Section, American Statistical Association 2006, 2718–2726.
3.
BeatonA.JohnsonE. (1992). Overview of the scaling methodology used in the national assessment. Journal of Educational Measurement, 29, 163–175.
4.
BrewerK.HanifM. (1983). Sampling with unequal probabilities, lecture notes in statistics (Vol. 15). New York: Springer-Verlag.
5.
ChantalaK.SuchindranC. (2006). Adjusting for unequal selection probability in multilevel models: A comparison of software packages. Proceedings of the Survey Research Methods Section, American Statistical Association 2006, 2815–2824.
6.
FoyP.KennedyA. (2008). PIRLS 2006 user guide for the international database. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
7.
FoyP.OlsonJ. (Eds.). (2009). TIMSS 2007 international database and user guide. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
8.
IDB Analyzer (Version 2) [Computer software and manual]. (2009). Hamburg, Germany: International Association for the Evaluation of Educational Achievement.
9.
JohnsonE.RustK. (1992). Population inferences and variance estimation for NAEP data. Journal of Educational Statistics, 17, 175–190.
10.
JudkinsD. (1990). Fay’s method for variance estimation. Journal of Official Statistics, 6, 223–239.
11.
MartinM.MullisI.KennedyA. (Eds.). (2007). PIRLS 2006 technical report. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
12.
MislevyR. (1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56, 177–196.
13.
MislevyR.BeatonA.KaplanB.SheehanK. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29, 133–161.
14.
MislevyR.JohnsonE.MurakiE. (1992). Scaling procedures in NAEP. Journal of Educational Statistics, 17, 131–154.
15.
MullisI.KennedyA.MartinM.SainsburyM. (2006). PIRLS assessment framework and specifications. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
16.
MullisI.MartinM.RuddockG.O’SullivanC.AroraA.ErberberE. (2005). TIMSS 2007 assessment frameworks. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
National Center for Education Statistics. (n.d.). NAEP Data Explorer. In National Center for Education Statistics. Retrieved January 7, 2010, from http://nces.ed.gov/nationsreportcard/naepdata/
19.
OlsonJ.MartinM.MullisI. (Eds.). (2008). TIMSS 2007 technical report. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
Organisation for Economic Co-operation and Development. (2005). PISA 2003 data analysis manual. Paris: Author.
22.
Organisation for Economic Co-operation and Development. (2009). PISA 2006 technical report. Paris: Author.
23.
PfeffermannC.SkinnerD.HolmesH.GoldsteinH.RasbashJ. (1998). Weighting for unequal probabilities in multilevel models. Journal of the Royal Statistical Society, Series B (Statistical Methodology), 60, 23–40.
24.
Rabe-HeskethS.SkrondalA. (2006). Multilevel modelling of complex survey data. Journal of the Royal Statistical Society. Series A (General), 169, 805–827.
RubinD. (1987). Multiple imputation for nonresponse in sample surveys. New York: John Wiley.
27.
RustK. (1985). Variance estimation for complex estimators in sample surveys. Journal of Official Statistics, 1, 381–397.
28.
SAS (Version 9) [Computer software and manual]. (2003). Cary, NC: SAS Institute.
29.
SchaferJ. (1999). Multiple imputation: A primer. Statistical Methods in Medical Research, 8, 3–15.
30.
SchaferJ. (2003). Multiple imputation in multivariate problems when the imputation and analysis models differ. Statistica Neerlandica, 57, 19–35.
31.
SPSS (Version 16.0) [Computer software and manual]. (2007). Chicago: SPSS, Inc.
32.
StapletonL. (2008). Variance estimation using replication methods in structural equation modeling with complex sample data. Structural Equation Modeling, 15, 183–210.
33.
SUDAAN (Release 8.0) [Manual]. (2001). Research Triangle Park, NC: Research Triangle Institute.
34.
ThomasS.HeckR. (2001). Analysis of large-scale secondary data in higher education research: Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517–540.
35.
ThomasS.HeckR.BauerK. (2005). Weighting and adjusting for design effects in secondary data analysis. New Directions for Institutional Research, 2005(127), 51–72.
von DavierM.GonzalezE.MislevyR. (2009). Plausible values: What are they and why do we need them?IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 2, 9–36.
41.
von DavierM.SinharayS.OranjeA.BeatonA. (2006). The statistical procedures used in National Assessment of Educational Progress: Recent developments and future directions. In RaoC. R.SinharayS. (Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp. 1039–1055). Amsterdam: Elsevier.
