In the current article, we consider the influential position of the Programme for International Student Assessment (PISA) and discuss several methodological areas that demonstrate the need for caution when using and interpreting PISA results. We motivate our argument by briefly describing the program’s increased influence in educational policy over time. Subsequently, we describe the methodological areas of interest, including sampling participants, the achievement estimation model, and measuring trends. We also offer our perspectives on how the Organisation for Economic Co-operation and Development might productively and more clearly communicate PISA’s limitations.
AdamsR. J.WuM. L. (2007). The mixed-coefficients multinomial logit model: A generalized form of the Rasch model. In CarstensenC. H. (Ed.), Multivariate and mixture distribution Rasch models (pp. 57–75). New York, NY: Springer. Retrieved from http://link.springer.com/chapter/10.1007/978-0-387-49839-3_4
2.
AdamsR. J.WuM. L.CarstensenC. H. (2007). Multivariate and mixture distribution Rasch models: Extensions and applications. In Application of multivariate Rasch models in international large scale educational assessment (pp. 271–280). New York, NY: Springer. Retrieved from http://www.springerlink.com/content/n33521777w9g0373/
3.
BraceyG. (2008, December9). International comparisons: More fizzle than fizz. Huffington Post. Retrieved from http://www.huffingtonpost.com
4.
BrownG.MicklewrightJ.SchnepfS. V.WaldmannR. (2007). International surveys of educational achievement: How robust are the findings?Journal of the Royal Statistical Society: Series A (Statistics in Society), 170(3), 623–646.
ChengK.-M. (2010). Shanghai and Hong Kong: Two distinct examples of education reform in China. In Strong performers and successful performers in education: Lessons from PISA for the United States (pp. 83–115). Paris, France: Organisation for Economic Co-operation and Development.
7.
DobbinsM.MartensK. (2012). Towards an education approach à la finlandaise? French education policy after PISA. Journal of Education Policy, 27(1), 23–43. http://doi.org/10.1080/02680939.2011.622413
EgelundN. (2008). The value of international comparative studies of achievement: A Danish perspective. Assessment in Education: Principles, Policy & Practice, 15(3), 245–251. http://doi.org/10.1080/09695940802417400
10.
ErtlH. (2006). Educational standards and the changing discourse on education: The reception and consequences of the PISA study in Germany. Oxford Review of Education, 32(5), 619–634. http://doi.org/10.1080/03054980600976320
11.
GoldsteinH. (2004). International comparisons of student attainment: Some issues arising from the PISA study. Assessment in Education: Principles, Policy & Practice, 11(3), 319–330. http://doi.org/10.1080/0969594042000304618
12.
GrekS. (2009). Governing by numbers: The PISA “effect” in Europe. Journal of Education Policy, 24(1), 23–37.
13.
HambletonR. K.RogersH. J. (1989). Detecting potentially biased test items: Comparison of IRT area and Mantel-Haenszel methods. Applied Measurement in Education, 2(4), 313–334. http://doi.org/10.1207/s15324818ame0204_4
14.
HopmannS.BrinekG.RetzlM. (Eds.). (2007). PISA according to PISA: Does PISA keep what it promises?Vienna, Austria: Lit Verlag.
15.
KreinerS.ChristensenK. B. (2014). Analyses of model fit and robustness. A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika, 79(2), 210–231. http://doi.org/10.1007/s11336-013-9347-z
16.
MellenberghG. J. (1982). Contingency table models for assessing item bias. Journal of Educational and Behavioral Statistics, 7(2), 105–118. http://doi.org/10.3102/10769986007002105
MillsapR. E. (2011). Statistical approaches to measurement invariance. New York, NY: Routledge.
19.
MislevyR. J. (1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56(2), 177–196.
20.
MislevyR. J.BeatonA. E.KaplanB.SheehanK. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29(2), 133–161.
21.
MislevyR. J.JohnsonE. G.MurakiE. (1992). Chapter 3: Scaling procedures in NAEP. Journal of Educational and Behavioral Statistics, 17(2), 131–154. http://doi.org/10.3102/10769986017002131
22.
MullisI. V. S.MartinM. O.FoyP.AroraA. (2012). TIMSS 2011 international results in mathematics. Boston, MA: TIMSS and PIRLS International Study Center, Boston College.
Organisation for Economic Co-operation and Development. (2013a). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving, and financial literacy. Paris, France: Author. Retrieved from http://dx.doi.org/10.1787/9789264190511-en
26.
Organisation for Economic Co-operation and Development. (2013b). PISA 2012 international database. Paris, France: Author. Retrieved from http://pisa2012.acer.edu.au/downloads.php
27.
Organisation for Economic Co-operation and Development. (2014a). PISA 2012 results: What students know and can do. Paris, France: Author.
28.
Organisation for Economic Co-operation and Development. (2014b). PISA 2012 technical report. Paris, France: Author.
29.
OliveriM. E.von DavierM. (2011). Investigation of model fit and score scale comparability in international assessments. Psychological Test and Assessment Modeling, 53(3), 315–333.
30.
PurvesA. C. (1987). The evolution of the IEA: A memoir. Comparative Education Review, 31(1), 10–28. http://doi.org/10.2307/1188220
31.
RaschG. (1980). Probabalistic models for intelligence and attainment tests (2nd ed.). Chicago, IL: University of Chicago.
32.
RizviF.LingardB. (2009). The OECD and global shifts in education policy. In CowenR.KazamiasA. M. (Eds.), International handbook of comparative education (pp. 437–453). Dordrecht, Netherlands: Springer. Retrieved from http://link.springer.com/chapter/10.1007/978-1-4020-6403-6_28
RubinD. B. (1987). Multiple imputation for nonresponse in surveys. Hoboken, NJ: Wiley.
35.
RutkowskiD.RutkowskiL. (2013). Measuring socioeconomic background in PISA: One size might not fit all. Research in Comparative and International Education, 8(3), 259–278.
RutkowskiL. (2014). Sensitivity of achievement estimation to conditioning model misclassification. Applied Measurement in Education, 27(2), 115–132. http://doi.org/10.1080/08957347.2014.880440
38.
RutkowskiL.RutkowskiD. (2010). Getting it “better”: The importance of improving background questionnaires in international large-scale assessment. Journal of Curriculum Studies, 42(3), 411–430. http://doi.org/10.1080/00220272.2010.487546
39.
RutkowskiL.RutkowskiD.ZhouY. (2016). Parameter estimation methods and the stability of achievement estimates and system rankings: Another look at the PISA model. International Journal of Testing, 16(1), 1–20.
40.
SahlbergP. (2011). Finnish lessons: What can the world learn from educational change in Finland?New York, NY: Teachers College Press.
41.
SahlbergP. (2013, December8). The PISA 2012 scores show the failure of “market based” education reform. Guardian. Retrieved from http://www.theguardian.com
SellarS.LingardB. (2013a). Looking east: Shanghai, PISA 2009 and the reconstitution of reference societies in the global education policy field. Comparative Education, 49(4), 464–485. http://doi.org/10.1080/03050068.2013.770943
ShoemakerD. M. (1973). Principles and procedures of multiple matrix sampling (Vol. 18). Oxford, UK: Ballinger.
46.
TakayamaK. (2008). The politics of international league tables: PISA in Japan’s achievement crisis debate. Comparative Education, 44(4), 387–407. http://doi.org/10.1080/03050060802481413
47.
von DavierM.GonzalezE.MislevyR. J. (2009). What are plausible values and why are they useful?IERI Monograph Series, 2, 9–36.
