In a symbiosis between social research methodology, statistics and information technology, a strategy can be found for evaluation in large systems and organizations. The paper outlines a multilevel design concept for studies of the outcome from interventions on different levels, including the whole system, its parts, and the individuals in it. For the case of the evaluation of the outcome of cuts in funding of schools, the presentation illustrates the necessity to collect data directly from teachers and students themselves. Methodological advancements in psychometrics and statistics are presented and illustrated.
Get full access to this article
View all access options for this article.
References
1.
Balke, Gudrun (1995) 'Decomposition of Reading Comprehension-Analysis of the IEA Reading Literacy Test'. Paper presented at the Annual Meeting of the American Educational Research Association, AERA , San Francisco. Göteborg University , Sweden.
2.
Barkman, Catharina and Stefan Fölster (1996) Can Government Agencies Evaluate Themselves? (In Swedish: Kan myndigheter utvardera sig själva?). The Expert Group on Public Finance, ESO, Ds 1996: 36Stockholm: The Ministry of Finance.
3.
Biemer, Paul and S. Lynne Strokes (1991) 'Approaches to the Modeling of Measurement Errors', in Paul Biemer et al. (eds) Measurement Errors in Surveys. New York: Wiley.
4.
Bulder, Bert, Frans Leeuw and Henk Flap (1996) 'Networks and Evaluating Public-sector Reforms' , Evaluation2(3): 261-76.
5.
Elley, Warwick B. (1992) The IEA Study of Reading Literacy: Achievement and Instruction in Thirty-Two School Systems. London: Pergamon.
6.
Furubo, Jan-Eric and Rolf Sandahl (1993) Does the Result Matter? The Role of Stakeholders, Expertise and Executors in the Production of Result-based DataStockholm: Swedish National Audit Bureau.
7.
Groves, Robert M. (1989) Survey Errors and Survey Costs. New York: Wiley.
8.
Gustafsson, J.-E. (1994) 'Hierarchical Models of Intelligence and Educational Achievement' in A. Demetriou and A. Etklides (eds) Intelligence, Mind and Reasoning. Structure and Development. Amsterdam: North-Holland.
9.
Gustafsson, Jan-Eric and Per-Arne Stahl (1997) STREAMS User's Guide, Version 2.0 for Windows. Göteborg, Sweden: Multivariate Ware.
Hoyle, Rick H., ed. (1995) Structural Equation Modeling-Concepts, Issues, and ApplicationsThousand Oaks, CA: Sage.
12.
Inkeles, Alex (1977) 'The International Evaluation of Educational Achievement—A Review' in Proceedings of the National Academy of Education4: 139-200.
13.
Jöreskog, Karl G. (1970) 'A General Method for Analysis of Covariance Structures' , Biometrika57: 239-51.
14.
Joreskog, Karl G. and Dag Sörbom (1993) LISREL 8: Structural Equation Modeling with the SIMPLIS Command LanguageChicago: SSI Scientific Software International.
15.
Kaplan, David and Pamela R. Elliott (1997) 'A Didactic Example of Multilevel Structural-Equation Modeling Applicable to the Study of Organizations', Structural Equation Modeling4(2): 1-24.
16.
Mueller, Ralph O. (1996) Basic Principles of Structural Equation Modeling-An Introduction to LISREL and EQS. New York: Springer-Verlag.
17.
Munck, Ingrid M. E. (1995a) Dimensions of Reading Literacy—Structural Multilevel Modelling of IEA Data. Symposium at American Educational Research Association Annual Meeting, AERA, San Fransisco. Stockholm: Statistics Sweden.
18.
Munck, Ingrid M. E. (1995b) Linking Reading Preferences with Reading Performance— Structural Relationships at Individual and School Levels in Two Different Educational Systems. Paper presented at the American Educational Research Association Annual Meeting, San Francisco. AERA. Stockholm: Statistics Sweden.
19.
Munck, Ingrid M. E. (1996) Modeling of the Statistics Sweden Personnel Questionnaire-a Nested Factor Design Analyzed with LISREL. Proceedings of the Conference on Methodological Issues in Official Statistics, Stockholm, June 12-13. Stockholm: Statistics Sweden.
20.
Munck, Ingrid M. E. (1997) 'Effekstudier vid utvärdering av skolan' (Effect Studies of Evaluations in Schooling—in Swedish with summary in English , Pedagogisk forskning i Sverige2(1): 1-24.
21.
Muthén, Bengt O. (1989) 'Latent Variable Modeling in Heterogenous Populations' , Psychometrika54: 557-85.
Muthén, Bengt O. and A. Satorra (1995) 'Complex Sample Data in Structural Equation Modeling' in P. V. Marsden (ed.) Sociological Methodology, pp. 267-316. Washington, DC: American Sociological Association.
24.
Oecd (1996) Education at a Glance . Paris: OECD.
25.
Pollitt, Christopher (1995) 'Justification by Works or by Faith? Evaluating the New Public Management', Evaluation1(2): 133-54.
26.
Ribe, Martin (1996) 'Statistical Quality in Evaluation', in European Harmonization, National Decentralization, and Quality. Proceedings of the Conference on Methodological Issues in Official Statistics, Stockholm, June 12-13. Stockholm: Statistics Sweden.
27.
Rosén, Monica (1995) Gender Differences in Reading Performance on Documents Across Countries. Paper presented at the Annual Meeting of the American Educational Research Association, AERA, San Francisco. Göteborg University, Sweden.
28.
Rosén, Monica (1997) Country Differences in Reading Performance: A Reanalysis of the IEA Reading Literacy Study. Paper to be presented at Symposium on Latent Variable Modeling of Complex Data at AERA Conference 1997 in Chicago. Göteborg University, Sweden.
29.
Siq (1996) The Swedish Quality Award 1996. Guidelines—A Tool for the Development of Organizations. Göteborg, Sweden: Swedish Institute for Quality.
30.
Vedung, Evert (1996) Public Policy and Program Evaluation. London: Transaction Publishers.
