Explaining Inconsistencies in the Education Distributions of Ten Cross-National Surveys – the Role of Methodological Survey Characteristics

Abstract

Surveys measuring the same concept using the same measure on the same population at the same point in time should result in highly similar results. If this is not the case, this is a strong sign of lacking reliability, resulting in non-comparable data across surveys. Looking at the education variable, previous research has identified inconsistencies in the distributions of harmonised education variables, using the International Standard Classification of Education (ISCED), across surveys within the same countries and years. These inconsistencies are commonly explained by differences in the measurement, especially in the response categories of the education question, and in the harmonisation when classifying country-specific education categories into ISCED. However, other methodological characteristics of surveys, which we regard as ‘containers’ for several characteristics, may also contribute to this finding. We compare the education distributions of nine cross-national surveys with the European Union Labour Force Survey (EU-LFS), which is used as benchmark. This study analyses 15 survey characteristics to better explain the inconsistencies. The results confirm a predominant effect of the measurement instrument and harmonisation. Different sampling designs also explain inconsistencies, but to a lesser degree. Finally, we discuss the results and limitations of the study and provide ideas for improving data comparability.

Keywords

Comparative research cross-national surveys survey characteristics education

References

Becker, G.S. 1993. Human Capital. A Theoretical and Empirical Analysis with Special Reference to Education, 3rd ed. Chicago: The University of Chicago Press.

Bekhuis, H., Lubbers, M., Verkuyten. M. 2014. “How Education Moderates the Relation Between Globalization and Nationalist Attitudes.” International Journal of Public Opinion Research 26(4): 487–500. DOI: https://doi.org/10.1093/ijpor/edt037.

Bethlehem, J.D., F. Cobben, and B. Schouten. 2011. Handbook of Nonresponse in Household Surveys. Hoboken: John Wiley & Sons.

Biemer, P.P. and L.E. Lyberg. 2003. Introduction to Survey Quality. Hoboken: John Wiley & Sons.

Blair, J., G. Menon, and B. Bickart. 2011. “Measurement Effects in Self vs. Proxy Response to Survey Questions: An Information-Processing Perspective.” In Measurement Errors in Surveys, edited by P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Matthiowetz, S. Sudman, 145–166. Hoboken: John Wiley & Sons.

Blau, P.M. and O.D. Duncan. 1967. The American Occupational Structure. New York: John Wiley & Sons.

Bol, T. and H.G. Van de Werfhorst. 2013. “Educational Systems and the Trade-Off Between Labor Market Allocation and Equality of Educational Opportunity.” Comparative Education Review 57(2): 285–308.

Bolzendahl C.I. and D.J. Myers. 2004. “Feminist Attitudes and Support for Gender Equality: Opinion Change in Women and Men, 1974–1998.” Social Forces 83(2): 759–789. DOI: https://doi.org/10.1353/sof.2005.0005.

Braun, M. and P. Ph. Mohler. 2003. “Background Variables.” In Cross-Cultural Survey Methods, edited by J.A. Harkness, F.J.R. Van de Vijver, and P. Ph. Mohler, 101–115. Hoboken: John Wiley & Sons.

10.

Breen, R. and J.O. Jonsson. 2000. “Analyzing Educational Careers: A Multinomial Transition Model.” American Sociological Review 65(5): 754–772. DOI: https://doi.org/10.2307/2657545.

11.

Breen, R. and J.O. Jonsson. 2005. “Inequality of Opportunity in Comparative Perspective: Recent Research on Educational Attainment and Social Mobility.” Annual Review of Sociology 31: 223–243. DOI: https://doi.org/10.1146/annurev.soc.31.041304.122232.

12.

Church, A.H. 1993. “Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-Analysis.” Public Opinion Quarterly 57(1): 62–79. DOI: https://doi.org/10.1086/269355.

13.

Coenders, M. and P. Scheepers. 2003. “The Effect of Education on Nationalism and Ethnic Exclusionism: An International Comparison.” Political Psychology 24(2): 313–343. DOI: https://doi.org/10.1111/0162-895X.00330.

14.

Couper, M.P. 2000. “Web Surveys: A Review of Issues and Approaches.” Public Opinion Quarterly 64(4): 464–494. DOI: https://doi.org/10.1086/318641.

15.

Daikeler, J., M. Bosnjak, and K. Lozar-Manfreda. 2019. “Web Versus Other Survey Modes: An Updated and Extended Meta-Analysis Comparing Response Rates.” Journal of Survey Statistics and Methodology. DOI: https://doi.org/10.1093/jssam/smz008.

16.

De Leeuw, E.D. and J. Van der Zouwen. 2001. “Data Quality in Telephone and Face-to-Face Surveys: A Comparative Meta-Analysis.” In Telephone Survey Methodology, edited by R.M. Groves, P.P. Biemer, L.E. Lyberg, J.T. Massey, W.L. Nicholls II, and J. Waksberg, 283–300. New York: John Wiley & Sons.

17.

Dever, J.A., A. Rafferty, and R. Valliant. 2008. “Internet Surveys: Can Statistical Adjustments Eliminate Coverage Bias?” Survey Research Methods 2(2): 47–60. DOI: https://doi.org/10.18148/srm/2008.v2i2.128.

18.

Duncan, O.D. and B. Duncan. 1955. “A Methodological Analysis of Segregation Indexes.” American Sociological Review 20(2): 210–217. DOI: https://doi.org/10.2307/2088328.

19.

Ehling, M. 2003. “Harmonising Data in Official Statistics: Development, Procedures, and Data Quality.” In Advances in Cross-National Comparison. A European Working Book of Demographic and Socio-Economic Variables, edited by J.H.P. Hoffmeyer-Zlotnik and C. Wolf, 17–31. New York: Kluwer Academic/Plenum Publishers.

20.

ESS. 2014a. Documentation of ESS Post-Stratification Weights. Available at: https://www.europeansocialsurvey.org/docs/methodology/ESS1-5_post_stratification_weights_documentation.pdf (accessed December 2019).

21.

ESS. 2014b. Weighting European Social Survey Data. Available at: https://www.europeansocialsurvey.org/docs/methodology/ESS_weighting_data_1.pdf (accessed December 2019).

22.

ESS. 2016a. European Social Survey Round 4 Data 2008. Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC data file version 4.4.

23.

ESS. 2016b. European Social Survey Round 5 Data 2010. Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC data file version 3.3.

24.

ESS. 2016c. European Social Survey Round 6 Data 2012. Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC data file version 2.3.

25.

Eurofound. 2011. European Working Conditions Survey (EWCS) 2010. Colchester: UK Data Archive. Study number 6971 data file version 1. DOI: https://doi.org/10.5255/UKDA-SN-6971-1.

26.

Eurofound. 2014. European Quality of Life Survey (EQLS) 2011–2012. Colchester: UK Data Archive. Study number 7316 data file version 3. DOI: https://doi.org/10.5255/UKDA-SN-7316-2.

27.

European Commission. 2012. Eurobarometer 73.2 & 73.3 (2-32010). GESIS Data Archive, Cologne. ZA5236 data file version 2.0.1. DOI: https://doi.org/10.4232/1.11473.

28.

Eurostat. 2008. European Union Labour Force Survey (EU-LFS) 2008. User Database, data file version 2016.

29.

Eurostat. 2010a. European Union Labour Force Survey (EU-LFS) 2010. User Database, data file version 2016.

30.

Eurostat. 2010b. European Union Statistics on Income and Living Conditions (EU-SILC) 2010. User Database, data file version CROSS-2010-6.

31.

Eurostat. 2011a. Adult Education Survey (AES). User Database, data file version 1.

32.

Eurostat. 2011b. European Union Labour Force Survey (EU-LFS) 2011. User Database, data file version 2016.

33.

Eurostat. 2012. European Union Labour Force Survey (EU-LFS) 2012. User Database, data file version 2016.

34.

Eurostat. 2019. Smarter, Greener, More Inclusive? Indicators to Support The Europe 2020 Strategy. Luxembourg: Publication Office of the European Union. Available at: https://ec.europa.eu/eurostat/documents/3217494/10155585/KS-04-19-559-ENN.pdf/b8528d01-4f4f-9c1e-4cd4-86c2328559de (accessed December 2019).

35.

EVS. 2016. European Values Study 2008, 4th wave. Integrated Dataset. GESIS Data Archive, Cologne. ZA4800 data file version 4.0.0. DOI:https://doi.org/10.4232/1.12458.

36.

Groves, R.M. and M.P. Couper. 1998. Nonresponse in Household Interview Surveys. New York: John Wiley & Sons.

37.

Groves, R.M., M.P. Couper, S. Presser, E. Singer, R. Tourangeau, G.P. Acosta, and L. Nelson. 2006. “Experiments in Producing Nonresponse Bias.” Public Opinion Quarterly 70(5): 720–736. DOI: https://doi.org/10.1093/poq/nfl036.

38.

Groves, R.M., F.J. Fowler, M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau. 2009. Survey Methodology, 2nd ed. Hoboken: John Wiley & Sons.

39.

Groves, R.M. and L.E. Lyberg. 2010. “Total Survey Error: Past, Present, and Future.” Public Opinion Quarterly 74(5): 849–879. DOI: https://doi.org/10.1093/poq/nfq065.

40.

Groves, R.M. and E. Peytcheva. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis.” Public Opinion Quarterly 72(2): 167–189. DOI: https://doi.org/10.1093/poq/nfn011.

41.

Heath, A., J. Martin, and T. Spreckelsen. 2009. “Cross-National Comparability of Survey Attitude Measures.” International Journal of Public Opinion Research 21(3): 293–315. DOI: https://doi.org/10.1093/ijpor/edp034.

42.

Hoffmeyer-Zlotnik, J.H.P. 2008. “Harmonisation of Demographic and Socio-Economic Variables in Cross-National Survey Research.” Bulletin de Méthodologie Sociologique 98: 5–24. DOI: https://doi.org/10.1177/075910630809800103.

43.

Hyman, H.H. and C.R. Wright. 1979. Education’s Lasting Influence on Values. Chicago: University of Chicago Press.

44.

ISSP Research Group. 2015. International Social Survey Programme: Health and Health Care - ISSP 2011. GESIS Data Archive, Cologne. ZA5800 data file version 3.0.0. DOI: https://doi.org/10.4232/1.12252.

45.

ISSP Research Group. 2016. International Social Survey Programme: Family and Changing Gender Roles IV - ISSP 2012. GESIS Data Archive, Cologne. ZA5900 data file version 4.0.0. DOI: https://doi.org/10.4232/1.12661.

46.

Kalmijn, M. 2003. “Country Differences in Sex-Role Attittudes: Cultural and Economic Explanations.” In The Cultural Diversity of European Unity. Findings, Explanations and Reflections from the European Values Study, edited by W. Arts, L.C.J.M. Halmann and J.A.P. Hagenaars: 311–337. Leiden: Koninklijke Brill.

47.

Kaminska, O. and P. Lynn. 2017. “Survey-Based Cross-Country Comparisons Where Countries Vary in Sample Design: Issues and Solutions.” Journal of Official Statistics 33(1): 123–136. DOI: https://doi.org/10.1515/JOS-2017-0007.

48.

Kieffer, A. 2010. “Measuring and Comparing Levels of Education: Methodological Problems in the Classification of Educational Levels in the European Social Surveys and the French Labor Force Surveys.” Bulletin de Méthodologie Sociologique 107: 49–73. DOI: https://doi.org/10.1177/0759106310369974.

49.

Kleven, Ø. and K. Ringdal. 2017. Level of Education – Measuring the Quality of Questions in Survey Interviews by Administrative Records on Education. Experiences from the Norwegian European Social Survey 2004 – 2014. In Proceedings of the 7th Conference of the European Survey Research Association (ESRA), July 20, 2017. Lisbon. Available at: https://www.europeansurveyresearch.org/conference/programme2017?sess=210#582 (accessed April 2020).

50.

Kohler, U. 2007. “Surveys from Inside: An Assessment of Unit Nonresponse Bias With Internal Criteria.” Survey Research Methods 1(2): 55–67. DOI: https://doi.org/10.18148/srm/2007.v1i2.75.

51.

Kohler, U. 2008. “Assessing the Quality of European Surveys - Towards an Open Method of Coordination for Survey Data.” In Handbook of Quality of Life in the Enlarged European Union, edited by J. Albers, T. Fahey, and C. Saraceno, 405–423. London, New York: Routledge.

52.

Kreuter, F., S. Eckman, K. Maaz, and R. Watermann. 2010. “Children’s Reports of Parents’ Education Level: Does It Matter Whom You Ask and What You Ask About?” Survey Research Methods 4(3): 127 – 138. DOI: https://doi.org/10.18148/srm/2010.v4i3.4283.

53.

Lohr, S.L. 2009. Sampling: Design and Analysis. 2nd ed. Boston: Brooks/Cole Cengage Learning.

54.

Lyberg, L.E. and D. Kasprzyk. 2011. “Data Collection Methods and Mesurement Error: An Overview.” In Measurement Errors in Surveys, edited by P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowwetz, and S. Sudman, 235–257. Hoboken: John Wiley & Sons.

55.

Minnesota Population Center. 2019. Integrated Public Use Microdata Series, International. Data Minneapolis, MN: IPUMS, data file version 7.2. DOI: https://doi.org/10.18128/D020.V7.2.

56.

Moore, J.C. 1988. “Self/Proxy Response Status and Survey Response Quality: A Review of the Literature.” Journal of Official Statisics 4(2): 155–172.

57.

Müller, W. and W. Karle. 1993. “Social Selection in Educational Systems in Europe.” European Sociological Review 9(1): 1–23. DOI: https://doi.org/10.1093/oxfordjournals.esr.a036652.

58.

Noelle-Neumann, E. and T. Petersen. 2000. “Das halbe Instrument, Die halbe Reaktion. Zum Vergleich von Telefon- und Face-to-Face Umfragen.” In Methoden in Telefonumfragen, edited by V. Hüfken, 183–200. Wiesbaden: Westdeutscher Verlag.

59.

OECD. 2013. Programme for the International Assessment of Adult Competencies (PIAAC) 2011 - Public Use Files: data file version 1.0. Available at: https://www.oecd.org/skills/piaac/data/ (accessed April 2020).

60.

OECD. 2015. Education at a Glance 2015: OECD Indicators. OECD Publishing. Available at: https://www.oecd-ilibrary.org/education/education-at-a-glance-2015_eag-2015-en (accessed December 2019).

61.

OECD. 2016. Education at a Glance 2016: OECD Indicators. OECD Publishing. Available at: https://www.oecd-ilibrary.org/education/education-at-a-glance-2016_eag-2016-en (accessed December 2019).

62.

OECD. 2017. Education at a Glance 2017: OECD Indicators. OECD Publishing. Available at: https://www.oecd-ilibrary.org/education/education-at-a-glance-2017_eag-2017-en (accessed December 2019).

63.

OECD, and Eurostat. 2014. Joint Eurostat-OECD Guidelines on the Measurement of Educational Attainment in Household Surveys. Available at: http://ec.europa.eu/eurostat/documents/1978984/6037342/Guidelines-on-EA-final.pdf (accessed December 2019).

64.

Ortmanns, V. and S.L. Schneider. 2016a. “Can We Assess Survey Representativeness of Cross-National Surveys Using the Education Variable?”. Survey Research Methods 10(3): 189–210. DOI: https://doi.org/10.18148/srm/2016.v10i3.6608.

65.

Ortmanns, V. and S.L. Schneider. 2016b. “Harmonization Still Failing? Inconsistency of Education Variables in Cross-National Public Opinion Surveys.” International Journal of Public Opinion Research 28(4): 562–582. DOI: https://doi.org/10.1093/ijpor/edv025.

66.

Ortmanns. V. 2020. Education distributions and survey characteristics of ten cross-national surveys. GESIS, SowiDataNet|datorium. Data file Version: 1.0.0. DOI: https://doi.org/10.7802/1.2002.

67.

Peytcheva, E. and R.M. Groves. 2009. “Using Variation in Response Rates of Demographic Subgroups as Evidence of Nonresponse Bias in Survey Estimates.” Journal of Official Statistics 25(2): 193–201.

68.

Schanze, J.-L. 2017. Report on Sampling Practices for the Institutionalized Population in Social Surveys. Deliverable 2.16 of the SERISS project funded under the European Union’s Horizon 2020 research and innovation programme GA No: 654221. Available at: www.seriss.eu/resources/deliverables (accessed December 2019).

69.

Scherpenzeel, A., A.M. Maineri, J. Bristle, S. Pflüger, I. Mindarova, S. Butt, S. Zins, T. Emery, and R. Luijkx. 2017. Report on the Use of Sampling Frames in European Studies. Deliverable 2.1 of the SERISS project funded under the European Union’s Horizon 2020 research and innovation programme GA No: 654221. Available at: www.seriss.eu/resources/deliverables (accessed December 2019).

70.

Schneider, S.L. 2009. Confusing Credentials: The Cross-Nationally Comparable Measurement of Educational Attainment. (PhD thesis). Oxford: University of Oxford, Nuffield College. Available at: http://ora.ouls.ox.ac.uk/objects/uuid:15c39d54-f896-425b-aaa8-93ba5bf03529 (accessed December 2019).

71.

Schneider, S.L., D. Joye, and C. Wolf. 2016. “When Translation Is Not Enough: Background Variables in Comparative Surveys.” In The SAGE Handbook of Survey Methodology, edited by C. Wolf, D. Joye, T.W. Smith, and Y.-c. Fu, 288–307. London: Sage Publications.

72.

Semyonov, M., R. Raijman, and A. Gorodzeisky. 2008. “Foreigners’ Impact on European Societies Public Views and Perceptions in a Cross-National Comparative Perspective.” International Journal of Comparative Sociology 49(1): 5–29. DOI: https://doi.org/10.1177/0020715207088585.

73.

Singer, E. and Ye, C. 2013. “The Use and Effects of Incentives in Surveys.” The ANNALS of the American Academy of Political and Social Science 645(1): 112–141. DOI: https://doi.org/10.1177/0002716212458082.

74.

Słomczyński, K.M., Jenkins, J.C., Tomescu-Dubrow, I., Kołczyńska, M., Wysmułek, I., Oleksiyenko, O., Powałko, P. and Zielinski, M.W. 2018. “Survey Data Recycling Dataverse - Master Box.” Harvard Dataverse, data file version 1.1. DOI: https://doi.org/10.7910/DVN/VWGF5Q.

75.

Słomczyński, K.M., Tomescu-Dubrow, I., Jenkins, J.C., Kołczyńska, M., Powałko, P., Wysmułek, I., Oleksiyenko, O., Zielinski, M.W. and Dubrow, J.K. 2016. Democratic Values and Potest Behavior - Harmonization of Data from International Survey Projects. Warsaw: IFiS Publishers.

76.

Smith, T.W. 1995. “Some Aspects of Measuring Education.” Social Science Research 24(3): 215–242. DOI: https://doi.org/10.1006/ssre.1995.1008.

77.

Smith, T.W. 2010. “The Globalization of Survey Research.” In Survey Methods in Multinational, Multiregional, and Multicultural Contexts, edited by J.A. Harkness, M. Braun, B. Edwards, T.P. Johnson, L.E. Lyberg, P.Ph. Mohler, B.-E. Pennell, and T.W. Smith, 477–484. Hoboken: John Wiley & Sons.

78.

Smith, T.W. 2011. “Refining the Total Survey Error Perspective.” International Journal of Public Opinion Research 23(4): 464–484. DOI: https://doi.org/10.1093/ijpor/edq052.

79.

Sodeur, W. 1997. “Interne Kriterien zur Beurteilung von Wahrscheinlichkeitsauswahlen.” In ZA-Information / Zentralarchiv Für Empirische Sozialforschung, 41: 58–82. Cologne. Available at: https://www.ssoar.info/ssoar/handle/document/19999 (accessed December 2019).

80.

Sodeur, W. 2007. “Entscheidungsspielräume von Interviewern bei der Wahrscheinlichkeitsauswahl - Ein Vergleich von ALLBUS-Erhebungen.” MDA - Methoden Daten Analysen 1(2): 107–130.

81.

UNESCO-UIS. 2006. International Standard Classification of Education ISCED 1997. Available at: https://unesdoc.unesco.org/ark:/48223/pf0000146967?posInSet=1&queryId=a8d996ff-eddd-4153-9468-5d7f9e8f7199 (accessed December 2019). Van Tuyckom, C. and P.F. Bracke. 2014. “Survey Quality and Cross-National Sports

82.

Research: A Case Study of the 2007 ISSP Survey.” European Journal of Sport Science 14(1): 228–234. DOI: https://doi.org/10.1080/17461391.2012.683814.

83.

Weakliem, D.L. 2002. “The Effects of Education on Political Opinions: An International Study.” International Journal of Public Opinion Research 13(2): 141–157. DOI: https://doi.org/10.1093/ijpor/14.2.141.