Abstract
As occupational data play a crucial part in many social and economic analyses, information on the reliability of these data and, in particular on the role of coding agencies, is important. Based on our review of previous research, we develop four hypotheses, which we test using occupation-coded data from the German General Social Survey and the field test data from the German Programme for the International Assessment of Adult Competencies. Because the same data were coded by several agencies, their coding results could be directly compared. As the surveys used different instruments, and interviewer training differed, the effects of these factors could also be evaluated.
Our main findings are: the percentage of uncodeable responses is low (1.8–4.9%) but what is classified as “uncodeable” varies between coding agencies. Inter-agency coding reliability is relatively low κ ca. 0.5 at four-digit level, and codings sometimes differ systematically between agencies. The reliability of derived status scores is satisfactory (0.82–0.90). The previously reported negative relationship between answer length and coding reliability could be replicated and effects of interviewer training demonstrated. Finally, we discuss the importance of establishing common coding rules and present recommendations to overcome some of the problems in occupation coding.
References
1.
Belloni, M., A. Brugiavini, E. Meschi, and K. Tijdens. 2016. “Measuring and Detecting Errors in Occupational Coding: An Analysis of SHARE Data.” Journal of Official Statistics 32(4): 917–945. Doi: http://dx.doi.org/10.1515/JOS-2016-0049.
2.
Bergmann, M.M. and D. Joye. 2005. “Comparing Social Stratification Schemata: CAMSIS, CSP-CH, Goldthorpe, ISCO-88, Treiman, and Wright.” Cambridge Studies in Social Research 10: 1–35. Available at: https://www.sociology.cam.ac.uk/research/srg/cs10 (accessed January 2019).
3.
Billiet, J. and G. Loosveldt. 1988. “Improvement of the Quality of Responses to Factual Survey Question by Interviewer Training.” Public Opinion Quarterly 52: 190–211. Doi: http://dx.doi.org/10.1086/269094.
4.
Bundesagentur für Arbeit. 2011. Klassifikation der Berufe 2010. Systematischer und alphabetischer Teil mit Erläuterungen . Nürnberg: Bundesagentur für Arbeit.
5.
Bushnell, D. 1998. “An Evaluation of Computer-Assisted Occupation Coding.” In Proceedings of the International Conference New Methods for Survey Research , August 21–22, 1998: 23–26. Chilworth Manor, Southampton, United Kingdom.
6.
Campanelli, P., K. Thomson, N. Moon, and T. Staples. 1997. “The Quality of Occupational Coding in the United Kingdom.” In Survey Measurement and Process Quality , edited by L. Lyberg, P. Biemer, M. Collins, E. De Leeuw, C. Dippo, N. Schwarz, and D. Trewin, 437–453. New York: John Wiley & Sons, Inc.
7.
Cantor, D. and J.L. Esposito. 1992. “Evaluating Interviewer Style for Collecting Industry and Occupation Information.” Proceedings of the Section on Survey Methods, American Statistical Association : 661–666.
8.
Conrad, F.G., M.P. Couper, and J.W. Sakshaug. 2016. “Classifying Open-Ended Reports: Factors Affecting the Reliability of Occupation Codes.” Journal of Official Statistics 32(1): 75–92. Doi: http://dx.doi.org/10.1515/JOS-2016-0003.
9.
Desrosières, Alain. 1996. “Statistical Traditions: An Obstacle to International Comparisons?” In Cross-National Research Methods in the Social Sciences , edited by L. Hantrais and S. Mangen, 17–27. New York: Cassel.
10.
Elias, P. 1997. “Occupational Classification (ISCO-88): Concepts, Methods, Reliability, Validity and Cross-National Comparability.” OECD Labour Market and Social Policy Occasional Papers 20. Doi: http://dx.doi.org/10.1787/304441717388.
11.
Freelon, D.G. 2010. “Recal: Intercoder Reliability Calculation as a Web Service.” International Journal of Internet Science 5(1): 20–33. Available at: http://www.ijis.net/ijis5_1/ijis5_1_freelon.pdf (accessed January 2019).
12.
Ganzeboom, H.B.G. and D.J. Treiman. 1996. “Internationally Comparable Measures of Occupational Status for the 1988 International Standard Classification of Occupations.” Social Science Research 25: 201–239. Doi: http://dx.doi.org/10.1006/ssre.1996.0010.
13.
Ganzeboom, H.B.G. and D.J. Treiman. 2003. “Three Internationally Standardised Measures for Comparative Research on Occupational Status.” In Advances in Cross-National Comparison. A European Working Book for Demographic and Socio-Economic Variables , edited by J.H.P. Hoffmeyer-Zlotnik and C. Wolf, 159–193. New York: Kluwer Acadamic/Plenum Publishers.
14.
Ganzeboom, H.B.G. and D.J. Treiman. 2012. “International Stratification and Mobility File: Conversion Tools.” Amsterdam: Department of Social Research Methodology. Available at: http://www.harryganzeboom.nl/ismf/index.htm. Retrieved 2017/02/27.
15.
Geis, A. and J.H.P. Hoffmeyer-Zlotnik. 2000. “Stand der Berufsvercodung.” ZUMANachrichten 47: 103–128. Available at: https://www.gesis.org/fileadmin/upload/forschung/publikationen/zeitschriften/zuma_nachrichten/zn_47.pdf (accessed January 2019).
16.
Gweon, H., M. Schonlau, L. Kaczmirek, M. Blohm, and S. Steiner. 2017. “Three Methods for Occupation Coding Based on Statistical Learning.” Journal of Official Statistics 33(1): 101–122. Doi: http://dx.doi.org/10.1515/jos-2017-0006.
17.
Hak, T. and T. Bernts. 1996. “Coder Training: Theoretical Training or Practical Socialization?” Qualitative Sociology 19(2): 235–257. Doi: http://dx.doi.org/10.1007/BF02393420.
18.
Hoffmann, E., P. Elias, B. Embury, and R. Thomas. 1995. What Kind of Work Do You Do? Data Collection and Processing Strategies When Measuring “Occupation” for Statistical Surveys and Administrative Records . Geneva: ILO.
19.
Hoffmeyer-Zlotnik, J.H.P., D. Hess, and A. Geis. 2004. “Computerunterstützte Vercodung der International Standard Classification of Occupations (ISCO-88).” ZUMANachrichten 55: 29–52. Available at: https://www.ssoar.info/ssoar/bitstream/handle/document/20762/ssoar-zuma-2004-55-hoffmeyer-zlotnik_et_al-computerunterstutzte_vercodung_der_international_standard.pdf?sequence=1 (accessed January 2016).
20.
International Labour Office (ILO). 2012. International Standard Classification of Occupations 2008 (ISCO-08): Structure, Group Definitions and Correspondence Tables . Geneva: ILO.
21.
Maaz, K., U. Trautwein, C. Gresch, O. Lüdtke, and R. Watermann. 2009. “Intercoder-Reliabilität bei der Berufscodierung nach der ISCO-88 und Validität des sozioökonomischen Status.” ZfE 12: 281–301. Doi: http://dx.doi.org/10.1007/s11618-009-0068-0.
22.
Schierholz, M., M. Gensicke, N. Tschersich, and F. Kreuter. 2017. “Occupation Coding During the Interview.” Journal of the Royal Statistical Society A 181: 379–407. Doi: http://dx.doi.org/10.1111/rssa.12297.
23.
‘t Mannetje, A. and H. Kromhout. 2003. “The Use of Occupation and Industry Classifications in General Population Studies.” International Journal of Epidemiology 32: 419–428. Doi: http://dx.doi.org/10.1093/ije/dyg080.
24.
Tijdens, K. 2014. “Dropout Rates and Response Times of an Occupation Search Tree in a Web Survey.” Journal of Official Statistics 30(1): 23–43. Doi: http://dx.doi.org/10.2478/jos-2014-0002.
25.
Tijdens, K. 2015. “Self-Identification of Occupation in Web Surveys: Requirements for Search Trees and Look-up Tables.” Survey Insights: Methods from the Field . Doi: http://dx.doi.org/10.13094/SMIF-2015-00008.