Beyond Algorithmic Accuracy: Understanding Data Quality in Interactive Occupational Coding with the COARSE Framework

Abstract

Survey research increasingly relies on automatic tools to classify open-ended occupational data. With the rise of machine learning (ML) and large language models (LLMs), this task is shifting from post-survey coding to live classification during interviews. Respondents actively evaluate algorithmic suggestions, creating a joint human–machine decision process where data quality cannot be assessed by accuracy metrics alone. Drawing on human–computer interaction and survey methodology, we propose the COARSE (Classification Outcomes and Respondent Engagement) evaluation framework, which distinguishes five outcomes: accurate classification, misclassification, omission error, commission error, and appropriate rejection. Evidence from a representative German survey deploying an interactive ML-based instrument (OccuCoDe) shows that respondents cannot reliably safeguard data quality when algorithms fail. Instead of rejecting poor suggestions, they often settle for “good enough” answers. When correct options are present, errors reflect misjudgments of subtle distinctions. When respondents reject machine suggestions, they report higher task difficulty afterward, especially when valid options were overlooked. These results show that data errors extend to human cognition and survey interaction, going beyond established machine learning metrics for algorithmic decision-making. As humans and machines collaborate in the survey-answering process, the COARSE framework offers a new lens to evaluate data quality and improve automated coding systems.

Keywords

occupational coding Human-Computer Interaction data quality Machine Learning respondent behavior

1. Introduction

With CAPI, CATI, and CAWI well established as modes of survey data collection, machine learning approaches now make it technically possible to embed complex tasks such as occupational coding directly into interview scripts, dynamically filtering answer options and follow-up questions based on each respondent’s input. Traditionally, occupational information, which plays a central role in labor market analysis and demographic research, has been collected as open-ended answers and then classified into standardized taxonomies such as the International Standard Classification of Occupations (ISCO) or national classification systems by professional coders. The emergence of the automatic coding tools began to complement this manual step, and make it feasible to transfer the task directly into the survey process.

Automatic coding tools take a respondent’s input (often a job title) and assign the most likely occupational code. Their performance is typically judged by production rate (the proportion of cases for which the tool produced an occupational suggestion) and accuracy rate (how many were classified correctly when compared to manual coders) (Bao et al. 2020; Schierholz et al. 2018). Once such tools are embedded in surveys, evaluation is no longer a straightforward comparison between automated coding and professional coding, since the classification outcome depends on both algorithmic suggestions and respondent cognitive process and choices. This shifts the unit of analysis from the coding tool to the respondent–algorithm interaction, as the observed outcome can no longer be attributed to the tool alone.

This scenario has direct implications for measurement practice. Standard accuracy metrics conflate errors originating in the algorithmic suggestion process with those arising from respondents’ interpretation and selection of the presented categories. As a result, they obscure the mechanisms that contribute to misclassification and offer limited guidance for improvement. Addressing algorithmic errors requires different strategies than addressing respondent-related difficulties, such as ambiguity in category labels or increased cognitive burden. An extended evaluation framework must therefore distinguish between these sources of error.

Building on human-computer interaction (HCI) literature and survey methodology, we propose a COARSE (Classification Outcomes and Respondent Engagement) analytical framework to evaluate respondent–algorithm interactions in surveys with algorithm-generated answer options (Figure 1). The COARSE framework distinguishes five possible scenarios that combine the accuracy of the algorithmic output with respondent behavior: Accurate Classification (AC), when respondents accept a correct suggestion; Misclassification Error (ME), when they select an incorrect suggestion although the correct one is available (this outcome arises only in multi-suggestion settings); Omission Error (OE), when they reject all suggestions despite a correct suggestion being present; Commission Error (CE), when they accept an incorrect suggestion if no correct option exists; and Appropriate Rejection (AR), when they reject all suggestions because none is correct. These outcomes simultaneously capture algorithmic output correctness and respondent behavior, offering a more precise assessment of data quality.

Figure 1.

The COARSE framework (classification outcomes and respondent engagement). It distinguishes five outcomes of respondent–algorithm interaction in survey-embedded occupational coding. White cells indicate appropriate outcomes (accurate classification and appropriate rejection), whereas grey cells indicate errors (misclassification, commission, and omission).

Our contribution is twofold. First, we argue that data quality in survey integrated automated occupational coding should be judged by COARSE outcomes, not by algorithmic accuracy alone. Second, using data from a probability-based nationwide CATI survey in Germany that embedded the OccuCoDe instrument (Simson et al. 2023), we demonstrate how COARSE framework identifies risks that traditional accuracy metrics overlook. More broadly, we show how embedding algorithms in surveys transforms data collection into a hybrid human–machine process, raising new challenges and opportunities for official statistics.

2. Background

2.1. Automatic Occupational Coding and Its Limits

Occupational information is primarily collected through surveys. Traditionally, respondents are asked several open-ended questions to describe their job title, job duties, and potentially the industry or company they work for (Christoph et al. 2020). Once the survey is completed, professional coders evaluate responses and typically assign a single code to classify a person’s occupation into a national (such as KldB in Germany, SOC in the USA, or CSCO in China) and/or international (such as ISCO [ILO 2012]) classification system. However it is a lengthy and costly process (Bao et al. 2020) additionally plagued by the fact that there is no universal gold standard for occupational coding. Meaning that assignment of each occupational category is a negotiation process and a final code might be a product of individual biases, institutional recommendations or cultural understandings (Houseworth and Fisher 2020; Kim et al. 2020; Massing et al. 2019). While a single occupational category is a practical and useful outcome of the coding process, non-trivial disagreement rates among coders attest to the inherent ambiguity of mapping complex work realities onto discrete classification categories (Russ et al. 2023)

Automatic tools have been developed to address some of these limitations. The underlying technology ranges from rule-based like CASCOT (Jones and Elias 2004), machine learning (ML) like OccuCoDe (Schierholz et al. 2018; Simson et al. 2023), SOCcer (Russ et al. 2023), or PROCODE (Chettou et al. 2024), and more recently LLM-based such as SOCbot (Sturgis et al. 2025). These tools are typically designed to classify occupations at the maximum precision level, with aggregation to broader groups applied afterwards as needed. While some approaches code occupations sequentially digit by digit (Safikhani et al. 2023), the desired outcome remains as detailed a code as possible. Performance of the tools is usually assessed in terms of algorithmic quality: the proportion of occupations coded automatically (production rate) and the share of correctly coded cases when compared with human coding (accuracy rate). This framing measured the success of automatic coding based on the power of automation itself: how fast and how accurately a model can replicate human coders.

This logic becomes faulty once automatic coding tools (Castañeda and Becuwe 2025; Peycheva et al. 2022; Schierholz et al. 2018; Sturgis et al. 2025; Tijdens 2014) are embedded directly inside surveys. In a semi-automatic mode, a shortlist of probable codes is offered to respondents, who select the best match or indicate that none applies. In this design, respondents are no longer just reporting their occupation but essentially evaluating algorithmic suggestions. And unlike professional coders they are not familiar with occupational taxonomies and may not consider that offered suggestions are subject to algorithmic errors. Accuracy-based evaluation relies on the correspondence between selected and reference codes, but does not consider the conditions under which that selection is made, in particular how respondents interpret and act on algorithmic suggestions.

2.2. Human–Computer Interaction as a Lens

To capture what accuracy omits, we borrow from research in HCI and human–AI decision-making. Rather than evaluating only algorithms, HCI studies examine how humans behave when interacting with machine suggestions. A common approach is to frame outcomes in terms of a confusion matrix of behaviors: a tabular framework that categorizes how people accept or reject machine input.

As shown in the Figure 2 the cells represent different outcomes: correct acceptance of a correct suggestion, correct rejection of an incorrect suggestion, over-reliance or commission (accepting an incorrect suggestion), and under-reliance or omission (rejecting a correct suggestion) (Beck et al. 2026; Vasconcelos et al. 2023). This framework makes explicit that “rejecting” a suggestion can itself be either appropriate (when the machine is wrong) or erroneous (when the machine is right). It therefore makes visible the very outcomes that accuracy rates treat as missing data.

Figure 2.

Human–ML confusion matrix: classic HCI framework. The framework identifies four outcomes of human–algorithm interaction. White cells indicate successful outcomes of algorithmic output and human behavior (correct acceptance and correct rejection), whereas grey cells indicate error outcomes (incorrect acceptance and incorrect rejection).

At the same time, applying HCI confusion matrix to occupational coding reveals its incompleteness. When algorithms present a shortlist of several possible codes, respondents may reject the correct suggestion and instead choose an incorrect alternative. This situation is not captured in the four standard HCI categories. We therefore add a fifth outcome (Misclassification) to COARSE framework (see Figure 1) to reflect the possibility that respondents select an incorrect but machine-provided code even though the correct one was available.

2.3. Insights from Survey Methodology

While HCI highlights the formal structure of reliance patterns, errors are often investigated with a focus on automation bias: a systematic tendency to over-rely on the output of machines and AI systems, even in the face of obvious mistakes (Beck et al. 2026; Goddard et al. 2012; Romeo and Conti 2025). In survey settings, however, respondents may not even realize that an algorithm is involved, making automation bias an incomplete explanation. Instead, survey research investigates a broader set of mechanisms that shape why respondents choose one answer over another.

2.3.1. Cognitive Shortcuts

Respondents generally follow the survey response process of comprehending a question, retrieving relevant information, forming a judgment, and reporting it, but they do so with varying levels of effort and accuracy (Tourangeau 2018; Tourangeau et al. 2000). Instead, when tasks are difficult or motivation is low, they may engage in satisficing behavior by giving “good enough” answers (Krosnick 1991). Weak satisficing can manifest as acquiescence (agreeing regardless of content) or as order effects such as primacy and recency (choosing the first or last option presented) (Krosnick et al. 2001; Vannette and Krosnick 2014). In the context of occupational coding, these shortcuts can possibly explain why respondents choose a wrong suggestion even if the correct one is present.

2.3.2. Residual Categories

Debates about the “other/none applies” response option are especially relevant. Unlike “don’t know,” selecting “other” can signal active processing and resistance to misclassification (Miller and Lambert 2014; Wells et al. 2012). Yet in occupational coding, responses of “other” run counter to the goal of automation, since they still require costly post-survey processing (Schierholz et al. 2018). From the perspective of HCI, these answers may reflect the well-judged outcome of appropriate rejection—recognizing that none of the suggested codes fits. At the same time, “other” can also result from satisficing or confusion, when respondents mistakenly reject a correct suggestion.

2.3.3. Pseudo-Opinions

Finally, studies on uninformed answers and pseudo-opinions show how respondents often provide responses even when they lack relevant knowledge (Bishop et al. 1986; Graeff 2002; Schwarz 2014; Sturgis and Smith 2010). Pressure to answer and conversational norms lead people to endorse fictitious items or map superficial cues onto options. In occupational coding, the same tendencies may explain why some respondents accept algorithmic suggestions that are incorrect.

Taken together, these strands of survey methodology research suggest that respondents’ cognitive shortcuts, interpretations of residual categories, and tendencies to offer pseudo-opinions can shape the distribution of outcomes when interacting with machine-generated suggestions.

2.4. Hypothesis

Building on this synthesis, we ask: How do respondents interact with a ML tool embedded into surveys, and how can a COARSE framework map the outcomes of such interactions to identify data quality risks not captured by accuracy or production metrics?

Hypothesis 1: Pseudo-opinion formation under ambiguity. When no correct answer exists among the ML suggestions, respondents will predominantly select an offered suggestion and make a commission error (CE) rather than acknowledge absence through “other” (appropriate rejection [AR]).

Hypothesis 2: Calibrated performance with correct suggestions. When the correct code is present, accurate classification (AC) will be the dominant outcome, significantly exceeding both misclassification errors and omission errors.

Hypothesis 3: Misclassification (ME) as active misjudgment. Even when the correct code is available, some respondents can misclassify if they misjudge subtle occupational distinctions during answer options evaluation. These errors will not appear random; rather, they will cluster around occupations similar to the correct code.

Hypothesis 4: Omission (OE) versus Appropriate Rejection (AR)—Different Cognitive Processes. We hypothesize that since omission errors happen due to cognitive burden and disengagement, while appropriate rejections reflect deliberate processing, we expect respondents to report higher task difficulty for OE outcomes than for AR outcomes.

3. Data and Methods

3.1. Data

The data used to test these hypotheses come from respondents’ interactions with OccuCoDe module embedded in the representative monthly multi-topic computer-assisted telephonic interview (CATI) survey, conducted in Germany between April 1 and June 30, 2019 by infas (Institute for Applied Social Science). The survey interviewed adults aged 18 and older via landline and mobile phones, with a sample split of 30% panel participants and 70% newly recruited respondents. The sample was 53% male, mean age 57; 66% classified their current job and 34% their most recent. The survey was conducted by 135 trained interviewers, however 51% of interviewers conducted five or less interviews, and 19% conducted only one interview.

Of 1,415 respondents, 1,379 were employed (currently or previously) and interacted with OccuCoDe. During the interview, the interviewer switched to the OccuCoDe interface within the CATI script and asked occupational questions in the sequence described in Section 3.2, which included real-time ML suggestion filtering based on respondents’ answers. Interviewers administered the CATI script and were not tasked with classifying respondents’ occupations themselves (see an example of a transcribed interview in Supplemental Appendix A.1).

After the fieldwork was completed, infas independently classified occupations using their in-house professional coders: roughly 50% via a dictionary-based electronic system and 50% manually. All cases received a second blind coding, with discrepancies resolved individually. infas coders did not have access to OccuCoDe suggestions but had access to auxiliary variables (e.g., education, industry). These manually coded results serve as the correct code—what we call here a “gold code”—for comparing OccuCoDe’s performance.

3.2. Occupation Classification Software

The OccuCoDe instrument (Schierholz et al. 2018; Simson et al. 2023) was embedded directly in the part of the survey that asked occupation-related questions. In this part, the interviewer starts by asking an open-ended question: “What is the occupational task that you mainly perform?” For those not currently employed, it refers to their previous job (Figure 3A).

Figure 3.

Overview of the OccuCoDe interface and functionality: (A) Open-ended question collecting the respondent’s occupation.(B) Suggested categories generated from (A); respondent selects the best match, (C) Example of a follow-up question, and (D) Open-ended clarification question shown when the respondent selects “I do something else” in (B).

After the answer is received, two backend processes occur. The ML algorithm (similarity-based reasoning [Simson et al. 2023]) assigns probabilities to each KldB 2010 code and filters up to five most likely matches. Instead of official occupational names or descriptions taken directly from classification taxonomies, Schierholz et al. (2018) developed AuxCo, mapping every KldB 2010 and ISCO-08 code to user-friendly descriptions. Respondents then get a multiple-choice question with up to five options: “We will now try to categorize your profession more precisely. Which one of the following descriptions best describes your profession? If several apply, please think of the task you mainly carry out.” An “I do something else” option is offered as well (Figure 3B).

Selecting an option assigns the corresponding 5-digit KldB 2010 and 4-digit ISCO-08 code. If more detail is needed (e.g., supervisory duties or educational background), up to two follow-up multiple-choice questions are asked (Figure 3C). If “I do something else” is chosen, no automatic code is assigned; manual post-survey coding is required, and respondents are asked to clarify job details (Figure 3D).

3.3. Measures

Our analysis requires three sets of variables: the ones that describe algorithmic output; the ones that describe respondent behavior; and the ones that estimate selection errors. First, we record whether OccuCoDe suggestions offered to the respondent included the gold code (the occupation assigned by professional coders at infas, serving as the ground truth) or not. It includes both the initial recommendations offered to the respondent (Figure 3B) and all potential follow-ups (Figure 3C). In addition we use OccuCoDe confidence score (0–1) as a proxy for calculating the prediction quality of the best suggestion.

Second, we use five COARSE outcomes to categorize respondent behavior when facing OccuCoDe suggestions: Accurate Classification (AC), where the respondent accepts the gold code suggestion; Misclassification Error (ME), where the respondent selects an incorrect suggestion although the gold code is available; Omission Error (OE), where the respondent rejects all suggestions despite the gold code being offered; Commission Error (CE), where the respondent accepts a suggestion when no gold code was available; and Appropriate Rejection (AR), where the respondent rejects all suggestions when no gold code was offered. We also record whether the respondent reported the classification task as difficult (task difficulty, 1/0), asked immediately after the OccuCoDe interaction.

Third, for respondents that made incorrect choices (ME, CE) outcomes we measure how close their chosen code was to the gold code using the length of the shared leading-digit prefix in the KldB 2010 hierarchy (0–5 digits). To assess whether misclassifications cluster near the gold code beyond chance, we compare observed high-depth matches against a random-choice benchmark. For each respondent, the random-choice probability is the proportion of OccuCoDe suggestions shown that share at least three leading digits with the gold code: $p_{high} = ({offered}_{3} + {offered}_{4} + {offered}_{5}) / n_{offered}$ , where ${offered}_{d}$ counts suggestions matching the gold code at depth $d$ (digit prefix length), and $n_{offered}$ is the total number of suggestions presented. For the regression analysis we use additional characteristics of the suggestions: length (number of words), reading complexity, respondents’ perceived accuracy of their selected choice relative to their job, and respondent and interviewer demographics. The full set of variables is available in Supplemental Appendix Table 1A.

3.4. Analysis

We test our four hypotheses (H1–H4) using a mix of exact tests (binomial, Poisson binomial, Fisher test) and regressions to model the five COARSE outcomes (Accurate Classification [AC], Misclassification Error [ME], Omission Error [OE], Commission Error [CE], Appropriate Rejection [AR]).

To test our first hypothesis that when no correct answer, or gold code, is present CE outcomes exceed AR outcomes, we first use one-sided exact binomial tests comparing: $H_{0} : p_{CE} \leq 0.50$ versus $H_{1} : p_{CE} > 0.50$ , $α = 0.05$ (one-sided). Second, we identify whether CE choices were random or concentrated around near correct answers. For each respondent we create a closeness indicator $y_{i} \in {0, 1}$ that measures whether the selection equals the gold code at depth 3 to 4. Since each respondent received a variable number of suggestions (1–5) with varying composition of near-correct and exact matches, each respondent’s null probability $p_{i}$ is calculated as the proportion of their non-gold offered options falling at depth 3 to 4 among all options offered. Under the null hypothesis that respondents pick one of the answer options at random, respondents’ selections can be described as independent Bernoulli trials, $Y_{i} ~ Bernoulli (p_{i})$ . Consequently, the random variable $\sum Y_{i}$ that describes the total number of close selections has a Poisson binomial distribution with expected value $E (\sum Y_{i}) = \sum p_{i}$ . If observed and expected totals diverge, $\sum y_{i} \approx / \sum p_{i}$ , this is evidence against random selection. We reject $H_{0}$ if the probability for even more extreme outcomes than observed, $p = P (\sum Y_{i} > \sum y_{i}) < 0.05 = α$ , is small. In addition we estimate a logistic regression model with CE(1)/AR(0) as the dependent variable. We evaluate how question characteristics (OccuCoDe confidence score, proximity of the OccuCoDe suggestions, total length, reading complexity), cognitive burden (task difficulty), and demographics (respondent sex, age, education, interviewer sex and experience) systematically vary for CE/AR outcomes. Due to multicollinearity we aggregate interviewer sex and experience variables.

To test our second hypothesis that when the gold code is present among the ML suggestions AC will be the dominant outcome we use one-sided exact binomial tests: $H_{0} : p_{AC} \leq 0.50$ versus $H_{1} : p_{AC} > 0.50$ , $α = 0.05$ (one-sided). In addition, we estimate a multinomial regression model with AC, ME, and OE as dependent outcomes. The model includes the same predictors as in the test of H1, with the exception of proximity of the OccuCoDe suggestions.

To test our third hypothesis that misclassification errors occur in close proximity to the gold code rather than at random, we apply the random-choice benchmark test described above in testing H1. This analysis is only for respondents in ME outcomes where the gold code appeared in the initial suggestion list (although respondents could encounter the gold code either directly in the initial suggestions or in follow-ups, we limit our test to the 201 out of 254 cases where the gold code was directly offered among the initial suggestions but the respondent still chose one of the incorrect answers).

To test our fourth hypothesis that perceived task difficulty will be higher for OE than AR outcomes, we use Fisher’s exact tests on the proportion of respondents reporting difficulty: AR versus OE: $H_{0} : p_{diff, OE} = p_{diff, AR}$ versus $H_{1} : p_{diff, OE} > p_{diff, AR}$ ; $α = 0.05$ (one-sided).

4. Results

4.1. Descriptive Statistics

Of 1,379 respondents who interacted with OccuCoDe, 150 were excluded (121 received no suggestions due to generic or misspelled inputs, six refused to answer, eight did not complete follow-ups, and fifteen were not fully codable by professional coders), leaving a final sample of 1,229. A total of ninety-nine occupations were reported more than once, with “Teacher” mentioned forty-six times and “Nurse” and “Secretary” thirteen times each.

OccuCoDe suggestion quality varied: lower confidence scores were mostly associated with cases where no correct answer was available (CE and AR outcomes; see Figure 4). If OccuCoDe were used in fully automatic mode, assigning the most likely occupational code as the final one without respondent input, 41% of occupations would be coded correctly. Adding the interactive component with the respondent in the loop resulted in 47% correct responses among those who selected an offered option (excluding respondents who chose “I do something else”) a rate typical for automatic coding tools (Schierholz et al. 2018; Wan et al. 2023)

Figure 4.

OccuCoDe confidence score by outcome.

Table 1 provides further nuances that become visible under the COARSE lens. In 64% of cases, the correct code appeared among the suggestions and respondents had a chance to select it; in the remaining 36%, it was absent entirely. Among all cases where respondents did not choose the gold code, 56% were due to the ML not offering it, while 44% were based on respondents’ own misjudgments. The nature of respondents’ jobs also seemed to matter, as some major occupational groups appeared more often in specific outcomes (see Table 2A in the Supplemental Appendix).

Table 1.

Descriptive Statistics Based on COARSE Outcomes.

Category	Gold code present			Gold code absent
Category	Accurate classification (AC); n = 447	Misclassification error (ME); n = 254	Omission error (OE); n = 90	Commission error (CE); n = 242	Appropriate rejection (AR); n = 196
Descriptions
Share of all interviews	36	21	7	20	16
Share of cases (gold code present/absent)*	57	32	11	55	45
Average ML confidence score	0.59	0.54	0.54	0.39	0.36
Reading complexity	9.2	9.0	9.0	9.0	9.0
Total length (number of words)	44	44	44	47	46
Proximity of the OccuCoDe suggestions to the gold code
ML offered perfect match (5-digit**)	88	80	100	0	0
ML offered close match (4-digit)	11	15	0	23	13
ML offered partial match (3-digit)	2	5	0	25	17
ML offered broad match (2-digit)	0	0	0	16	12
ML offered very broad match (1-digit)	0	0	0	14	11
ML offered no match (0-digit)	0	0	0	21	47
Proximity of selected answer to the gold code
Resp. chose a perfect match (5-digit)	100	0	—	0	—
Resp. chose a close match (4-digit)	0	39	—	7	—
Resp. chose a partial match (3-digit)	0	32	—	26	—
Resp. chose a broad match (2-digit)	0	9	—	17	—
Resp. chose a very broad match (1-digit)	0	10	—	21	—
Resp. chose a no match (0-digit)	0	9	—	29	—
Task difficulty
Resp. coding task perceived as difficult	12	27	60	34	48
Perceived accuracy
Resp. selected answer fit job they do	79	60	—	59	—

Note. Means reported for Average ML score; reading complexity, and total length. All other are percentages.

Proportions of AC + ME + OE = 100% and CE + AR = 100%.

Gold code refers to the correct label, which could appear in either initial or follow-up codes. Proximity values are based only on initial (default) codes offered to respondents (see Figure 3B). A value less than 100% in AC and ME outcomes indicates that some gold codes were available only through follow-ups.

4.2. H1: Pseudo-Opinion Formation Under Ambiguity

Consistent with the pseudo-opinion literature, when the gold code was not among ML suggestions, 55% of respondents made commission errors (CE). This outcome was statistically significant (one-sided exact binomial test: 242/438 cases, 55.3%, p = .02, h = 0.11), supporting H1. Although the effect size relative to the 50% threshold appears small, commission errors carry a substantive benchmark of zero, as they represent undesirable outcomes that should, in an ideal instrument, not occur. Finding CE to be a very frequent response when no correct option is available is therefore practically meaningful regardless of the margin. Further, CE choices were not random. Respondents picked 3 to 4-digit (observed) matches far more often than expected under a random-choice benchmark (80 observed vs. 63 expected; $p < . 0001$ , effect size 1.3) and 0 to 1 digits matches significantly less often than expected (observed 122, expected 144, $p < . 0001$ , effect size 0.84).

Logistic regression analysis (Table 3A in the Supplemental Appendix) provides further insight into this pattern. Higher proximity of OccuCoDe suggestions is associated with a higher likelihood of CE (OR = 1.441, $p < . 001$ ), while AR outcome is associated with greater perceived difficulty (OR = 0.533, $p = . 004$ ). These findings suggest that respondents are actively engaged with the task but cannot serve as reliable gatekeepers against poor ML suggestions. By default they attempt to map their occupational knowledge onto categories even when the correct one is absent, with CE behavior more likely when suggestions approximate the gold code. The correct rejection behavior (AR) is instead associated with an increased burden and lack of good suggestions.

4.3. H2: Calibration When Gold Is Present

When the gold code was present, one-sided exact binomial tests showed that accurate classification (AC) was significantly more likely than the combined alternatives ME + OE (447/791 cases, 57%, $p = . 0001$ , h = 0.13), thus supporting H2. Additional regression analysis (Table 4A in the Supplemental Appendix) showed that lower ML confidence scores were marginally associated with ME outcomes (RRR = 0.538, $p = . 066$ ), suggesting that algorithmic uncertainty affects respondent decision-making. Higher reading complexity was associated with a higher likelihood of OE outcomes (RRR = 1.380, $p = . 013$ ), and perceived task difficulty with both ME and OE outcomes, though the effect was much stronger for OE (RRR = 10.814, $p < . 001$ ). These patterns suggest that respondents generally recognize the correct option when it is available, yet their responses display distinct tendencies under different conditions. When machine learning outputs were marked by low confidence, misclassifications were more frequent. By contrast, when the task was experienced as cognitively demanding, omission responses were more common.

4.4. H3: Misclassification as Active Misjudgment

When the gold code was present, most misclassifications occurred at the more detailed levels of the KldB classification (see Table 1). In 39% of ME cases, the chosen occupation differed from the gold code only in the final digit (4-digit match), and in another 32% the difference was in the last two digits (3-digit match). As hypothesized (H3), these choices were not random: respondents selected 3 to 4-digit matches significantly more often than expected under a random-choice benchmark (128 observed vs. 100 expected; $p < . 0001$ ). The fact that most misclassifications occur at fine-grained levels, and frequently in closely related occupations, points to a possible design issue: future coding instruments should scrutinize how the phrasing of response options and gold codes shapes respondents’ choices.

4.5. H4: Omission Versus Appropriate Rejection—Different Cognitive Processes

While OE and AR outcomes both involve selecting “other,” they arise from distinct contexts. OE occurs when the correct code is present among suggestions (average ML confidence = 0.54), but respondents face high cognitive demands from task difficulty and complex language. AR occurs when no correct code is available; with 48% of options sharing no digits with the correct code (average ML confidence = 0.36), this outcome is also associated with post hoc reported task difficulty. A direct comparison still shows the distinction: OE respondents were significantly more likely to report task difficulty than AR respondents (53 of 89, 60% vs. 92 of 193 48%; one-sided Fisher’s exact test, $p = . 042$ ), supporting H4 that these outcomes reflect different cognitive processes. However, further research is needed to examine these differences in more depth. Four respondents who did not answer the task difficulty question were excluded from this comparison.

5. Conclusion

Embedding automatic occupational coding tools directly into surveys transforms how data quality must be understood. Rather than compensating for machine mistakes, respondents often compound them by accepting inadequate suggestions when they should reject them, or overlooking correct options under cognitive burden. This means that data quality risks are not just technical, or associated with algorithmic bias, but behavioral, arising from the interaction between tool design and respondent decision-making. The COARSE framework therefore provides not only a diagnostic tool for identifying these risks but also a call to rethink how survey instruments are designed when automatic coding becomes part of the interview itself.

Not all respondents’ errors are equal, and they signal distinct challenges for survey design. When the correct answer is not among the ML suggestions offered, respondents often provide “good enough” answers rather than rejecting them, which is consistent with pseudo-opinion research. When the correct answer is actually present, respondents typically identify it, especially when ML confidence is high. Misclassification errors highlight cases of active misjudgment, where close but incorrect codes are selected. Although omission errors and appropriate rejection both involve not accepting the suggestions offered and choosing “other,” they reflect different cognitive processes, with omission being linked to greater burden. Errors driven by misjudgment point to the need for clearer interfaces and guidance, while omission errors reflect the cognitive costs of processing long or complex lists.

These patterns carry several practical implications for tool and survey design. On quality of the ML answer options: offering respondents low-quality suggestions (a strategy that aims to increase the production rate) would likely create no gold code scenarios and contribute to the commission errors, as we cannot rely on respondents to predictably reject such options. Future research should either use recommended ML confidence score thresholds (for OccuCode, this is 0.535 [Simson et al. 2023], though unused in our analysis) or work toward ensuring higher quality input from respondents on which ML bases suggestions. This could be achieved by modifying the initial occupation inquiry question (Kononykhina et al. 2025) or introducing interactive exchanges supported by LLMs to generate a codable job title (Sturgis et al. 2025).

On interface and question design: The way the question was formulated likely encouraged respondents to select an option rather than choose “other,” which aligns with typical survey methodology expectations. However, the atypically long answer options, the lack of upfront information about how many options would be read aloud, and the absence of clear cues that “other” could be correct possibly increased the burden on respondents and contributed to errors. We showed that the complex language of some answer options were associated with omission error outcome, thus OccuCoDe could benefit from linguistic revisions to simplify answer options. In addition, ML suggestions were offered according to likelihood of matching the job title, with the first option always being the most probable match. This introduced a significant primacy effect (between 34% and 77% of respondents opted for the first option). This shows that model predictions and interface design interact in ways that shift how respondents make choices. Future research could investigate whether reducing the number of suggestions from five to three or four options could preserve accuracy while shortening interviews and reducing cognitive load.

On interviewer training: Previous research (Schierholz et al. 2018) showed that interviewers frequently failed to read aloud the “other” option, further straining respondents to select a category. Interviewer training should emphasize the imperfect nature of ML suggestions and the importance of offering the “other” option. A separate line of inquiry could examine the effect of not presenting “other” as an explicit option, instead leaving it to interviewers to use when respondents struggle. While this approach might reduce omission errors, it would also introduce additional interviewer effect that needs to be quantified before such a change can be recommended.

Our study has several limitations. First, the COARSE framework is built on the commonly utilized idea that there exists a correct occupational category (gold code) and that it is known from human codings. The reliability and quality of human coding, though, has been called into question (Kim et al. 2020; Massing et al. 2019). The universal assumption underlying occupation coding that a single category must always be correct is not self-evident; instead, multiple categories could be appropriate. With this in mind, the COARSE framework may overestimate the size of the various errors. For example, agreement between coders in our data reveals variation across COARSE outcomes: it was highest for cases where the gold code was present—AC (91%), OE (84%), and ME (79%), and lower for AR (62%) and CE (57%). The relatively low agreement of CE and AR outcomes suggests that COARSE classifications potentially conflates coding ambiguity with respondent behavior. To ensure such results are properly interpreted, future work should systematically examine the typical errors of human coders and establish the likely bounds of error against which COARSE can be benchmarked. Second, while interviewer effects (Ongena 2005; Schnell and Kreuter 2005; Stefkovics et al. 2025; West and Blom 2017) are known to affect survey data quality, we could not isolate this influence due to the sparse number of interviews per interviewer within each COARSE outcome category. Third, the COARSE framework has been applied to the data from CATI-based survey. Testing the framework in self-administered survey contexts, where respondents interact with the algorithm directly, would contribute to the generalizability of our findings. Forth, our findings on primacy effects cannot distinguish between two plausible explanations: respondents may select the first option because the algorithm ranks the best match first (algorithmic quality), or response-order effects are likely to influence what respondents select, augmenting or attenuating the likelihood that the first option gets selected. Since suggestions are always sorted by probability in OccuCoDe, these two mechanisms are fully confounded in our design, and it will require new data collection to isolate the effects. Fifth, since reporting one’s occupation is a known subject of aspirational reporting, reflecting the perceived prestige or desired roles in the job (ILO 1949), future research should examine whether and how ML suggested occupational categories may contribute to such bias. Sixth, while our study accounts for basic respondent satisfaction with the interactive occupational coding process and outcomes, future research should examine in depth the cognitive and behavioral cues in human-machine interactions that could improve options design and interviewer training.

Our study raises concerns that human–AI collaboration in survey settings will become a natural addition to the survey process. The COARSE framework reveals that data quality risks in automated coding systems extend beyond algorithmic accuracy and are deeply embedded in human cognitive processes and the dynamics of survey interaction itself. This finding has significant implications for the growing integration of AI systems in survey research. The path forward requires designing systems that account for predictable human responses to automated suggestions. Our framework provides the foundation for this more nuanced approach, moving the field beyond simple production and accuracy metrics toward a deeper understanding of how humans and machines can (or cannot) effectively collaborate in the pursuit of high-quality data.

Supplemental Material

sj-docx-1-jof-10.1177_0282423X261455656 – Supplemental material for Beyond Algorithmic Accuracy: Understanding Data Quality in Interactive Occupational Coding with the COARSE Framework

Supplemental material, sj-docx-1-jof-10.1177_0282423X261455656 for Beyond Algorithmic Accuracy: Understanding Data Quality in Interactive Occupational Coding with the COARSE Framework by Olga Kononykhina, Malte Schierholz and Frauke Kreuter in Journal of Official Statistics

Footnotes

Acknowledgements

The work was done [in part] while one of the authors (Frauke Kreuter) was visiting the Simons Institute for the Theory of Computing. We thank Jacob Beck and Mariel McKone Leonard for valuable comments, and Regina List for proofreading and editing. We used Open AI GPT-5 for R code suggestions, formatting Overleaf tables and text edits; The abbreviation COARSE framework was also suggested by OpenAIãs GPT-5. We used Elicit, Asta and Connected Papers to identify and collect relevant papers for the literature review.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This paper is written with the support of the DFG grant 290773872.

ORCID iDs

Olga Kononykhina

Frauke Kreuter

Supplemental Material

Supplemental material for this article is available online.

Received: September 29, 2025

Accepted: May 15, 2026

References

Bao

Baker

C. J. O.

Adisesh

2020. “Occupation Coding of Job Titles: Iterative Development of an Automated Coding Algorithm for the Canadian National Occupation Classification (ACA-NOC).” JMIR Formative Research 4 (8): e16422. DOI: https://doi.org/10.2196/16422.

Beck

Eckman

Kern

Kreuter

2026. “Bias in the Loop: How Humans Evaluate AI-Generated Suggestions.” Harvard Data Science Review. Published electronically March 12, 2026. https://hdsr.mitpress.mit.edu/pub/nrcn4h7d.

Bishop

G. F.

Tuchfarber

A. J.

Oldendick

R. W.

1986. “Opinions on Fictitious Issues: The Pressure to Answer Survey Questions.” Public Opinion Quarterly 50 (2): 240–50. DOI: https://doi.org/10.1086/268978.

Castañeda

Becuwe

2025. “Optimising Occupation and Industry Data Collection in Cross-National Surveys.” Presentation at the 10th Conference of the European Survey Research Association (ESRA 2025), Utrecht, The Netherlands.

Chettou

Charreau

Schaufelberger

Faivre

Canu

I. Guseva

. 2024. “P-036 PROCODE: A Web Application for Coding and Recoding Occupations and Economic Activities.” Occupational Medicine 74 (Suppl. 1): 485. DOI: https://doi.org/10.1093/occmed/kqae023.0485.

Christoph

Matthes

Ebner

2020. “Occupation-Based Measures—An Overview and Discussion.” Kölner Zeitschrift für Soziologie und Sozialpsychologie 72 (Suppl. 1): 41–78. DOI: https://doi.org/10.1007/s11577-020-00673-4.

Goddard

Roudsari

Wyatt

J. C.

2012. “Automation Bias: A Systematic Review of Frequency, Effect Mediators, and Mitigators.” Journal of the American Medical Informatics Association 19 (1): 121–7. DOI: https://doi.org/10.1136/amiajnl-2011-000089.

Graeff

T. R.

2002. “Uninformed Response Bias in Telephone Surveys.” Journal of Business Research 55 (3): 251–9. DOI: https://doi.org/10.1016/S0148-2963(00)00138-7.

Houseworth

Fisher

2020. “Measurement Error in Occupation and the Impact on Intergenerational Mobility.” Research in Social Stratification and Mobility 70: 100469. DOI: https://doi.org/10.1016/j.rssm.2019.100469.

10.

ILO. 1949. “International Standard Classification of Occupations: Report Prepared for the Seventh International Conference of Labour Statisticians.” Geneva Series Number: 15 Series: Studies and Reports, New Series, International Labour Office, Geneva, Switzerland.

11.

ILO. 2012. “International Standard Classification of Occupations: ISCO-08 Volume 1: Structure, Group Definitions and Correspondence Tables.” Technical Report, International Labour Office, Geneva, Switzerland.

12.

Jones

Elias

2004. CASCOT: Computer-Assisted Structured Coding Tool. Warwick Institute for Employment Research, University of Warwick.

13.

Kim

Ban

2020. “Do You Know What You Do for a Living? Occupational Coding Mismatches Between Coders in the Korean General Social Survey.” Research in Social Stratification and Mobility 70: 100467. DOI: https://doi.org/10.1016/j.rssm.2019.100467.

14.

Kononykhina

Kreuter

Schierholz

2025. The Impact of Question Framing on the Performance of Automatic Occupation Coding. ArXiv:2501.05584. DOI: https://doi.org/10.48550/arXiv.2501.05584.

15.

Krosnick

J. A.

1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5 (3): 213–36. DOI: https://doi.org/10.1002/acp.2350050305.

16.

Krosnick

J. A.

Holbrook

A. L.

Berent

M. K.

, et al. 2001. “The Impact of ‘No Opinion’ Response Options on Data Quality.” Public Opinion Quarterly 66 (3): 371–403. DOI: https://doi.org/10.1086/341394.

17.

Massing

Wasmer

Wolf

Züll

2019. “How Standardized Is Occupational Coding? A Comparison of Results from Different Coding Agencies in Germany.” Journal of Official Statistics 35 (1): 167–87. DOI: https://doi.org/10.2478/jos-2019-0008.

18.

Miller

A. L.

Lambert

A. D.

2014. “Open-Ended Survey Questions: Item Nonresponse Nightmare or Qualitative Data Dream?” Survey Practice 7 (5): 1–11. DOI: https://doi.org/10.29115/SP-2014-0024.

19.

Ongena

2005. “Interviewer and Respondent Interaction in Survey Interviews.” PhD thesis, Research External, Graduation External, Vrije Universiteit, Amsterdam, The Netherlands.

20.

Peycheva

D. N.

Sakshaug

J. W.

Calderwood

2022. “Occupation Coding During the Interview in a Web-First Sequential Mixed-Mode Survey.” Journal of Official Statistics 37 (4): 981–1007. DOI: https://doi.org/10.2478/jos-2021-0042.

21.

Romeo

Conti

2025. “Exploring Automation Bias in Human–AI Collaboration: A Review and Implications for Explainable AI.” AI & SOCIETY 41 (1): 259–78. DOI: https://doi.org/10.1007/s00146-025-02422-7.

22.

Russ

D. E.

Josse

Remen

, et al. 2023. “Evaluation of the Updated SOCcer V2 Algorithm for Coding Free-Text Job Descriptions in Three Epidemiologic Studies.” Annals of Work Exposures and Health 67 (6): 772–83. DOI: https://doi.org/10.1093/annweh/wxad020.

23.

Safikhani

Avetisyan

Föste-Eggers

Broneske

2023. “Automated Occupation Coding with Hierarchical Features: A Data-Centric Approach to Classification with Pre-Trained Language Models.” Discover Artificial Intelligence 3 (1): 1–6. DOI: https://doi.org/10.1007/s44163-023-00050-y.

24.

Schierholz

Gensicke

Tschersich

Kreuter

2018. “Occupation Coding During the Interview.” Journal of the Royal Statistical Society Series A: Statistics in Society 181 (2): 379–407. DOI: https://doi.org/10.1111/rssa.12297.

25.

Schnell

Kreuter

2005. “Separating Interviewer and Sampling-Point Effects.” Journal of Official Statistics 21 (3): 389–410.

26.

Schwarz

2014. Cognition and Communication: Judgmental Biases, Research Methods, and the Logic of Conversation. Psychology Press.

27.

Simson

Kononykhina

Schierholz

2023. “occupationMeasurement: A Comprehensive Toolbox for Interactive Occupation Coding in Surveys.” Journal of Open Source Software 8 (88): 5505. DOI: https://doi.org/10.21105/joss.05505.

28.

Stefkovics

Batiz

Zsófia Grubits

Sára Ligeti

2025. “What Types of Survey Questions Are Prone to Interviewer Effects? Evidence Based on 29,000 Intra-Interviewer Correlations from 28 Countries of the European Social Survey.” Sociological Methods & Research. Published electronically September 9, 2025. DOI: https://doi.org/10.1177/00491241251372509.

29.

Sturgis

Robinson

Fung

Roberts

2025. SOCbot: Using Large Language Models to Measure and Classify Occupations in Surveys. ESRC Survey Futures Programme.

30.

Sturgis

Smith

2010. “Fictitious Issues Revisited: Political Interest, Knowledge and the Generation of Nonattitudes.” Political Studies 58 (1): 66–84. DOI: https://doi.org/10.1111/j.1467-9248.2008.00773.x.

31.

Tijdens

2014. “Dropout Rates and Response Times of an Occupation Search Tree in a Web Survey.” Journal of Official Statistics 30 (1): 23–43. DOI: https://doi.org/10.2478/jos-2014-0002.

32.

Tourangeau

2018. “The Survey Response Process from a Cognitive Viewpoint.” Quality Assurance in Education 26 (2): 169–81. DOI: https://doi.org/10.1108/QAE-06-2017-0034.

33.

Tourangeau

Rips

L. J.

Rasinski

2000. The Psychology of Survey Response. Cambridge University Press.

34.

Vannette

D. L.

Krosnick

J. A.

2014. “Answering Questions: A Comparison of Survey Satisficing and Mindlessness.” In The Wiley Blackwell Handbook of Mindfulness, edited by Le

Ngnoumen

C. T.

Langer

E. J.

John Wiley & Sons.

35.

Vasconcelos

Jörke

Grunde-McLaughlin

Gerstenberg

Bernstein

M. S.

Krishna

2023. “Explanations Can Reduce Overreliance on AI Systems During Decision-Making.” Proceedings of the ACM on Human-Computer Interaction 7 (CSCW1): 129:1–129:38. DOI: https://doi.org/10.1145/3579605.

36.

Wan

C. B.

Friesen

M. C.

, et al. 2023. “Automated Coding of Job Descriptions from a General Population Study: Overview of Existing Tools, Their Application and Comparison.” Annals of Work Exposures and Health 67 (5): 663–72. DOI: https://doi.org/10.1093/annweh/wxad002.

37.

Wells

Rao

Link

M. W.

Pierce

2012. “Flagging Speeders in a Multi-Mode (Mobile and Online) Survey.”Proceedings of the 2012 Annual Meeting of the Midwest Association for Public Opinion Research, Chicago, IL, USA.

38.

West

B. T.

Blom

A. G.

2017. “Explaining Interviewer Effects: A Research Synthesis.” Journal of Survey Statistics and Methodology 5 (2): 175–211. DOI: https://doi.org/10.1093/jssam/smw024.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.04 MB