Sage Journals: Discover world-class research

Abstract

Workers in many vocations rely on a wide range of information to solve problems, including Dutch municipal enforcement officials (MEOs) who require this skill when planning surveillance routes. The present study focused on the development of a digital simulation game intended for assessing domain-specific information problem solving (IPS) in aspiring MEOs. Using a think-aloud method with five final-year MEO students, this study explored how well the IPS model—typically used in academic contexts—applied to planning surveillance routes. The findings suggest that the IPS model's constituent skills align with the process of planning a surveillance route, although differences were found in the underlying activities. The insights resulted in an adjusted IPS model for solving information problems in this vocational context. The adjusted IPS model could extend the simulation game to additional vocational contexts or even a generic game-based assessment of vocational IPS. Further investigation is needed to assess its generalisability.

Keywords

information problem solving vocational education educational design research simulation game

Workers in many vocations rely on a wide range of information to determine the best course of action. With the rapid development of information and communication technologies in the recent decades, the amount of available information is constantly increasing. This is also true for Dutch municipal enforcement officials (MEOs), that is, those responsible for the upkeep of public safety and security within a municipality. MEOs have access to a growing amount of information about their municipality, which they use to carry out data-driven surveillance of a designated area. Surveillance is one of the core tasks of an MEO: therefore, it is not surprising that the ability to plan a surveillance route is one of the certification requirements for an aspiring MEO (Samenwerkingsorganisatie Beroepsonderwijs Bedrijfsleven, 2023). In this plan, data are essential for determining the best course of action to catch irregularities in the public domain. Given that a surveillance plan demonstrates the capacity to combine and process information and translate it into a series of actions, it can be seen as a product required for solving an information problem.

Solving information problems requires two twenty-first-century skills: information literacy and problem solving (Brand-Gruwel et al., 2005; van Merriënboer & Kirschner, 2018). Information literacy refers to the ability to recognise a need for information and subsequently locate, critically assess, and effectively use the information. Successful information problem solving (IPS) requires the ability to work with large quantities of information to determine what information is relevant and combine and integrate it into a fitting solution. The components of any problem are described as (1) the current situation, (2) the goal situation and (3) the challenges in progressing from the first to the second (Mayer, 1992, as cited in Funke, 2010). By engaging in IPS, a person moves from an information need to an end product in which the required information is adequately synthesised.

IPS is a domain-general skill, meaning that it is applied across various domains (van Merriënboer & Kirschner, 2018). Moreover, van Merriënboer and Kirschner (2018) state that despite its generic nature, a domain-specific context is needed for someone to engage in IPS. Therefore, to assess the IPS skills of an (aspiring) MEO, the individual should be exposed to authentic information problems that allow them to demonstrate their skills (Gulikers et al., 2004; Gulikers et al., 2006).

Currently, the exams to qualify for an MEO certificate are administered in an exam street: a simulated street with public facilities found within a municipality (e.g., parking spots, a crosswalk, the entrance to a shopping centre, a bus stop, etc.; see Figure 1). These facilities are used to act out scenarios that MEOs can encounter during their work. For standardised assessments of sanctioning skills (i.e., through role-playing), the exam street is a suitable environment. However, several factors in the exam street raise the concern that the assessment validity of a students’ IPS skills cannot be guaranteed.

Figure 1.
Layout of an Exam Street for the Assessment of an Aspiring MEO.

First, the exam street provides a limited space for students to plan a surveillance route, particularly when compared with the size of a municipality. As a result, the plan would have limited representativeness compared with the real working environment. Second, the exam street fits a limited number of facilities, hence providing a lower variation in associated safety and security issues than a municipality. Thus, the amount and the type of facilities present in a limited space on the exam street do not yield the same complexity and variety found in the real working environment and, therefore, are less authentic, which limits the construct validity.

Instead, students are expected to demonstrate IPS during their internship. Although this places students in authentic situations that require them to act as professionals (Gulikers et al., 2004; Gulikers et al., 2006), workplace assessment has several drawbacks. First, the assessment content cannot be controlled because it is tied to the characteristics of the workplace. Therefore, incongruencies in the amount and type of information are likely to occur. On the one hand, this could lead to construct-irrelevant variance when the task goes beyond the competencies required by an entry-level employee. On the other hand, construct underrepresentation could occur when the information available in the municipality does not provide enough complexity. Second, workplace assessment outcomes depend on the competence of the supervisor. Although the ability to plan a surveillance route is a qualification requirement, not all MEOs plan surveillance routes on a regular basis. Especially in large municipalities, there are dedicated teams that plan surveillance routes and teams that execute them. In other words, the real working environment shows too much variation to assess students in a standardised way. The focus of the present study is on addressing the issue through the development of a simulation game, in which the controlled game-based environment ensures that MEO students are assessed equally based on their IPS skills.

Simulation Games

Simulation games are a type of serious game: a (digital) game-based environment, which is designed for more than just entertainment (Susi et al., 2007), such as assessment purposes. Simulation games are characterised by (1) providing students with real-world tasks in a realistic environment (Levy, 2013; Sitzmann, 2011; Wools et al., 2019), and (2) including game mechanics that shape the characteristics of games, such as establishing a clear goal and provide rules, choices, and feedback (Bijl et al., 2024; Charsky, 2010; Kim & Shute, 2015; Lameras et al., 2017; Prensky, 2001). The use of simulation games offers several advantages, combining the qualities of both the exam street and workplace assessments. First, simulation games allow for a uniform and controlled assessment while simultaneously facilitating the collection of rich (process) data through a complex and interactive task (Schwartz & Arena, 2013; Shaffer & Gee, 2012; Shute et al., 2009). Second, simulation games enable the recreation of real-life criterion situations, which is especially beneficial for important yet rarely encountered scenarios in the real working environment (Bell et al., 2008; Dörner et al., 2016; Fonteneau et al., 2020; Harteveld, 2011; Michael & Chen, 2006). Finally, although not the main aim, simulation games are designed to create an enjoyable experience that could mitigate negative emotions associated with testing (e.g., test anxiety; Mavridis & Tsiatsos, 2017) while fostering engagement and autonomy (Boyle et al., 2012).

Information Problem Solving

Much research surrounding IPS is rooted in finding information in sources from the library or internet (Brand-Gruwel et al., 2005; Brand-Gruwel et al., 2009; Frerejean et al., 2016; Frerejean et al., 2019; Hinostroza et al., 2018; Moore, 1995). Researchers have agreed that IPS is not a singular skill but that it consists of multiple constituent skills or steps that underlie successful IPS.

Figure 2 presents the different steps that comprise the IPS process based on earlier skill decompositions (Brand-Gruwel et al., 2005; Frerejean et al., 2016; Frerejean et al., 2019). IPS can be deconstructed into five constituent skills or steps: (1) defining the information problem, (2) searching for information, (3) scanning the information, (4) processing the information, and (5) organising and presenting the information. Throughout the process, regulatory activities take place. These activities include task orientation, process monitoring, performance steering, time management, and content and quality evaluation in light of task requirements.

Figure 2.
The Skill Decomposition of the Process of Information Problem Solving.

The first step to solving an information problem to have a clear definition of the information problem. The activities in this step include gaining an understanding of the task (requirements), hence concretising the problem, and activating relevant prior knowledge. The second step is to find and select the relevant sources needed to solve the problem. For IPS conducted on the internet, this step involves creating and executing a search query. This step includes the first evaluation of the quality, relevance and reliability of the information. The third step is scanning the available information to determine whether it is connected to the given information problem. Activities include evaluating the information and source and judging the scanned information; for IPS using the internet this refers to obtained websites. The fourth step is (deep) processing the information. Activities include gaining a deep understanding of the information, judging the processed information and integrating different pieces of information with relevant prior knowledge. The final step is organising and presenting the information. In this step, the information is synthesised into an end product that solves the information problem. The activities include formulating the solution as well as structuring and formatting the end product to match the task requirements.

Workers in many vocations rely on information from domain-specific sources to solve problems and make decisions. In comparison, the model described above is more focused on using sources from the internet. That being said, the underlying process is expected to be similar. The current study focuses on developing a digital simulation game aimed at measuring aspiring MEOs’ (domain-specific) IPS skills. The development centred around two main research questions. First, to what extent can the IPS model be applied to the process of planning a surveillance route? Second, to what extent can the IPS model serve as a framework for developing construct-relevant tasks within a simulation game that operationalise the process of solving information problems in vocational contexts?

Methods

Educational Design Research

The simulation game to assess MEOs’ IPS skills was developed through an educational design research methodology. Educational design research has two concurrent aims: gaining theoretical understanding while simultaneously designing practical solutions to real-world problems (McKenney & Reeves, 2019; Plomp, 2013). To develop an intervention that is both grounded in theory and supported by practice, educational design research is carried out in three distinct phases: analysis and exploration, design and construction, and evaluation and reflection. A research project can repeatedly iterate through these phases as the solution is adjusted and refined. The aim of the first phase—analysis and exploration—is to gain a full understanding of the problem at hand through consulting subject matter experts and reviewing the relevant literature. The second phase is design and construction, where the most viable solution of those that have been explored in phase one is created. In the third phase, evaluation and reflection, the solution is tested, and insights are used as the input for the further development of both the solution and theoretical understanding.

Phase 1: Analysis and Exploration—Domain Analysis

This phase was conducted prior to the design iterations and was aimed at gaining a full understanding of the problem. To this end, several aspects of the assessment of planning a surveillance route were examined. First, the test blueprint for the current examination was investigated. Second, a domain analysis was carried out by consulting a subject matter expert and analysing the kwalificatiedossier [qualification file] (KD; Samenwerkingsorganisatie Beroepsonderwijs Bedrijfsleven, 2023) of the vocational domain. Third, the different systems that MEOs consult in practice when planning surveillance routes were inspected. From there, the alignment between how the construct is being assessed and how it was operationalised in practice became clear. The discrepancies that were found (i.e., lower complexity and authenticity) were described in the introduction of the present study.

Municipal Enforcement Officials

In the Netherlands, MEOs are responsible for the upkeep of public safety and security within a municipality. MEOs have two main responsibilities: (1) being serviceable to those residing in the municipality and (2) detecting and reducing irregularities within the municipality. For the latter responsibility, important activities include recognising aberrant behaviour, conducting disciplinary talks with offenders, issuing sanctions and diffusing escalated situations. Most commonly, MEOs deal with offences such as parking violations, nuisance behaviour, and the incorrect disposal of waste. MEOs are authorised to issue fines in public spaces. The types of fines and other authorisations granted to MEOs depend on the specific safety and security concerns within a given municipality.

To become certified, aspiring MEOs must pass a series of standardised certification tests (both theoretical and practical) that assess the key competencies required for an entry-level MEO. Although anyone meeting the minimal education standard can sign up to become certified, these certification tests are generally embedded within a 2- or 3-year vocational school curriculum. This curriculum is based on the minimal requirements of a starting practitioner as stated within the KD. The KD states that the surveillance of a designated area is a core task of an MEO. When preparing for surveillance, MEOs are expected to make a plan that is guided by information from various sources. A successful plan requires the ability to process a large quantity of information, judge its reliability and relevance and decide on the best course of action. For example, when the information points to parking issues around a school during drop-off times, an MEO makes sure to plan to be there during the drop-off times.

In practice, MEOs receive a clearly structured task that serves as a guideline for the surveillance route, including (1) the area in which the surveillance should take place, (2) the main (and potentially secondary) focus of the surveillance and (3) the authorisations granted to MEOs within the given municipality. MEOs receive their information about the municipality through four main sources in different modalities (see Table 1). Where they are helpful and available, images are attached as well.

Table 1.
Main Sources Provided to MEOs Within a Municipality.

Source Main modality Content

Supervisor Briefing (presentation)
Current affairs

Relevant developments

Colleagues Email
Requests

Relevant developments

Professional partners Email
Requests

Relevant developments

Residents Tickets
Complaints

Inquiries

Information Problem Solving

Decisions among MEOs may differ even when they are supplied with the same information, resulting in different surveillance routes. The key lies in the reasoning behind the decision: as long as the reasoning is sound, differences in the surveillance routes are justified. Therefore, when evaluating the performance of planning a surveillance route, the emphasis was on the process rather than on the end product alone. The IPS model was implemented to provide insight into this process.

The IPS model (Brand-Gruwel et al., 2005; Frerejean et al., 2016; Frerejean et al., 2019) was originally intended for more academic contexts with text-based solutions, for example, writing an essay to answer a research question. Despite this, the model was used as a basis for this vocational design research for two reasons. First, the process described in the domain analysis suggests a common underlying process for solving information problems in vocational contexts. Thus, the IPS model can help identify the specific processes that MEOs follow in planning a surveillance route. Second, the main steps of the IPS model are domain general. As such, it provides a good starting point for discovering how and in what capacity these skills are also utilised when solving vocational information problems. That being said, the model was expected to include activities that fit the academic context better than the current vocational context.

Phase 2: Environment—Design Iterations

The environment Crossroads was developed in three iterations by following the principles of educational design research (McKenney & Reeves, 2019; Plomp, 2013). For each iteration, the insights gained from the previous iteration were incorporated for further development.

Paper Prototype

The first iteration of the development of the simulation game commenced with a paper prototype: a cost-effective test of the overall feasibility of the task at the basis of Crossroads. For this paper prototype, several materials were prepared. First, a realistic grid map of a city in the Netherlands was created and printed on A3 paper. This map was fitted with the basic infrastructure of the city, and a legend was created to indicate several landmarks within the city.

Second, three different sources were created: a briefing given on a slide deck, tickets received over the phone (i.e., phrased by customer support) or through an online form (i.e., phrased by residents themselves) and emails received from colleagues and professional partners. All information included a clear subject line. The tickets and emails additionally showed a date and source; for the briefing this was inherently clear as a briefing is generally given on the day by a colleague or supervisor. All pieces of information were given a coordinate through which they could be located using the grid on the map. The content of the information was developed in collaboration with subject matter experts.

Finally, an instruction form was created with background information about the role of the MEO in the municipality, as well as some guidelines that should be followed in planning the surveillance route. These guidelines included the main theme of surveillance (i.e., a focus on parking violations in the city). To aid in planning a surveillance route on the map as the output, three types of materials were provided: Post-it notes, pens and various coloured markers.

The paper prototype was tested with students in the final year of their vocational education and training (VET) programme to become an MEO (N = 5). The participants were all expected to have the same level of prior knowledge about planning a surveillance route; they followed the same curriculum and had all gained practical experience through an internship. Students were given approximately 30 min to plan a surveillance route using the materials described above. Two of the five participants were asked to take into account travelling time between locations and time estimates of the intended actions that followed from the provided information. All participants were instructed to think aloud while planning the surveillance route. After the task, the participants were asked to evaluate the task (i.e., in relation to the real working environment).

Interface Design

Based on the results of the paper prototype, the next design iteration focused on the envisioned interface design of a digital prototype. To this end, a mock-up was created in AdobeXD. The mock-up was designed to resemble a computer desktop with a split screen and taskbar. The taskbar shows three icons corresponding with one source each. The left side of the screen could display one of these sources at a time. The right side of the screen showed a digital map, which served as the end product for the surveillance route. To serve the digital environment of Crossroads, two additional features were designed in the mock-up: (1) a list with all planned tasks so that the order could be changed dynamically, and (2) an urgent notification that needed to be incorporated into the surveillance route at the end. For the mock-up, some of the envisioned interactions were already available. The screens were used to conduct a usability test with a different group of students in the final year of their VET programme (N = 8). Following the same research procedures as with the paper prototype, the students were instructed to click through the mock-up in pairs. The findings were incorporated into the design of the final prototype.

Final Prototype

Based on the input from both design iterations, the final prototype Crossroads was developed (see Figure 3 for a screenshot). Crossroads is built as a web application with React (React Team, 2025), hosted on Firebase (Google, 2025), ensuring compatibility across most devices and platforms via a web browser, provided there is internet connection. Students can thus access Crossroads on devices they are familiar with without installing any software. The content of Crossroads is organized and updated through FireCMS (FireCMS, 2025), a flexible open-source content management system. This system supports easy creation of unique content within Crossroads, simplifying the process of adjusting the content of Crossroads to generate different versions of the assignment. Additionally, Crossroads is built as a modular system, which enables repurposing the architecture for new target groups without needing to build a new application.

Figure 3.
Screenshot of Crossroads.

Game mechanics (i.e., a clear goal, freedom of choice, rules or restrictions and feedback) are integrated throughout the workflow of the final prototype. Upon login, students are forwarded to a desktop-like environment where they can interact with separate windows for each source and the map. The interface allows students to freely navigate between windows and to move, resize or minimize them dynamically, creating a flexible and realistic workspace that provides students with the freedom of choice. The interface additionally includes a taskbar, in which students are constantly provided with feedback on how many pieces of information still require a decision. Students are restricted to a predefined number of decisions on how to handle each piece of information. They can choose to (1) include it in their surveillance route, (2) reject it and pass it on to the next shift, or (3) forward it to a professional partner. These decisions should be based on the quality, reliability and relevance of the information given the goal detailed in the assignment. At any point, students can revise their choices until the final surveillance route is completed.

For each piece of information included in the surveillance route, students must further detail their planning by providing (1) an action description of the related task, (2) a time estimate, and (3) a priority appraisal. All the tasks planned in the surveillance are combined into an ordered list that students can reorder freely. Below the ordered list, students are given feedback on whether their end time meets the time requirements set in the assignment. Once students finalise their initial surveillance route, they receive two urgent notifications that must be incorporated into a final version. The complete workflow, including how the IPS model is implemented in Crossroads, is illustrated in Figure 4.

Figure 4.
The Process of Planning a Surveillance Route in Crossroads with Associated Steps of IPS.

Student interactions are time-stamped and stored anonymously in Firebase (Google, 2025). As students engage with Crossroads, their interactions are recorded through in-game choices that function as assessment items. For example, when students estimate the time needed for a task, they select one of six duration options which is logged as the response to a multiple-choice item. Similarly, other in-game choices are logged as responses to multiple-choice items, multiple-response items or open-ended items. This data is used to evaluate student performance through a structured assessment form that aligns with the constituent skills in the IPS model (except defining the information problem and regulation). Students are awarded points for searching for information when focusing on the right key information in their substantiation for each piece of information. Points are awarded for scanning the information when they correctly evaluate whether a piece of information should be included in the surveillance route or not. For processing the information, students are awarded points for correctly elaborating on the planned items: giving a correct task description and assigning a correct time estimate and priority level. Finally, organising and presenting the information is assessed by evaluating whether two set task requirements – the required end time and designated break time – were met.

Phase 3: Think-Aloud Method—Evaluation

For the paper prototype—the first design iteration (Phase 2) —a think-aloud method (TAM) was used to ascertain the extent to which the steps of the IPS model were present in the process of planning a surveillance route. TAMs are used when researchers are more interested in the cognitive processes underlying problem solving than the end product (van Someren et al., 1994). In this method, the participants are asked to solve a problem and concurrently verbalise everything that comes to mind. All participants were recorded during the task and the evaluation, and these recordings were later transcribed.

Analysis of the transcripts was done through deductive thematic analysis, meaning that the themes were determined prior to data analysis (Boeije, 2010). The categorisation in deductive thematic analysis highly depends on how transcript passages and category definitions are interpreted. To assess coding reliability, transcripts were double-coded using ATLAS.ti Windows (version 22.2.5; version 23.0.8). The categories within one theme were mutually exclusive. Krippendorff's coefficient was calculated to determine the intercoder reliability of the transcripts. The reliability analysis was done at the sentence level; when one sentence was coded into more categories, it was split into the biggest possible unit. When the split resulted in trivial sentence parts (e.g., linking words), these were removed from the analysis.

Prior to coding, the categories were piloted on the transcript of one participant to resolve the initial ambiguity. Then, two rounds of coding were conducted. First, both coders independently decided on the relevant passages and their corresponding categories. The final coding was achieved after resolving systematic misinterpretations and elaborating categories where needed. The overall agreement was 93% (Krippendorff's _Cuα = .913). Disagreements between the two coders in the final coding were resolved by a third independent coder. The same transcript was used to familiarise the third coder with the categories. The final coding consisted of 926 units of which 86.1% (N = 769) were coded to an IPS category. The average number of coded units per student was 153.8 (SD = 27).

Results

Table 2 presents the results from the evaluation phase of the educational design research method, where students planned a surveillance route using the paper prototype. For each student, the distribution of the units identified as IPS is shown. As can be seen, variations exist in how the units per IPS step are distributed per student.

Table 2.
Distribution of Steps of Information Problem Solving per Student.

Information problem solving Student Total (%)

1 2 3 4 5

Defining the information problem 11 4 7 3 1 26 (3.4%)

Searching for information 60 16 23 25 40 164 (21.3%)

Scanning the information 32 17 17 16 17 99 (12.9%)

Processing the information 14 69 7 76 56 222 (28.9%)

Organising / Presenting the information 36 23 12 25 10 106 (13.8%)

Regulation 39 29 43 19 22 152 (19.8%)

Total 192 158 109 164 146 769

The students showed both similarities and differences to the IPS model in those activities underlying the constituent skills, which are described in the next section. Thereafter, the findings are summarised in an adapted IPS model intended for vocational information problems.

Information Problem Solving for Surveillance Planning

Defining the Information Problem

All students in the sample had the least units assigned to defining the information problem. These units were found at the beginning and during the task. The statements made in this regard were mostly clarifying questions about the task requirements and end product (i.e., the surveillance route). It is important to mention that the latter was intentionally left undefined, to use the output to guide further design iterations. Some students voiced the task in their own words, for example, by saying, ‘Parking nuisance is my first priority, the secondary task comes after that’ (S5). The students did not activate prior knowledge because the information problem was about making a surveillance route for an unknown city.

Searching for Information

The task given to the students did not require them to search for information. Instead, students were provided with a large amount of information from different sources. In searching for information, the students were mainly focused on quickly evaluating the key message of the task. This process occurred throughout the task because the students often referred to the information while planning the surveillance route. The category mainly included statements in which students described the most important information from the given sources. For instance, ‘That is about litter on the street’ (S1). Most often, the students immediately followed up to determine the relevance and further processing of the information.

Scanning the Information

In scanning the information, the students answered the following question: Is this information relevant to include given the task? Statements relating to this process occurred throughout the task. The current information problem required students to determine whether the information should be included in the surveillance route given alignment with the task, the source of the information, the priority of the content and the location on the map. Information deemed irrelevant for the information problem was additionally judged by students: both in terms of for whom or when the information would be relevant. For example, when a student decided that information was not relevant for their surveillance route, they said, ‘Here on E2 [coordinate], that is something we can’t do anything with ourselves, that really needs to be forwarded to the municipality’ (S2).

Processing the Information

Students engaged in processing the information elaborated on the information in two ways. First, they elaborated on what tasks were associated with the information. Second, they elaborated on how the information would need to be incorporated into the end product. For the latter, students often activated prior knowledge. Some students also engaged in elaboration that was not relevant to the end product. For example, one individual stated, ‘Ask the municipality to take a sample so we can see what is going on’ (S2). Additionally, students processed information given in the task that did not work towards the information problem but did go beyond the information that was provided. This was often preceded by determining relevance to the information problem. For example, another participant stated ‘It will probably go sideways in that street, they will get into an argument’ (S3).

Organising and Presenting the Information

Students were engaged in organising and presenting the information when they were working on creating the end product. In most statements, the students structured the different pieces of information and synthesised them to fit the end product. More specifically, the students determined the relative order in which tasks should be planned in the surveillance route. Some students created an outline of the surveillance prior including the specific tasks: ‘I would like to first go down here [on the map], and then I can start on the outside [of the city centre] and work my way in. Then I can potentially spend the remainder of my shift on the right side [of the map]’ (S2).

Regulation

Within regulation, student processes included describing the strategy they would use to plan the surveillance route, which included their approach in categorising the different pieces of information and monitoring the progress they made. For example: ‘And writing it [the legend] down here to see what is what, otherwise I will forget’ (S3). Additionally, statements about time management were found, such as inquiries about how much time was left.

Framework for Vocational Information Problem Solving

Applying the IPS model to the process of planning a surveillance route provided insight into an adjusted framework for information problem solving in vocational contexts. An adapted IPS model is presented in Figure 5.

Figure 5.
Skill Decomposition of Information Problem Solving of Information Problems in Vocational Contexts.

First, while planning the surveillance route students did not activate prior knowledge while defining the information problem. In contrast, prior knowledge was activated to appropriately process individual pieces of information. Thus, in the adapted model, the activation of prior knowledge is shifted to processing the information.

Second, problem solvers in vocational contexts use specific sources within their vocation to solve information problems. Often, they do not require internet searches to find relevant information. Instead, information problems in vocational contexts require problem solvers to explore the available information and understand the core message. Therefore, the subskills specifically targeted at internet skills under searching for information (i.e., creating a search query and executing a search query) were not executed by students. These were removed from the model.

Third, the students tended to go beyond the information that was relevant to the information problem at hand while planning the surveillance route. When it was not something to be planned in the surveillance route, the students talked about for whom and when it would be relevant. Additionally, the students elaborated on what should be done while they would not do it in surveillance. For vocational information problems, it is more natural not to disregard the information but to process it concurrently with the immediate task. For MEOs, the information not handled in the surveillance should be handled eventually by a colleague or a different department. Therefore, the model was adjusted to include the decision on irrelevant information as well because this is an explicit process that students go through.

Finally, solutions to vocational information problems can take on numerous modalities. Although the students did provide some text to accompany the surveillance route in the current information problem, the surveillance route itself is the product required to solve the information problem. Thus, the activity formulate solution under organising and presenting the information was removed from the model because it explicitly refers to formulating text.

Discussion

The present study focused on developing a digital simulation game to be used as a measure of MEO students’ (domain-specific) IPS skills. The development, following the principles of educational design research (McKenney & Reeves, 2019; Plomp, 2013), focused on two research questions. First, to what extent can the IPS model be applied to the process of planning a surveillance route? Second, to what extent can the IPS model serve as a framework for developing construct-relevant tasks within a simulation game that operationalises the process of solving information problems in vocational contexts?

Regarding the first research question, the results of the TAM show that the IPS model aligns with the processes involved in planning a surveillance route. While working on the paper prototype, all students made statements for each of the steps described within the model. Thus, although the IPS model is generally used in a more academic context (i.e., answering a research question through internet sources), it can provide insight into solving information problems in this vocational context.

Some differences were found in terms of the activities underlying the constituent skills in the IPS model and the students’ process of planning a surveillance route, which were summarised in an adapted model of IPS for vocational contexts. In the adjusted model, the activation of prior knowledge was shifted to processing the information, specific internet search activities were removed, the decision-making process for irrelevant information was made explicit, and the varied modalities of solutions in vocational information problems was made less constrictive.

Regarding the second research question, the design of the digital environment Crossroads shows that an adjusted version of the IPS model can be a useful starting point for developing construct-relevant tasks in which the process of solving vocational information problems is operationalised in a simulation game. Although the present study focused on a domain-specific task, it is not inconceivable that this model can be applied to other contexts as well. For example, consider an executive assistant in charge of managing a calendar (i.e., the information problem). This information problem is solved through evaluating and processing information from multiple sources (e.g., emails and meeting minutes) and incorporating these in the calendar appropriately. That being said, in the current study the model was tested and adjusted based on a small sample of participants. Further research should be conducted to determine whether these adjustments to the model are appropriate for other vocational contexts as well.

There is much potential in using a more generic model for IPS. First, a generic IPS model provides support for judging student performance in terms of both the end product and students’ processes. Information problems do not have one right answer, so this focus on the process of creating the solution is warranted. Second, having a generic model to assess IPS in vocational contexts would enable the creation of more fantasy, game-like environments (Charsky, 2010; Prensky, 2001) in which IPS can be assessed. Although examples of serious games exist that include fantasy contexts (IJgosse et al., 2020; McCord et al., 2019), further research should determine its feasibility and validity for assessing vocational skills. When IPS or other domain general skills can be assessed in a generic context, this would pave the way for game-based environments that are both sustainable and widely applicable.

The present study also has some limitations. First, the current study is a small-scale qualitative study limiting the generalisability of the findings. That being said, the sample size is consistent with other design studies, as the majority of usability problems can be uncovered with a small number of participants (Nielsen & Landauer, 1993). Despite this, the sample size is not adequate to validate to applicability of the IPS model for the assessment of vocational information problem solving. Therefore, a separate study on the validation of Crossroads was conducted with MEO students and practitioners (Bijl et al., 2025). Second, the TAM has a potential confounding effect. Students are not accustomed to thinking aloud while working on a problem, which could have altered the process. Although TAMS are a common way to extract the process, it is impossible to retrospectively determine whether students would have done the same if they were not instructed to think aloud. Other methods exist that aim to give insights into the cognitive processes underlying problem solving, such as retrospective reporting and cued retrospective reporting (van Gog et al., 2005; van Someren et al., 1994); however, retrospective reporting methods tend to result in loss of information (van Gog et al., 2005) or recall bias (Russo et al., 1989). Third, the IPS model applied to the process of planning a surveillance route has been used primarily for solving information problems through the internet. More specific models for similar vocational competencies could have been chosen as well. For example, Rausch and Wuttke (2016) developed a model that has been used for problem solving in the domain of controlling education (Rausch & Wuttke, 2016; Seifried et al., 2020). However, although Rausch and Wuttke (2016) focused mainly on problem solving, the IPS model used in the current study (Brand-Gruwel et al., 2005; Frerejean et al., 2016; Frerejean et al., 2019) additionally focused on the information literacy aspect of IPS. Because the aim of the present study was to gain insights into the process of planning a surveillance route, the additional steps present were advantageous in the given context.

To conclude, the current study aimed to develop a digital simulation game through which MEO students could be assessed on their ability to plan a surveillance route. The present study has given preliminary evidence that, with modifications, the IPS model adapted from Brand-Gruwel et al. (2005), Frerejean et al. (2016) and Frerejean et al. (2019) can be aligned with the process underlying planning surveillance route. That being said, the limited sample size warrants further investigation into its applicability. The adapted framework has the potential to be more generally applicable to information problems in vocational contexts, supporting the process of creating construct-relevant tasks through which the process of vocational IPS can be assessed.

Source	Main modality	Content
Supervisor	Briefing (presentation)	Current affairs Relevant developments
Colleagues	Email	Requests Relevant developments
Professional partners	Email	Requests Relevant developments
Residents	Tickets	Complaints Inquiries

Information problem solving	Student	Total (%)
Defining the information problem	11	4	7	3	1	26 (3.4%)
Searching for information	60	16	23	25	40	164 (21.3%)
Scanning the information	32	17	17	16	17	99 (12.9%)
Processing the information	14	69	7	76	56	222 (28.9%)
Organising / Presenting the information	36	23	12	25	10	106 (13.8%)
Regulation	39	29	43	19	22	152 (19.8%)
Total	192	158	109	164	146	769

Footnotes

ORCID iDs

Aranka Bijl

Sebastiaan de Klerk

Bernard P. Veldkamp

Eva de Schipper

Saskia Wools

Ethical Considerations

The research was approved by the ethics committee of the Behavioural, Management, and Social Sciences department of the University of Twente (request number 230152).

Consent to Participate

All participants were provided with written information and gave their informed consent to participate by signing a consent form in which they could specify specific elements of consent.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The data that support the findings of this study are not publicly available due to confidentiality agreements, but samples of the data can be accessed by contacting the author.

Author biographies

Aranka Bijl is a PhD candidate at the University of Twente and a researcher at Cito. Her research focuses on the implementation of valid and reliable performance assessment through digital serious games in high-stakes contexts.

Sebastiaan de Klerk is a senior educational researcher at Cito. His research focuses on the use of educational technology, especially in relation to innovative educational assessment.

Bernard P Veldkamp is vice-dean of research at the University of Twente. His research focuses on psychometrics and artificial intelligence for educational assessment.

Eva de Schipper is a researcher at Cito. Her research focusses on using educational assessment data in innovative ways to add value to educational practice.

Saskia Wools is the CEO of Cito. Her research focuses on validity, validation and innovation in assessment such as technology enhanced assessment and formative assessment practices.

References

Bell

B. S.

Kanar

A. M.

Kozlowski

S. W. J.

(2008). Current issues and future directions in simulation-based training in North America. The International Journal of Human Resource Management, 19(8), 1416–1434. https://doi.org/10.1080/09585190802200173

Bijl

de Klerk

Veldkamp

B. P.

Wools

(2025). Capturing information problem solving: Validation evidence for the simulation game-based assessment of complex vocational skills [Manuscript submitted for publication].

Bijl

Veldkamp

B. P.

Wools

de Klerk

(2024). Serious games in high-stakes assessment contexts: a systematic literature review into the game design principles for valid game-based performance assessment. Educational Technology Research and Development. https://doi.org/10.1007/s11423-024-10362-0

Boeije

(2010). Analysis in qualitative research. Sage.

Boyle

E. A.

Connolly

T. M.

Hainey

Boyle

J. M.

(2012). Engagement in digital entertainment games: A systematic review. Computers in Human Behavior, 28(3), 771–780. https://doi.org/10.1016/j.chb.2011.11.020

Brand-Gruwel

Wopereis

Vermetten

(2005). Information problem solving by experts and novices: Analysis of a complex cognitive skill. Computers in Human Behavior, 21(3), 487–508. https://doi.org/10.1016/j.chb.2004.10.005

Brand-Gruwel

Wopereis

Walraven

(2009). A descriptive model of information problem solving while using internet. Computers & Education, 53(4), 1207–1217. https://doi.org/10.1016/j.compedu.2009.06.004

Charsky

(2010). From edutainment to serious games: A change in the use of game characteristics. Games and Culture, 5(2), 177–198. https://doi.org/10.1177/1555412009354727

Dörner

Göbel

Effelsberg

Wiemeyer

(2016). Introduction. In Dörner

Göbel

Effelsberg

Wiemeyer

(Eds.), Serious games: Foundations, concepts and practice (pp. 1–34). Springer International Publishing. https://doi.org/10.1007/978-3-319-40612-1_1

10.

Exameninstelling Toezicht en Handhaving. (n.d.). Plattegrond Examenstraat [Lay-out Exam Street]. https://exth.nl/wp-content/uploads/Plattegrond-examenstraat.pdf

11.

FireCMS. (2025). FireCMS [Computer Software]. Retrieved from https://firecms.co/

12.

Fonteneau

Billion

Abdoul

Hadchouel

Drummond

(2020). Simulation game versus multiple choice questionnaire to assess the clinical competence of medical students: Prospective sequential trial. Journal of Medical Internet Research, 22(12), Article e23254. https://doi.org/10.2196/23254

13.

Frerejean

van Strien

J. L. H.

Kirschner

P. A.

Brand-Gruwel

(2016). Completion strategy or emphasis manipulation? Task support for teaching information problem solving. Computers in Human Behavior, 62, 90–104. https://doi.org/10.1016/j.chb.2016.03.048

14.

Frerejean

Velthorst

G. J.

van Strien

J. L. H.

Kirschner

P. A.

Brand-Gruwel

(2019). Embedded instruction to learn information problem solving: Effects of a whole task approach. Computers in Human Behavior, 90, 117–130. https://doi.org/10.1016/j.chb.2018.08.043

15.

Funke

(2010). Complex problem solving: A case for complex cognition? Cognitive Processing, 11(2), 133–142. https://doi.org/10.1007/s10339-009-0345-0

16.

Google. (2025). Firebase [Computer Software]. Retrieved from https://firebase.google.com/

17.

Gulikers

J. T. M.

Bastiaens

T. J.

Kirschner

P. A.

(2004). A five-dimensional framework for authentic assessment. Educational Technology Research and Development, 52(3), 67–86. https://doi.org/10.1007/bf02504676

18.

Gulikers

Bastiaens

Kirschner

(2006). Authentic assessment, student and teacher perceptions: The practical value of the five-dimensional framework. Journal of Vocational Education & Training, 58(3), 337–357. https://doi.org/10.1080/13636820600955443

19.

Harteveld

(2011). Foundations. In Triadic game design: Balancing reality, meaning and play (pp. 31–93). Springer London. https://doi.org/10.1007/978-1-84996-157-8_2

20.

Hinostroza

J. E.

Ibieta

Labbé

Soto

M. T.

(2018). Browsing the internet to solve information problems: A study of students’ search actions and behaviours using a ‘think aloud’ protocol. Education and Information Technologies, 23(5), 1933–1953. https://doi.org/10.1007/s10639-018-9698-2

21.

IJgosse

van Goor

Rosman

Luursema

J.-M.

(2020). Construct validity of a serious game for laparoscopic skills training: Validation study. JMIR Serious Games, 8(2), Article e17222. https://doi.org/10.2196/17222

22.

Kim

Y. J.

Shute

V. J.

(2015). The interplay of game elements with psychometric qualities, learning, and enjoyment in game-based assessment. Computers & Education, 87, 340–356. https://doi.org/10.1016/j.compedu.2015.07.009

23.

Lameras

Arnab

Dunwell

Stewart

Clarke

Petridis

(2017). Essential features of serious games design in higher education: Linking learning attributes to game mechanics. British Journal of Educational Technology, 48(4), 972–994. https://doi.org/10.1111/bjet.12467

24.

Levy

(2013). Psychometric and evidentiary advances, opportunities, and challenges for simulation-based assessment. Educational Assessment, 18(3), 182–207. https://doi.org/10.1080/10627197.2013.814517

25.

Mavridis

Tsiatsos

(2017). Game-based assessment: Investigating the impact on test anxiety and exam performance. Journal of Computer Assisted Learning, 33(2), 137–150. https://doi.org/10.1111/jcal.12170

26.

McCord

J.-L.

Harman

J. L.

Purl

(2019). Game-like personality testing: An emerging mode of personality assessment. Personality and Individual Differences, 143, 95–102. https://doi.org/10.1016/j.paid.2019.02.017

27.

McKenney

Reeves

T. C.

(2019). Conducting educational design research (2nd ed.). Routledge.

28.

Michael

Chen

(2006). Serious games: Games that educate, train, and inform. Muska & Lipman/Premier-Trade.

29.

Moore

(1995). Information problem solving: A wider view of library skills. Contemporary Educational Psychology, 20(1), 1–31. https://doi.org/10.1006/ceps.1995.1001

30.

Nielsen

Landauer

T. K.

(1993). A mathematical model of the finding of usability problems. Proceedings of the INTERACT ‘93 and CHI ‘93 Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands. https://doi.org/10.1145/169059.169166

31.

Plomp

(2013). Educational design research: An introduction. In Plomp

Nieveen

(Eds.), Educational design research part A: An introduction (pp. 10–51). Netherlands Institute for Curriculum Development (SLO).

32.

Prensky

(2001). Fun, play and games: What makes games engaging? In Prensky

(Ed.), Digital game-based learning (pp. 16–47). McGraw-Hill.

33.

Rausch

Wuttke

(2016). Development of a multi-faceted model of domain-specific problem-solving competence and its acceptance by different stakeholders in the business domain. Unterrichtswissenschaft, 44(2), 169–184.

34.

React Team. (2025). React [Computer Software]. Retrieved from https://react.dev/

35.

Russo

J. E.

Johnson

E. J.

Stephens

D. L.

(1989). The validity of verbal protocols. Memory & Cognition, 17(6), 759–769. https://doi.org/10.3758/BF03202637

36.

Samenwerkingsorganisatie Beroepsonderwijs Bedrijfsleven. (2023). Handhaver toezicht en veiligheid Openbare ruimte (domein I). https://kwalificatie-mijn.s-bb.nl/kwalificatie/handhaver-toezicht-en-veiligheid-openbare-ruimte-domein-i/cmVzdWx0YWF0VHlwZT01 O2Rvc3NpZXJJZD01NzkzO2t3YWxpZmljYXRpZUlkPTEzOTA3Mzk=

37.

Schwartz

D. L.

Arena

(2013). Measuring what matters most: Choice-based assessment for the digital age. The MIT Press.

38.

Seifried

Brandt

Kögler

Rausch

(2020). The computer-based assessment of domain-specific problem-solving competence—A three-step scoring procedure. Cogent Education, 7(1), Article 1719571. https://doi.org/10.1080/2331186X.2020.1719571

39.

Shaffer

D. W.

Gee

J. P.

(2012). The right kind of GATE: Computer games and the future of assessment. In Mayrath

M. C.

Clarke-Midura

Robinson

D. H.

(Eds.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research (pp. 211–228). Information Age Publishing.

40.

Shute

V. J.

Ventura

Bauer

Zapata-Rivera

(2009). Melding the power of serious games and embedded assessment to monitor and foster learning: Flow and grow. In Ritterfeld

Cody

M. J.

Vorderer

(Eds.), Serious games: Mechanisms and effects (pp. 295–321). Routledge. https://doi.org/10.4324/9780203891650

41.

Sitzmann

(2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology, 64(2), 489–528. https://doi.org/10.1111/j.1744-6570.2011.01190.x

42.

Susi

Johannesson

Backlund

(2007). Serious games: An overview [IKI Technical Reports]. https://www.diva-portal.org/smash/get/diva2:2416/FULLTEXT01.pdf

43.

van Gog

Paas

van Merriënboer

J. J. G.

Witte

(2005). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology: Applied, 11(4), 237–244. https://doi.org/10.1037/1076-898X.11.4.237

44.

van Merriënboer

J. J. G.

Kirschner

P. A.

(2018). Ten steps to complex learning: A systematic approach to four-component instructional design (3rd ed.). Routledge. https://doi.org/10.4324/9781315113210

45.

van Someren

M. W.

Barnard

Y. F.

Sandberg

J. A. C.

(1994). The think aloud method: A practical guide to modelling cognitive processes. Academic Press.

46.

Wools

Molenaar

Hopster-den Otter

(2019). The validity of technology enhanced assessments—threats and opportunities. In Veldkamp

B. P.

Sluijter

(Eds.), Theoretical and practical advances in computer-based educational measurement (pp. 3–19). Springer International Publishing. https://doi.org/10.1007/978-3-030-18480-3_1

Information Problem Solving in Vocational Contexts: Designing a Simulation Game for Dutch Municipal Enforcement Officials

Abstract

Keywords

Simulation Games

Information Problem Solving

Methods

Educational Design Research

Phase 1: Analysis and Exploration—Domain Analysis

Municipal Enforcement Officials

Information Problem Solving

Phase 2: Environment—Design Iterations

Paper Prototype

Interface Design

Final Prototype

Phase 3: Think-Aloud Method—Evaluation

Results

Information Problem Solving for Surveillance Planning

Defining the Information Problem

Searching for Information

Scanning the Information

Processing the Information

Organising and Presenting the Information

Regulation

Framework for Vocational Information Problem Solving

Discussion

Footnotes

ORCID iDs

Ethical Considerations

Consent to Participate

Funding

Declaration of Conflicting Interests

Data Availability Statement

Author biographies

References