Sage Journals: Discover world-class research

Abstract

From a societal perspective, vocational education and training must enable young adults to meet the challenges of the labour market in a globalized world, reduce the mismatch of supply and demand of qualifications (e.g. youth unemployment leading to disadvantages for individuals, society and national economies) and improve social cohesion. From an individual perspective, vocational education and training should develop young adults’ vocational competencies, support their individual personality development and their integration into the labour market and society, help secure their livelihood and enable them to lead self-determined lives as citizens. Therefore, the assessment of competencies obtained in vocational education and training programmes has emerged as a critical issue to develop workforces and the capacity for life-long learning and to foster civic participation as a responsible citizen. This article provides some insights into the modelling and measurement of competencies in vocational education and training, where occupational and cross-occupational competencies are necessary to cope with the requirements of workplaces, as a responsible citizen and in private life. In this article, cross-occupational economic competencies and occupation-specific commercial competencies in the area of business and administration are discussed. Both constructs are based on economic theories, concepts and central terms; nevertheless, the situation-specific context and requirements may vary substantially. Thus, different approaches to define and measure both constructs seem to be necessary.

Keywords

competence measurement civic economic competencies commercial competencies technology-based assessment in VET vocational education and training

Introduction

Nations worldwide face the challenge of educating and credentialing the next generation. With the recent beginning of the Lisbon–Copenhagen process, there is a stronger focus on the achieved qualification levels of the different programmes of vocational education and training (VET) in European Union (EU) countries. From a societal perspective, VET must enable young adults to meet the challenges of the labour market in a globalized world, reduce the mismatch of supply and demand of qualifications (e.g. youth unemployment leading to disadvantages for individuals, society and national economies) and improve social cohesion. From an individual perspective, VET should develop young adults’ vocational competencies, support their individual personality development and their integration into the labour market and society, help secure their livelihood and enable them to lead self-determined lives as citizens. Although VET is crucial for both the integration of young people into work and society and for economic growth, competiveness and welfare, the significance of VET has often been neglected in political discussions. Today, VET and, in particular, its educational outcomes receive more attention. Youth unemployment (OECD, 2015) and a lack of qualified workers (CEDEFOP, 2014) are central challenges for national economies. Across countries, we find not only effective preparation of young adults for the labour market but also maladaptation in terms of skills mismatches (young people are in jobs for which they are overqualified, but they also show skill deficits and are not proficient in their jobs). Skills mismatches affect both economic outcomes and individual needs. Irrespective of the political problems with arranging common and comparable qualification levels, there is very little scientific and empirically based knowledge about the cross-occupational and occupational competencies of learners in VET programmes in different European countries (OECD, 2012). The assessment of competencies obtained in VET programmes has emerged as a critical issue to develop workforces and the capacity for life-long learning and to foster civic participation as a responsible citizen. While the assessment of competencies may be critical to national well-being in the twenty-first century, as Hartig et al. (2008) noted, ‘the theoretical modeling of competencies, their assessment, and the usage of assessment results in practice present new challenges for psychological and educational research’ (p. V), particularly with regard to VET.

This article provides some insights into the modelling and measurement of competencies in VET, where occupational and cross-occupational competencies are necessary to cope with the requirements of workplaces and in private life. I would like to concentrate my presentation on cross-occupational economic competence in the sense of both a responsible employee and a responsible citizen, which is an important goal in VET. I will also discuss the occupational and commercial competencies in the area of business and administration. Both constructs are based on economic theories, concepts and central terms; nevertheless, the situation-specific context and requirements may vary substantially (acting as a responsible citizen and acting as a commercial employee in business situations). Thus, different approaches for the measurement of both constructs are necessary. In this article, central aspects of the construct definitions and specificities of measurement of both constructs will be discussed.

Economic competencies in the sense of a responsible citizen means, on one hand, understanding economic processes that work in society and judging economic policy and strategies from an independent perspective, including the exercise of economy-related democratic rights and duties. On the other hand, it means to be able to act as a well-informed consumer in both commercial and personal situations (Schumann and Eberle, 2014). Commercial competencies in an occupational context can be described as the complex ability to perform commercial activities and solve economic entrepreneurial problems, for example, processing a consumer’s order, making decisions about different offers, calculating warehouse costs, working out a transport plan, evaluating the economic success of a marketing strategy, or interpreting ratio values and performance indicators (cf. Winther and Achtenhagen, 2009: 100).

On one hand, we speak about economic competencies (or economic literacy) in the sense of participation as a responsible economic citizen in society and as a learning goal for general and vocational education. On the other hand, economic thinking and acting in the interest of companies and business are gaining more and more importance across occupations. This type of competency is characterized by the judgement of business processes from an entrepreneurial perspective. Hence, this construct can be considered as an important facet of employability (cf. Wolter, 2013), and in the case of commercial jobs, it is included in the construct of domain-specific commercial competencies in occupational contexts (see Figure 1). As noted above, commercial competency refers to occupation- or job-specific economic knowledge and skills as well as attitudes that are necessary to, for example, pursue a commercial career. Commercial occupations require an in-depth understanding of business matters and the ability to manage business processes. Hence, commercial competency must be distinguished from general economic competency (in the sense of literacy) in society and private life, which I will discuss further later.

Figure 1.

General economic and business competencies (see Schumann and Eberle, 2014; also Dubs, 2013: 15).

Approaches to the concept of competence in VET

In educational research in general and in vocational educational research in particular, different conceptualizations of the construct ‘competency’ or ‘competence’ can be observed. Potgieter and Van Der Merwe (2002) remark ‘that ideas and thinking surrounding the notion of competence is far from uniform and that the term has different meanings in different contexts’ (p. 60). From a more general perspective, Rychen and Salganik (2003) define competency as ‘ability to successfully meet complex demands in a particular context through the mobilization of psychosocial prerequisites (including both cognitive and non-cognitive aspects)’ (p. 43). From an occupational point of view, Björnavold (2000: 208) defines competency as ‘the proven/demonstrated – and individual – capacity to use know-how, skills, qualifications or knowledge to meet usual – and changing – occupational requirements’ (see also CEDEFOP, 2003: 39). The European Community offers a broad definition of competency. In respective papers of the Lisbon–Copenhagen process, for example, occupation-related competency is seen as an ability that ‘extends beyond the possession of knowledge and skills’. This conceptualization includes cognitive facets involving the use of theory and concepts, as well as informal tacit knowledge and skills or know-how. The latter indicates those things ‘that a person should be able to do when they work in a given area’. Furthermore, occupation-related competency encompasses the personal competency ‘to conduct oneself in a specific situation’ as well as ethical competency (Lisbon-to-Copenhagen-to-Maastricht Consortium Partners, 2004: 89; for an overview of different competence definitions, see Winther, 2010: 20ff.).

A variety of definitions have been developed (Blömeke et al., 2015: 5; see also Mulder et al., 2006), but two basic demand-oriented (or more functional) approaches play an important role in the present discussion about modelling and measuring vocational (and professional) competencies (see Figure 2). The performance-oriented approach with measurement concepts of behavioural assessment was developed within industrial psychology (see Blömeke et al., 2015: 5; for an overview, see also Potgieter and Van Der Merwe, 2002). This approach implies a holistic view in which the different cognitive and non-cognitive resources as well as observable performance are linked together. Despite the broad recognition of this approach in VET- and workplace-related research, the question of how to assess such a complex construct as a whole is still unsolved (see Nickolaus and Seeber, 2013) and carries serious problems with regard to the compliance of diagnostic standards and quality criteria of measurement (cf. Baethge, 2010: 31). The second, cognitive-oriented and more analytical approach was developed in the field of educational research. This perspective on the construct of competence requires the measurement of the dispositions underlying individuals’ behaviour (Blömeke et al., 2015: 5). This position divides the construct of competence into different latent abilities, for example, knowledge (declarative and procedural), skills and attitudes. Competencies in this understanding are used to predict behaviour in specific (real-world) situations (Blömeke et al., 2015: 6).

Figure 2.

Dichotomous perspectives on the construct of competence in educational measurement (cf. Blömeke et al., 2015: 5; see also Nickolaus and Seeber, 2013; Winther, 2010).

According to Blömeke et al. (2015: 5), there is both overlap and disagreement in these dichotomous perspectives with regard to the definition and assessment of competence: both approaches regard competence as learnable and stress the importance of real-world situations as essential points of reference. Despite obvious differences, ‘processes such as the perception and interpretation of a specific job situation together with decision-making (Schoenfeld, 2010) may mediate between disposition and performance’ (Blömeke et al., 2015: 7). Blömeke et al. argued that

instead of insisting on an unproductive dichotomy view of competence, in particular knowledge or performance, competence should be regarded as a process, a continuum with many steps in between. Thus, we suggest that trait approaches recognize the necessity to measure behaviourally and that behavioural approaches recognize the role of cognitive, affective, and conative resources. (p. 7)

At the international level, there is a broad consensus regarding selected aspects of defining and modelling competency in the contexts of vocational training and work. Most empirical researchers in the area of VET and workplace learning define and operationalize competence according to Weinert (2001), who states that competencies refer to

clusters of cognitive prerequisites that must be available for an individual to perform well in particular content area (e.g., chess playing, piano playing … ). The domains of specialized competencies can be very narrowly defined (e.g., chess competency) or very broadly and openly defined (e.g., diagnostic competencies in medicine). (p. 47)

According to Shavelson (2012),

the construct, competence, as a complex ability closely related to performance can be characterized by the following seven facets: (1) complexity: a complex physical and/or intellectual ability or skill, (2) performance: a capacity not just to «know» but also to be able to do or perform, (3) standardization: tasks, responses, scoring-rubric, testing conditions (etc.) are the same for all individuals, (4) fidelity: tasks provide a high fidelity representation of situations in which, competence is to be demonstrated in the real world, (5) level: performance meets some level of ‘good enough’ to be competent, (6) improvement: the abilities and skills measured can be improved over time by education, training, and deliberative practice, and (7) disposition: personal and social characteristics such as identity, perspective taking, self-regulation, social responsibility that motivate high levels of learning and performance. (p. 78)

This perspective on the construct of competence has consequences for the measurement of competence in educational fields, and particularly in VET programmes and their work-related outcomes. Here, assessment would measure

complex physical and/or intellectual abilities and skills to produce observable performance on a common standardized set of tasks that simulate with high fidelity the performance that are expected to be enacted in the ‘real world’ (‘criterion’) situations to which inferences of competence are to be drawn, with scores reflecting the level of performance (mastery or continuous) on tasks where improvement can be made through dispositions for self-regulation, learning, and deliberative practice. (Shavelson, 2012: 78)

Beyond that, controversy remains regarding how to integrate non-cognitive facets of competence like emotion, motivation and willingness to cope with situational requirements, aspects of self-regulation and so on in the different competence models (see Sembill et al., 2013), and subsequently, how to measure these aspects of competence.

In our research on economic competencies and occupation-related commercial competencies in the field of business and administration, we picked up the idea of bridging the gap between those dichotomous perspectives and sought to develop a modern, computer-based assessment that provides a reliable picture of the business trainees’ competencies at the end of their 3-year-training programmes (Winther et al., 2016b; see also Winther et al., 2016a). The project was funded by the German Ministry of Education and Research (BMBF) and embedded in the funding initiative ‘ASCOT’ pursuing the objective of measuring the vocational competencies that are required for working in high-quality jobs in a changing working world (Bundesministerium für Bildung und Forschung (BMBF), 2011). ASCOT stands for the technology-based assessment of skills and competencies in VET (cf. BMBF, 2011; Oser et al., 2016).

The assessment of economic and commercial competencies in VET

Considerations of different institutional learning settings in VET

Competence measurement in VET requires taking the specifics of the training system and programme into account. Otherwise, it would not be possible to draw conclusions on the influence of learning environments or measure mental representations learned from different curricula and learning opportunities provided by different VET institutions. The so-called ‘dual system’ constitutes the main part of VET programmes in Germany. It brings together theory and practice in a unique combination of workplace and school-based learning. Consequently, an assessment measuring the outcome of VET must refer to different institutional learning settings (companies, schools). Both learning sites offer different organizational and didactical learning opportunities and address different kinds of knowledge and skills. Learning in companies refers to action strategies and procedural knowledge, whereas learning in vocational schools is more closely associated with declarative knowledge (see Figure 3). In particular, it can be assumed that significant differences in the educational structure of the learning content exist that should be reflected in the domain modelling and the assessment concept. Winther (2010: 93) stresses the fact that apprentices’ learning in companies is learning in real work situations, while in school settings learning is based on more or less authentic real-world models or on abstractions of professional reality. Therefore, it can be assumed that while different knowledge representations are developed, both are necessary for addressing professional requirements. Subsequently, the assessment of occupation-related competencies must refer to both learning contexts: to the specifics of learning in the workplace and learning in vocational schools (Winther, 2010: 93ff.; for the link between curriculum, instruction and assessment, see Achtenhagen, 2012; Pellegrino, 2010).

Figure 3.

Different institutional learning settings in VET as reference points for modelling and measuring competencies.^a

This distinction between procedural knowledge and action strategies, on one hand, and declarative and schematic knowledge. on the other, is in line with the position of Gelman and Greeno (1989), who draw a line between interpretative and procedural competencies.

At the international level, there is a broad consensus that assessments in VET contexts should refer to

Goals and contents for teaching and learning in VET, also with respect to different institutional learning contexts;

Conceptual structure of the domain; and

Work procedures and techniques in the respective domain; and

Should use measurement techniques that provide insight into apprentices’ minds.

Reflections on the assessment concept

As mentioned above, competencies are complex dispositions to handle specific situations. This means that the domain of acting must play an important role in modelling and measuring the respective competencies. Furthermore, accepting the triangle of curriculum, instruction and assessment mentioned above, the different learning opportunities and learning characteristics in companies and vocational schools must be carefully considered. According to Messick (1994),

the construction of an assessment should begin by asking what complex of knowledge, skills, or other attributes should be assessed, presumably because they are tied to explicit or implicit objectives of instruction or are otherwise valued by society. Next, what behaviors or performances should reveal those constructs, and what tasks or situations should elicit those behaviors? (p. 16)

An assessment model that seems appropriate to meet these challenges is the evidence-centred assessment design (ECD; see Mislevy and Haertel, 2006; Mislevy and Riconscente, 2006; Figure 4). A central concern of the ECD is the validity of tests, which is taken into account with each of the five ‘layers’. Each layer is characterized by its role in the assessment development process (Mislevy and Haertel, 2006: 4).

Figure 4.

Evidence-centred assessment design.

‘The domain analysis layer requires gathering substantive information about the domain that is to be assessed’ (Mislevy and Haertel, 2006: 5). In our case, the assessment has been designed to measure commercial competencies as well as economic competencies at the vocational level. The domain analyses were focused on information about concepts, terminology, representational forms and ways of interacting in commercial workplaces in companies and on concepts and knowledge that educators in schools have found useful to help apprentices understand and interpret commercial processes, as well as to fulfil the requirements of workplaces, on one hand, and to act as a responsible citizen in an economy-driven world, on the other. A multimethod approach was used to analyse and model the domain:

Analyses of the relevant training standards and curricula, as well as analyses of final exams from the last 3 years;

Analyses of apprentices’ portfolios, of textbooks and workbooks used in vocational schools;

Interviews with trainers in companies and with teachers in vocational school to gather information about the ‘implemented’ curriculum;

Analyses of job profiles and job descriptions;

Observations at workplaces;

Discussions and workshops to judge the idea, the concept, and the representativeness and quality of assessment tasks with trainers in companies and teachers in vocational schools.

In addition to the identification of valued knowledge, skills and abilities for acting professionally in commercial workplaces and responsibly judging economic processes in society as well as for individual economic participation, it is also important to take into account the psychological perspective assumed in instruction and assessment (domain modelling):

In the domain modeling layer we organize the information and relationships discovered in domain analysis into the shape of assessment arguments. There is a transition from the substantive, specialized compendium of knowledge about the target domain to forms that guide the building of the assessment machinery in the layer labeled the conceptual assessment framework. (Mislevy and Haertel, 2006: 7)

The next step, the assessment framework, concerns the technical specifications, particularly the student model (variables that reflect aspects of students’ proficiency), the task models (test environment, forms in which performance will be captured) and the evidence models (concerns the item scoring and thus bridges the gap between student and task model; Mislevy and Haertel, 2006: 14). The assessment implementation includes the finalization of test forms and automated scoring rules. Finally, the assessment delivery layer includes the interaction of students with tasks, their performance and the reports about test results (Mislevy and Haertel, 2006: 20ff.).

Our measurements of VET followed the criterion-sampling approach by defining a domain of real-world tasks drawn from work situations. We worked to keep the responses parallel to those expected in the real world. There are fewer multiple-choice items in the assessment because work does not often present itself as a set of alternatives with only one correct method of action (on the criterion-sampling approach, see Shavelson, 2012: 81ff.).

Modelling economic and commercial competence

The CoBALIT project (Competencies in the Field of Business and Administration – Learning, Instruction and Transition) network developed a test platform for measuring the full range of skills and competencies in the area of business and administration. In particular, the modelling and measurement of occupational (commercial) competencies as well as general economic competencies were calibrated for industrial clerks and freight forwarding and logistics service clerks.

The aim was to learn to what extent trainees are able to understand and plan relevant operations in terms of value creation, sustainability, or internal and external communication. The project models an entire business as well as individual processes that are essential for its operation (cf. Winther et al., 2016b).

Testing refers to specific situations and includes job-related skills in the economic and business management field. In a cross-border effort together with a partner from Switzerland, data are collected that enable not only an international comparison but also the comparison of different commercial occupations (i.e. industrial clerks and freight forwarding and logistics service clerks). Although we have finished the data collection and have completed the initial analyses, we can state that after 4 years of intensive research in this field, we have only just begun the journey rather than nearing its end.

According to Greeno et al. (1984), we distinguish between domain-linked and domain-specific knowledge, which was also considered in the assessment framework I am discussing here (see Winther, 2010: 29ff.). Following Winther (2010: 31), domain-specific competencies are necessary to fulfil the requirements in a specific domain, whereas domain-linked competencies (such as reading, mathematical competencies or economic competencies) can support domain-specific acting (see Figure 5). Domain-linked competencies such as reading or mathematics, as well as general economic competencies (all in the sense of literacy), refer to a less well-defined domain (acting in society and aspects of employability), while the occupation-related competencies such as commercial competencies refer to professional acting in companies.

Figure 5.

Competence model for commercial and economic competencies.

The competence model of the CoBALIT project consists of two central dimensions: domain-linked competencies, such as economic competencies, mathematical and reading competencies, and domain-specific (commercial) competencies. For the area of business and commerce, we followed the approach proposed in the Management Model of St. Gallen, which postulates a systemic, network-like structure of business processes with central nodes and subnets (cf. Dubs, 2012; Rüegg-Stürm, 2005). Furthermore, we differentiated between different kinds of knowledge, that is, procedural and conceptual competence, and the tasks also vary in their cognitive requirements (according to Achtenhagen and Winther, 2009). Additionally, an assessment focuses on at least three different competence aspects (see also Greeno et al., 1984) and their relationships, which are considered in the item development and the scaling and analyses of test scores:

- conceptual competence corresponds to factual knowledge as knowledge of facts, structures, and knowledge nets that can be transmitted into action schemata;

- procedural competence subsumes the application of knowledge, i.e., how to operate with facts, structures, knowledge nets and their corresponding elements, and

- interpretative competence provides connections between features of the problem setting and goals of the learner. (Winther and Achtenhagen, 2009: 91)

All tasks were embedded in an authentic assessment environment, the ALUSIM enterprise (for the first version from a preliminary study, see Achtenhagen and Winther, 2009; Winther, 2010; Winther and Achtenhagen, 2009; for the revised and extended version, see Winther et al., 2016b). Different steps were necessary to develop the virtual ALUSIM enterprise (Achtenhagen and Winther, 2009; Winther et al., 2016b): (1) selection of a representative (real-world) company, (2) construction of a model of the real company (corporate history, balance sheets, selection of products, integrating production sizes and production costs, and information about costumer and supplier), and (3) didactic modelling of typical business cases and value-added processes (complex modelling of (interactive) processes and requirements in workplace reality; simulation of different options for action, valuation of options and decision-making (for an overview about key functions of the virtual enterprise ALUSIM, see Figure 6)). As stated above, the description of the business processes of the virtual ALUSIM company was inspired by the St. Galler Management Model (Rüegg-Stürm, 2005; for the domain model, see Winther, 2010: 88). This model represents enterprises as a complex system embedded in different environments ‘and enables the assignment of all tasks (including those of industrial clerks) that occur in a company’ (Winther et al., 2016a: 153).

Figure 6.

Key functions and processes of the ALUSIM company.

Every module starts with a video vignette, and every task was embedded in different situations for which we used different stimuli, for example, a supplier inquiry email, a telephone request from a consumer or an urgent assignment from the supervisor. Figure 7 shows an item example of the logistical module with the following business situation: in a video clip, the trainer explains to the apprentice that an urgent consumer request for transportation reached the company. She outlines that the truck to Liverpool is not fully loaded and asks the trainee to calculate the remaining loading metres. She ensures the trainee sends all necessary documents by email immediately. The screenshot in Figure 7 shows the preliminary freight list as a pdf document, which the trainee (and the testees as well) must open, and the prefilled answering email with open spaces to fill in the requested information. After calculating the loading metres left on the truck, the trainee is asked to add the necessary information and send this information via email to the trainer. The answer (here, the number of loading metres, which is in a box integrated into the return email) is scored automatically.

Figure 7.

Item example.

Despite intensified research on economic literacy (e.g. the contributions in OECD, 2014; Wuttke et al., 2016), ‘there is still a lack of clarity regarding the exact definitions, and specifically, which components they cover in detail’ (Wuttke et al., 2016: 9). Economic literacy is discussed from a variety of perspectives, for example, ‘individuals as ‘manager[s]’ of their financial affairs’, ‘as well-informed customer[s]’ or ‘as responsible citizen[s]’ (cf. Leumann et al., 2016: 20ff.). While the first and second concepts strengthen an individual perspective, the third approach refers to a systemic dimension. In the study of Schumann and Eberle (2014), both dimensions are integrated. As noted in the first chapter, Schumann and Eberle (2014: 107) define economic competencies as (1) economic knowledge and skills as a prerequisite for solving economic problems (the core dimension of economic competencies), (2) interest in economic problems and motivational orientations with the aim of solving economic problems and (3) attitudes and values that allow economic problems to be deliberated and responsibly resolved. In the CoBALIT study, however, we concentrated on the cognitive aspects of economic competencies. The relationship between knowledge and skills, interests and attitudes was investigated in the OEKOMA project (Economic Competencies in Swiss Upper Secondary Education) by Schumann and Eberle (2014). The researcher discovered a positive correlation between cognitive and non-cognitive aspects of competence such as motivational orientations, interests and attitudes towards knowledge and skills (Schumann and Eberle, 2014: 115).

The conceptualization of economic competencies as an important aspect of citizenship includes important issues from economic science and a few general issues of business administration. The instrument primarily addresses general economic concepts, including micro-economic and macro-economic perspectives, and exchange issues. Four items picked up business issues from a management perspective (production, finance, personnel). As indicated above, it is not easy to define a set of adequate measures for economic competence. After an intensive discussion with school administrators, the curriculum developer and teachers in vocational schools, the research group included the following topics (Eberle et al., 2016):

Euro crises;

Growing government debts;

Energy policy and aspects of sustainability;

Pension scheme and pension policy;

Juvenile indebtedness;

Salaries in top management;

Selected management issues.

Items were drawn from previous studies and empirically validated instruments; in particular, 14 items were selected from the OEKOMA project (Schumann and Eberle, 2014; for the domain modelling, see Figure 8), and 12 items were derived from the German version of the Test of Economic Literacy (TEL; in German, Wirtschaftskundlicher Bildungstest (WBT); cf. Beck and Krumm, 1998). An additional 30 items were developed within the framework of the CoBALIT project (cf. Eberle et al., 2016). In summary, the domain-linked economic competence in the sense of a responsible citizen was measured by 56 items in different answer formats (forced-choice items, single-choice items, open answers to argue a decision and force respondents to outline the advantages and disadvantages of political decisions for different societal groups).

Figure 8.

Domain modelling of economic competencies.

Findings

Sample

The study data were collected in 2014 in the Federal States of Lower Saxony, North Rhine-Westphalia, Hessen and Thuringia with N = 816 testees (185 industrial clerks and 631 freight forwarding and logistic service clerks) who filled out the logistics module. Participation in the survey was optional. A trained test administrator conducted the investigation in vocational schools. The data discussed here have an average response rate across items of approximately 85%. We used a test booklet design to reduce the test load. Nonetheless, the entire assessment took approximately 240 minutes for each participant. The students were tested in the second part of their last training year and shortly before their final examination (Table 1).

Table 1.

Sample.

Type of apprenticeship	Test module
	Test module ‘commercial competence in transport and logistics’	Test module ‘economic competence’	Both test modules
Freight forwarding and logistics service clerks	631	200	199
Industrial clerks	185	467	Not available
Total	826	667	199

Source: Seeber et al. (2015).

Instruments, scaling methods and test quality

In the following sections, selected findings with respect to the economic and commercial competencies of apprentices training as industrial clerks and freight forwarding and logistics service clerks will be discussed. For the occupational (commercial) competencies, I will concentrate on the test scores achieved in the module of transport and logistics. A substantial correlation between this module and the other commercial test modules can be assumed (see the close correlation between different content areas of business administration in previous studies, for example, Förster et al., 2013; Jähnig, 2014; Seeber, 2008). For the test component ‘commercial competence in the field of transport and logistics’, 18 items have been developed and used for the individual model parameters. The test of economic competence consists of 56 complex tasks; 26 items were used to estimate students’ economic competence (30 items with an open answer format are not yet available due to a complex rating process; see also the discussion about the more accurate measure of multiple-choice tests in comparison with open answer format tests, for example, Rauch and Hartig, 2010). The items of both constructs fulfilled at least the qualitative psychometric criteria for probabilistic scaling applied to polytomous items (Fischer and Molenaar, 1995), estimated using the computer programme ACER ConQuest (Wu et al., 2007). The psychometric quality for both constructs is sufficient, although it is better for the economic competence scale than for the commercial competence scale. The EAP/PV scale of the economic items showed a reliability of 0.82, while the scale of commercial competence did not reach an adequate reliability level (EAP/PV reliability = 0.65, EAP/PV reliability is estimated by an item response model, it gives an overall precision of person estimates). Therefore, test scores are limited and merely suitable for individual diagnosis; nevertheless, the scale can provide some information about the strengths and weaknesses of commercial apprentices at the end of their training and about the relationship between domain-specific and domain-linked competencies.

Different model tests were run for each model, for example, unidimensional versus multidimensional models, and a likelihood ratio (LR) test (cf. Andersen, 1973) was also carried out. The LR test was used to compare the fit of the two models. The unidimensional Rasch model showed the best fit to the data of the commercial scale (cf. Liedtke and Seeber, 2015). The construct economic competence is also a unidimensional scale and shows sufficient item fit statistics and a sufficient distribution of item difficulty (cf. Eberle et al., 2016).

For the comparison between the two groups, the industrial clerks and the freight forwarding and logistics service clerks, it was important to check the scale quality with regard to differential item functioning (DIF). DIF is a statistical characteristic of an item that shows the extent to which items might be measuring different abilities in separate groups (e.g. by gender or by specific training programmes). ‘In IRT terms, a scale item displays DIF if examinees with the same latent-trait level have different probabilities of endorsing an item’ (Embretson and Reise, 2000: 319). This is particularly important for our assessment. We expected DIF in the scale measuring commercial competence with regard to the different learning opportunities and learning environments in companies. Industrial clerks are trained in larger industrial companies, whereas freight forwarding and logistics service clerks go through an apprenticeship in small- and medium-sized logistics companies. Furthermore, apprentices in transport and logistics should be more familiar with logistics requirements than industrial clerks according to their weekly work experience during the practical part of the training programme (weekly, they spend approximately 2 days in vocational school and 3 days working in a company). Indeed, our analyses display DIF for the commercial competence scale in favour of the freight forwarding and logistics service clerks. That means apprentices in logistics with the same underlying true ability like industrial clerk apprentices have a higher probability of successfully solving the items of the logistics module (see Table 2). The negative sign of the estimate of the industrial clerk group indicates lower test results on average for this group in the logistic tasks. In summary, 10 of the 18 items used for scaling show DIF, and eight items are suitable for a common scaling. This result is – as noted above – not unexpected.

Table 2.

Results of the DIF analyses.

Type of apprenticeship	Estimate	Error
Test module ‘Commercial competence in transport and logistics’
Freight forwarding and logistics service clerks	0.505	0.071
Industrial clerks	−0.505*	0.071
Chi-square test of parameter equality = 49.96, df = 1
Test module ‘Economic competence’
Freight forwarding and logistics service clerks	0.088	0.017
Industrial clerks	−0.088	0.017
Chi-square test of parameter equality = 25.78, df = 1

Source: Seeber et al. (2015).

A significant DIF between the two groups can also be found for the scale of economic competence in the sense of a responsible citizen. A complete measurement invariance between the different training groups is not given. Nonetheless, the advantages and disadvantages of one or another group – compared to the job-specific commercial skills in the field of transport and logistics – were significantly lower. Thus, a common scale of the construct of economic competence and a comparison between both groups were considered acceptable (see Seeber et al., 2015).

Central findings

The future freight forwarding and logistic service clerks perform better in the test module ‘transport and logistics’ than the future industrial clerks. With regard to the high school–leaving degrees of industrial clerks at the beginning of an apprenticeship (60% with a university entrance degree) and with respect to the substantial overlap of the intended vocational curriculum in the content area of transport and logistics between both groups, this result was initially surprising (see Figure 9).

Figure 9.

Wright map of domain-specific commercial competence in the field of transport and logistics.

Figure 9 shows that items on the logistics tool are quite appropriate for the apprentices in the logistics area, but these items may be slightly too challenging relative to the overall ability level of future industrial clerks. The different average item difficulty for both groups can be explained by the learning opportunities and work experiences in the different types of companies where the young adults are trained.

Our next analysis refers to the relationship between the two groups. There is a significant medium correlation between domain-linked economic competence and domain-specific commercial competence in the logistics test module (r = 0.49). A positive correlation was expected, but it is higher than assumed with regard to the different requirements, the content differences and the different reference points. In the light of the small sample size, the results should be interpreted very carefully (cf. Seeber et al., 2015).

Results drawn from industrial clerks (N = 494) with different test instruments verify the previously reported results (cf. Winther et al., 2016b). In this sample, the scale of commercial competence consists of items in the commercial key functions of planning and controlling, strategic and operative purchasing, sales, marketing and communication. The scale ‘Commercial Competence’ comprises 36 items embedded in the commercial functions mentioned above, and the scale ‘Economic Literacy’ consists of 36 items. In the Wright map shown in Figure 10, one can see that the industrial clerks perform better in ‘economic literacy’ as a domain-linked competence than in the domain-specific commercial area. It should be mentioned that the construct of ‘economic literacy’ was closely defined with regard to employability. An example of an item for the economic literacy scale in the sense of employability is the conversion of foreign currency into Euros for the purpose of a price comparison (cf. Winther et al., 2016b: 63; also Winther et al., 2016a: 154). It is unsurprising that a closer correlation between this definition of economic literacy and commercial competence was observed. The correlation between both latent constructs is substantial with r = 0.898 (cf. Winther et al., 2016b: 63). Winther et al. (2016a) argue that

the boundary between domain-linked items and domain-specific items is not absolute or consistently clear-cut, so it often cannot be determined clearly. There are also questions about the extent to which the context influences basic abilities and what constitutes common economic ability or requires specific training. (p. 155)

A second finding for the commercial competence of industrial clerks in the area of planning and controlling, purchasing, marketing and communication is noticeable: obviously, only a few apprentices reached a professional level of commercial competence at the end of their third training year, or in the case of fast track trainees (apprentices with a university entrance degree), at the end of their second year (see Figure 10). The average item difficulty is much higher than the average commercial ability.

Figure 10.

Wright map of domain-linked economic literacy and domain-specific commercial competence of industrial clerks.

Final remarks

The projects embedded in the funding initiative ‘Technology-based Assessment of Skills and Competencies in Vocational Education and Training (Ascot)’ aimed to contribute to the modelling of job-related competencies in VET and to promote the reliable documentation of performance levels in different occupational fields. Competence models and computer-based authentic assessment tools have been developed and validated. The findings provide new insights into the structure of vocational competencies, into specific competence profiles, into proficiency levels and into some strengths and weaknesses of the training programmes.

We have learned to be careful with constructs and to look closely at the definitions and operationalizations of a certain construct. Despite all the restrictions, it was possible to learn more about the cognitive structure, nature of and relationship between selected domain-linked economic competence and domain-specific commercial competence. Despite these first findings on the proficiency of economic competencies and commercial competencies among young adults at the end of their apprenticeships, many research questions remain open. We strengthened the perspective of domain-specific and domain-linked competencies. Nevertheless, we cannot address how economic and commercial competencies influence each other; instead, we can only report the correlation between both latent constructs. Rosendahl and Straka (2011) found a substantial correlation between both latent constructs at the end of the first training year for apprentices studying to become bank clerks in a longitudinal study. At the latter measurement points (second and third training year), they could observe a declining statistical correlation. Unfortunately, they could provide only a limited explanation of these findings, and they were unable to explain the results against the background of learning opportunities in companies and vocational schools. The assumption of Winther (2010: 31ff.; 79ff.) that domain-linked competencies will or can support the development of domain-specific competencies remains unanswered. A cross-sectional study like the CoBALIT project cannot explain the mutual influence of the respective competence constructs. Is there a positive impact on commercial competence when economic competencies will be explicitly developed during the apprenticeship? Does the development of commercial competence influence an apprentice’s economic thinking as a responsible citizen and consumer? These questions can only be investigated by a longitudinal study.

In our study, however, we specifically considered knowledge and abilities, not attitude-related aspects of vocational competence in terms of motivation and volition or moral values. Thus, the question about the relationship between economic thinking; attitudes towards different economic decisions, strategies, and corporate or individual behaviour, as well as moral values; and judgements with regard to acting as a responsible citizen in society and as an employee in companies (Beck, 1989) have been rather widely neglected by educational research in VET.

Regarding the lack of scientific knowledge about the effects of VET programmes, the ASCOT initiative and our embedded project are an important step to gain further insight into the characteristics of learners’ competencies for both the development of citizenship and the development of domain-specific, job-related knowledge, skills and attitudes. Against this background, our study provides hints with respect to the strengths and weaknesses of the training and can thus contribute to an evidence-based discussion about developing vocational training and strengthening its effectiveness.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

References

Achtenhagen

(2012) The curriculum-instruction-assessment triad. Empirical Research in Vocational Education and Training 4(1): 5–25.

Achtenhagen

Winther

(2009) Konstruktvalidität von Simulationsaufgaben: Computergestützte Messung berufsfachlicher Kompetenz – am Beispiel der Ausbildung von Industriekaufleuten. Abschlussbericht für das Bundesministerium für Bildung und Forschung, Berlin and Bonn. Available at: http://www.bmbf.de/pub/Endbericht_BMBF09.pdf

Andersen

(1973) A goodness of fit test for the Rasch model. Psychometrika 38(1): 123–140.

Anderson

Krathwohl

(2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. New York: Longman.

Baethge

(2010) Ein europäisches Berufsbildungs-PISA als methodisches und politisches Projekt. In: Münk

Schelten

(eds) Kompetenzvermittlung für die Berufsbildung. Bielefeld: W. Bertelsmann, pp.19–36.

Beck

(1989) ‘Ökonomische Bildung’ – Zur Anatomie eines wirtschaftspädagogischen Begriffs. Zeitschrift für Berufs- und Wirtschaftspädagogik 85: 579–596.

Beck

Krumm

(1998) Wirtschaftskundlicher Bildungs-Test (WBT). Handanweisung. Göttingen: Hogrefe.

Björnavold

(2000). Assessing NonFormal Learning: European Developments and Paradoxes. CEDEFOP, Thessaloniki.

Blömeke

Gustafsson

J. E.

Shavelson

R. J.

(2015). Beyond dichotomies: Competence viewed as a continuum. Zeitschrift für Psychologie 223(1), pp. 3–13.

10.

Bundesministerium für Bildung und Forschung (BMBF) (2011) Vocational skills and competencies made visible The ASCOT research initiative. Available at: http://ascot-vet.net/_media/ascot__MASTER_Broschuere_Projekte_EN_V06.pdf

11.

CEDEFOP (Ed.) (2003). Terminology of Vocational Training Policy: A Multilingual Glossary for an Enlarged Europe. Available at http://www.europass.cedefop.eu.int/img/dynamic/c-313/cv-1_en_USglossary_4030_6k.pdf

12.

CEDEFOP (2014) Matching skills and jobs in Europe. Insights from CEDEFOP’S European skills and jobs survey. Available at http://www.cedefop.europa.eu/de/publications-and-resources/publications/8088

13.

Dubs

(2012) Das St. Galler Management-Modell: Ganzheitliches unternehmerisches Denken. Linz: Trauner Verlag.

14.

Dubs

(2013) Ökonomische Allgemeinbildung in der Sekundarstufe II. In: Retzmann

(ed.) Ökonomische Allgemeinbildung in der Sekundarstufe II. Konzepte, Analysen und empirische Befunde. Schwalbach am Taunus: Wochenschau Verlag, pp.11–25.

15.

Eberle

Schumann

Kaufmann

. (2016) Modellierung und Messung wirtschaftsbürgerlicher Kompetenz von kaufmännischen Auszubildenden in der Schweiz und in Deutschland. In: Oser

Landenberger

Beck

(eds) Technologiebasierte Kompetenzmessung in der beruflichen Bildung – Ergebnisse aus der BMBF-Förderinitiative ASCOT. Reihe Wirtschaft – Beruf – Ethik (Bd. 32). Bielefeld: W. Bertelsmann, pp.93–117.

16.

Embretson

Reise

(2000) Item Response Theory for Psychologists. Mahwah, NJ: Lawrence Erlbaum.

17.

Fischer

Molenaar

(1995) Rasch Models – Foundations, Recent Developments, and Applications. New York: Springer.

18.

Förster

Zlatkin-Troitschanskaia

Brückner

. (2013) WiwiKom – Modeling and measuring competencies in business and economics among students and graduates by adapting and further developing existing American and Mexican measuring instruments. In: Blömeke

Zlatkin-Troitschanskaia

(eds) The German Funding Initiative ‘Modeling and Measuring Competencies in Higher Education’ (KoKoHs Working Papers, 3). Berlin and Mainz: Humboldt University & Johannes Gutenberg University, pp.19–21.

19.

Gelman

Greeno

(1989) On the nature of competence: Principles for understanding in a domain. In: Resnick

(ed.) Knowing, Learning and Instruction: Essays in Honor of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum, pp.125–186.

20.

Greeno

Riley

Gelman

(1984) Conceptual competence and children’s counting. Cognitive Psychology 16: 94–143.

21.

Hartig

Klieme

Leutner

(eds) (2008) Assessment of Competencies in Educational Contexts: State of the Art and Future Prospects. Göttingen: Hogrefe & Huber.

22.

Jähnig

(2014) Die Messung betriebswirtschaftlichen Wissens von Studierenden. Eine quantitativempirische Untersuchung situativer Testaufgaben (Reihe Berufsbildung, Arbeit und Innovation). Bielefeld: Bertelsmann Verlag.

23.

Leumann

Heumann

Syed

. (2016) Towards a comprehensive financial literacy framework: Voices from stakeholders in European vocational education and training. In: Wuttke

Seifried

Schumann

(eds) Economic Competence and Financial Literacy of Young Adults: Status and Challenges (Research in Vocational Education, vol. 3). Leverkusen: Barbara Budrich, pp.19–39.

24.

Liedtke

Seeber

(2015) Modellgeltungstests und Einflussfaktoren auf differentielle Item funktionen in einem computergestützten Assessment für kaufmännische Berufe. Zeitschrift für Berufs- und Wirtschaftspädagogik, 111/2015/2. Stuttgart: Steiner Verlag, pp.242–267.

25.

Lisbon-to-Copenhagen-to-Maastricht Consortium Partners (eds) (2004) Achieving the Lisbon Goal: The contribution of VET – Final Report to the European Commission. Available at: http://europa.eu.int/comm/education/policies/2010/studies/maastricht_en.pdf

26.

Messick

(1994) Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist 50(9): 741–749.

27.

Mislevy

Haertel

(2006) Implications of Evidence-Centered Design for Educational Testing. Menlo Park, CA: SRI International and University of Maryland.

28.

Mislevy

Riconscente

(2006) Evidence-centered assessment design: Layers, concepts, and terminology. In: Downing

Haladyna

(eds) Handbook of Test Development. Mahwah, NJ: Lawrence Erlbaum, pp.61–90.

29.

Mulder

Weigel

Collins

(2006) The concept of competence in the development of vocational education and training in selected EU member states. A critical analysis. Journal of Vocational Education and Training 59(1): 65–85.

30.

Nickolaus

Seeber

(2013) Berufliche Kompetenzen: Modellierungen und diagnostische Verfahren. In: Frey

Lissmann

Schwarz

(eds) Handbuch Berufspädagogische Diagnostik. Weinheim and Basel: Springer, pp.155–180.

31.

OECD (2012) Learning for Jobs. OECD Publishing. Available at: http://www.oecd.org/education/innovation-education/learningforjobs.htm

32.

OECD (2014) PISA 2012 Results: Students and Money: Financial Literacy Skills for the 21st Century, vol. VI: OECD Publishing. Available at: http://dx.doi.org/10.1787/9789264208094-en

33.

OECD (2015) OECD Skills Outlook 2015. Youth, Skills and Employability. OECD Publishing. Available at: http://www.oecd.org/education/oecd-skills-outlook-2015-9789264234178-en.htm

34.

Oser

Landenberger

Beck

(eds) (2016) Technologiebasierte Kompetenzmessung in der beruflichen Bildung – Resultate aus dem Forschungsprogramm ASCOT. Bielefeld: Bertelsmann.

35.

Pellegrino

(2010) The design of an assessment system for the race to the top: A learning sciences perspective on issues of growth and measurement. Educational Testing Service (ETS). Available at: http://www.k12center.org/rsc/pdf/PellegrinoPresenterSession1.pdf

36.

Potgieter

Van Der Merwe

(2002) Assessment in the workplace: A competency-based approach. Journal of Industrial Psychology 28(1): 60–66.

37.

Rauch

Hartig

(2010) Multiple-choice versus open-ended test format of reading test items: A two-dimensional IRT analysis. Psychological Test and Assessment Modeling 52: 354–379.

38.

Rosendahl

Straka

(2011) Effekte personaler, schulischer und betrieblicher Bedingungen auf berufliche Kompetenzen von Bankkaufleuten während der dualen Ausbildung. ITB Forschungsberichte 51/2011. Available at: http://elib.suub.uni-bremen.de/edocs/00102039-1pdf

39.

Rüegg-Stürm

(2005) Das neue St. Galler Management-Modell. Bern: Haupt Verlag.

40.

Rychen

Salganik

(wds) (2003) Key Competencies for a Successful Life and a Well-Functioning Society. Göttingen: Hogrefe & Huber.

41.

Schoenfeld

(2010) How We Think: A Theory of Goaloriented Decision Making and Its Educational Applications. New York: Routledge.

42.

Schumann

Eberle

(2014) Ökonomische Kompetenzen von Lernenden am Ende der Sekundarstufe II. Zeitschrift für Erziehungswissenschaft 17(1): 103–126.

43.

Seeber

(2008) Ansätze zur Modellierung beruflicher Fachkompetenz in kaufmännischen Ausbildungsberufen. Zeitschrift für Berufs- und Wirtschaftspädagogik, 104 Band, Heft 1(2008), Stuttgart: Franz Steiner, pp.74–97.

44.

Seeber

Schumann

Nickolaus

(2015) Ökonomische Kompetenzen: Konzeptuelle Grundlagen und empirische Befunde. In: Weißeno

Schelle

(eds) Empirische Forschung in gesellschaftswissenschaftlichen Fachdidaktiken – Ergebnisse und Perspektiven. Heidelberg and Berlin: Springer, pp.169–184.

45.

Sembill

Rausch

Kögler

(2013) Non-cognitive facets of competence: Theoretical foundations and implications for measurement. In: Beck

Zlatkin-Troitschanskaia

(eds) From Diagnostics to Learning Success (Proceedings in Vocational Education and Training). Rotterdam: Sense Publishers, pp.199–211.

46.

Shavelson

(2012) Assessing business-planning competence using the Collegiate Learning Assessment as a prototype. Empirical Research in Vocational Education and Training 4(1): 77–90.

47.

Shavelson

Ruiz-Primo

Wiley

(2005) Windows into the mind. Higher Education 49(4): 413–430.

48.

Weinert

(2001) Concept of competence: A conceptual clarification. In: Rychen

Salganik

(eds) Defining and Selecting Key Competencies. Kirkland, WA; Toronto, ON, Canada; Bern; Göttingen: Hogrefe & Huber Publishers, pp.45–65.

49.

Winther

(2010) Kompetenzmessung in der beruflichen Bildung. Bielefeld: Bertelsmann.

50.

Winther

(2013) Konstruktion technologiebasierter Kompetenzmessungen im kaufmännischen Bereich – Die Projekte CoBALIT und DomPL-IK. Vortrag im Rahmen der Erweiterten Fachbeiratssitzung der ASCOT-Forschungsinitiative am 26 November. Berlin: BMBF.

51.

Winther

Achtenhagen

(2009) Measurement of vocational competencies – A contribution to an international large-scale assessment on vocational education and training. Empirical Research in Vocational Education and Training 1: 85–102.

52.

Winther

Festner

Sangmeister

. (2016a) Facing commercial competence: Modeling domain-linked and domain-specific competence as key elements of vocational development. In: Wuttke

Seifried

Schumann

(eds) Economic Competence and Financial Literacy of Young Adults: Status and Challenges (Research in Vocational Education, vol. 3). Leverkusen: Barbara Budrich, pp.149–163.

53.

Winther

Seeber

Festner

. (2016b) Large scale-assessments in der kaufmännischen Berufsbildung (CoBALIT). In: Oser

Landenberger

Beck

(eds) Technologiebasierte Kompetenzmessung in der beruflichen Bildung – Resultate aus dem Forschungsprogramm ASCOT. Bielefeld: Bertelsmann, pp.55–73.

54.

Wolter Ä (2013) Studium und Beruf im Wandel: Von der akademischen Persönlichkeitsbildung zur Beschäftigungsfähigkeit? In: Buck

Kabaum

(eds) Ideen und Realitäten von Universitäten. Frankfurt: Peter Lang, pp.169–198.

55.

Adams

Wilson

. (2007) ACERConQuest Version 2: Generalised Item Response Modelling Software. Camberwell, VIC, Australia: Australian Council for Educational Research.

56.

Wuttke

Schumann

Seifried

(2016) Editorial: Economic competence and financial literacy of young adults: Status and challenges. In: Wuttke

Seifried

Schumann

(eds) Economic Competence and Financial Literacy of Young Adults: Status and Challenges (Research in Vocational Education, vol. 3). Leverkusen: Barbara Budrich, pp.9–16.

57.

Wuttke

Seifried

Schumann

(eds) (2016) Economic Competence and Financial Literacy of Young Adults: Status and Challenges (Research in Vocational Education, vol. 3). Leverkusen: Barbara Budrich.

Economic competencies and situation-specific commercial competencies: Reflections on conceptualization and measurement

Abstract

Keywords

Introduction

Approaches to the concept of competence in VET

The assessment of economic and commercial competencies in VET

Considerations of different institutional learning settings in VET

Reflections on the assessment concept

Modelling economic and commercial competence

Findings

Sample

Instruments, scaling methods and test quality

Central findings

Final remarks

Footnotes

Declaration of Conflicting Interests

Funding

References