Abstract
Is there a trend towards internationalization of educational research in Europe? Educational research is said to follow a tradition of nationally oriented studies and interventions supported by a national publication culture. Publications are a suitable source of empirical analysis of research output, as they reflect results, emergence and impact of research. This study focuses on publication based bibliometric indicators, which represent measurable characteristics of international orientation of research publications and which can be surveyed in time course. Being aware that the Web of Science (WoS) databases cover a crucial but rather limited proportion of the worldwide educational research output, this study provides bibliometric insights into the development of national publication outputs in educational research in the WoS and what idiosyncrasies are revealed for European countries, into the role of English as a publication language, into the trend towards transnational co-authorship as an indicator of international cooperation, and into citation frequencies as a measurement of research communication or research impact.
Keywords
Introduction
Is there a trend towards internationalization of educational research in Europe? The challenge of internationalization of educational research is justified by the overall process of economic globalization. Owing to ambitions of the European Union (EU) as well as international assessments, educational systems have faced developments of harmonization and standardization. In order to support the process of internationalization in educational research, political funding programs and scholarly societies 1 have been set up in recent decades. Consequently, many national scientific administration bodies use specific indicators of internationalization when evaluating research. The severity of use of such indicators varies from nation to nation and across disciplines. Educational research, a discipline at the interface of social sciences and humanities, is said to follow a tradition of nationally oriented studies and interventions supported by a national publication culture (Diem and Wolter, 2013). There is still not much evidence on to what extent educational research followed a process of internationalization and if there are major differences on national level as a result of traditional publication cultures (Knaupp et al., 2014).
Publications are a suitable source of empirical analysis of research output, as they reflect results, emergence and outreach of research. This study focuses on publication based indicators, which represent measurable characteristics of international orientation of research publications and which can be analyzed in time course:
The number of publications in journals which are peer-reviewed with regard to relevance for the international research community (which is one of the selection criteria of the Web of Science (WoS) 2 ).
The number of transnational co-authorships as a characteristic of international cooperation.
The language of publications and the growth of English-language publications, as an indicator that authors of non-English-speaking countries support the international awareness of their publications.
The number of citations as a measure of reception or impact of publications.
Thus, internationalization is operationalized according to the criteria of an increased number of publications in the WoS, a growth of transnational co-authored publications and a rising number of English-language publications in educational research.
All these characteristics can be evaluated on national aggregations, if suitable databases are used. These ‘classical’ bibliometric analyses need to be based on a set of special metadata that can only be provided by bibliometric databases on a sufficient level of standardization, encompassing not only metadata recording citations, but also controlled metadata for the names of all authors and their institutional affiliation.
Already in 1998, it was stressed that the transformation of the disciplinary field of educational sciences may be fruitfully analyzed by means of bibliometrics (Schriewer, 2000). According to Schriewer and Tenorth, we ‘need more analyses of educational knowledge using bibliometric methods such as citation indices or quantifying analyses of communication patterns both within Education and between the discipline and its environment’ (Schriewer and Tenorth, 1998: xv).
Bibliometrics are used descriptively to determine publishing patterns, relationships among publishing entities, and the impact in terms of citation numbers of published work. Lately, bibliometric analyses have increasingly become a part of evaluations and ratings of research performance. The rating system in academia builds gradually upon bibliometric indicators to measure the productivity of scientific staff. Bibliometric analyses in general but especially the bibliometric assessment of individual scholars have been criticized in many ways right from the start (e.g. Moed, 2005: 25). The community of bibliometric professionals has replied by claiming the observance of ethical standards; the most recent declaration being The Leiden Manifesto for research metrics 3 . These standards accentuate mainly that bibliometric analyses need to be applied preferably on highly aggregated populations and require cautious interpretation against the backdrop of field-specific characteristics. In compliance with these ethical restrictions, bibliometric analyses are still convincing as an instrument to observe large numbers of publications. Diem and Wolter (2013) justify that the preference for bibliometrics is rooted in the compact presentation of data material.
The present study focuses on the very high aggregation level of countries and exclusively on the field of educational research. But educational research is a distinctly multi-disciplinary field, which shows considerable variations in publication cultures. These variations can be based in national traditions, but also in sub-disciplinary developments generating specific forms of communication and methodological conventions (Schriewer, 1990). Especially the discipline of educational psychology is said to have reached a high level of cross-national autonomy: ‘While education research incorporates rather diverse and culturally restricted research cultures, (educational) psychology seems to display rather unified research cultures across national borders.’ (Knaupp et al., 2014: 84). This study makes an effort to capture the comprehensive field of research aiming at educational phenomena irrespective of specific publication cultures.
In this study, we make use of the standard bibliometric databases in the WoS by Thomson Reuters. The presented results were part of a much broader German project which aimed at the development and application of indicators for monitoring projects and publications of educational researchers (Botte et al., 2015). For this article, the focus was extended to European countries and data were partly updated.
Before delineating our research questions, we will look at preliminary bibliometric studies in the field of educational research and their limitations, and we will argue why we chose to focus on the WoS data for the purpose of this study.
Coverage of educational research in the WoS
The study builds upon the bibliometric database of the Competence Center for Bibliometrics hosted at the former Institute for Research Information and Quality Assurance 4 , now the German Centre for Higher Education Research and Science Studies 5 . This database includes tagged data as recorded in the WoS on the basis of a license. Starting in 1961 with the Science Citation Index (SCI) by Eugene Garfield, the WoS incorporated the Social Sciences Citations Index (SSCI) in 1973, followed by the Arts & Humanities Citation Index (A&HCI) in 1978. The WoS mainly references peer-reviewed journal articles which are the primary mode of communication in the natural sciences. Book chapters, book reviews and proceedings are covered to a small degree only. A number of authors have stressed that journals are not the primary vehicle for scholarly communication in humanities and many social sciences disciplines, where book-based formats play a major role (e.g. Andersen, 2000; Glänzel and Schoepflin, 1999; Hicks, 1999).
Although the WoS primarily covers journals, these are nonetheless covered selectively. The WoS journal selection process 6 refers to the assumption that only journals which are run in agreement with international publishing standards publish the internationally most relevant research. These publishing criteria comprise a standardized form of peer review, timely publication, and English as the international publication language. Thomson Reuters’ selection criteria were developed in accordance with the publication culture practiced in most natural sciences disciplines. In social sciences and humanities, many journals follow editorial cultures that differ from the sciences. Consequently, the coverage of journals from the social sciences remains at issue in the WoS.
The insufficient representation of European educational research journals in the SSCI was underlined by Botte (2007). Togia and Tsigilis (2006) found that while a total of 1,226 active refereed scholarly journals are listed under the subject heading ‘Education’ in Ulrich’s Periodicals Directory, only 136 were included in the WoS in 2004. Earp (2010) counted 206 WoS journals from a total of 4,268 scholarly journals allocated to the field of ‘Education’ in Ulrich’s. Togia and Tsigilis (2006) stressed the limitations of impact factors of educational journals and showed that education journals included in the WoS represent only about 11% of the active academic journals. In a German study by Dees (2008) based on a complete survey on publications of university and non-university educational research institutes in Germany (2004–2006), Dees found that journal articles accounted for only one-third of all publications. Only 13.5% of the journal articles and consequently merely 4.5% of all analyzed documents were published in the WoS-indexed journals. A small amount of 6.6% (university institutes) responded and 13% (non-university research institutes) of publications was in English. 7
These low shares indicate that the WoS is not a comprehensive database for journals relevant to European educational research. However, the WoS has actively reduced the general predominance of United States journals in recent years. 8 To meet the demands of an international research community, Thomson Reuters reviewed around 10,000 ‘regional journals’ for potential inclusion (Testa, 2011), most of them belonging to the social sciences and humanities. Regional journals are defined as journals published outside the US or United Kingdom (UK), presenting research which is mainly targeted at a national audience. From 2007 on, approximately 1,600 of these ‘regional’ journals meeting Thomson Reuters’ criteria were continually integrated into the database (Testa, 2011). Thus, the WoS critique of anglocentric aspects by Kehm and Teichler (2007) needs qualification. The effects are evident in terms of the emergence of new countries, languages represented in educational research literature, and sudden increases in publication numbers. Thus, the WoS is gradually turning into a representative albeit incomplete source of internationally communicated research articles, also in the field of educational research.
Moreover, the WoS offers a set of genuine bibliometric instruments and valuable information on the publication and citation behavior of educational researchers, which indicate, for example, collaboration patterns and characteristics of internationalization. Not least, quantitative bibliometric analyses are replicable and relatively inexpensive. In order to reduce drawbacks, such as the bias towards sub-disciplinary differences in publication and citation cultures, numeric data require interpretation in context (Botte, 2007).
Publication and citation behavior in educational research
Citation analyses and numeric indicators were originally created and applied in the natural sciences. The specific communication practice in social science disciplines is different and the differences have been analyzed in terms of publication output of researchers and citation behavior. According to Hicks (1999), social science research is characterized by a more national orientation which includes the target audience as well as the received citations which are indeed mainly restricted to the respective country or language area. Line (1999) argued that social sciences lack international concepts and terms, showing preference to publish in the native language and a great tolerance to duplicate previous research.
Because of resulting disadvantages in coverage and performance of bibliometric analyses of educational research publications, studies on bibliometrics in educational research were rather published by educational researchers than by bibliometricians. At the beginning, these were mainly qualitative studies focusing on a set of journals. Following Tight (2008, 2009), research gradually developed into more quantitative bibliometric studies: Tight primarily compared US–American journals with international ones, identified highly cited papers in educational research and evaluated the productivity of authors, collaborations and research methods applied.
According to Goodyear et al. (2009: 702), educational research is ‘inherently interdisciplinary’ and diverse in content. The authors examined whether a core of journals in educational research can be identified. Their results show a high diversity of journals and content, and raise the question to what degree educational researchers share the same knowledge base. Earp (2010) analyzed and discussed the impact of The Journal of Higher Education (JHE) between 1998 and 2002. He concluded that JHE had a much stronger influence on the field than the traditional journal impact factor could express (which is based on a two-year citation window only). Budd and Magnuson (2010) investigated citations in papers from three core journals in higher education: Research in Higher Education; Review of Higher Education; and JHE. They found that authors most often cite journal articles (45.5%), followed by books (26.3%), and book chapters (11.0%).
Nowadays, international comparative studies of educational systems are essential for reasons of benchmarking and the identification of ‘best practices’ and ‘best solutions’ (Teichler 2014: 397). Teichler argues that comparative perspectives are important to deconstruct the national perspective and to foster an international view on educational research in order to identify effective education policies (OECD, 2013). The study by Kosmützky and Krücken (2014) confirms the significance of international comparative research in a globalized higher education landscape.
Kuzhabekova et al. (2015) applied bibliometric methods to identify key individuals, key institutions, and main countries contributing to research in international higher education. They studied patterns of connectivity among authors, journals and topics showing that despite a high number of institutions in international higher education around the world, only 11.3% of articles are authored by researchers from two or more countries.
Therefore, notwithstanding a clearly genuine publication culture of disciplines contributing to social and educational research, the WoS is still the main instrument used for transnational comparisons of publication output. Lately, the database Scopus by Elsevier has also been increasingly used for alternative or supplementary bibliometric analyses. A set of Scopus data for the field of educational research was also created and analyzed within the German monitoring project underlying this publication (Botte et al., 2015). We found out that differences between WoS and Scopus were only quantitative, not qualitative. There are indisputably more non-English publications in Scopus, but the trends and effects of using English or not as publication language are overall the same. For the benefit of clarity, we chose to limit the presentation of results to WoS data.
Research questions
This article explores whether bibliometric data indicate that trends in educational research have led to more internationally communicated publication activities in journals covered by WoS. In order to be able to look at sub-disciplinary differences we refer to an enhanced delineation of educational research publications.
Being aware that the WoS indexes cover a crucial but rather limited proportion of the worldwide educational research output, this study provides bibliometric insights into the following questions:
What do bibliometric indicators tell about the development of national publication outputs in educational research in WoS and what idiosyncrasies are revealed for European countries?
Is there a trend towards transnational co-authorship as an indicator of international cooperation?
Is there a change of the role of English as a publication language?
How do citation measures develop over time and how are they connected to the publication language?
Data and methods
The analyses are based on a sophisticated field delineation encompassing only publications that deal with educational research. Most bibliometric studies apply a so called macro-level delineation, which refers to the selection of complete sets of journals in a subject area. The definition of subject areas to which journals belong builds upon the proper classification system of a database. In WoS we can associate the following four Subject Categories with relevant publications in educational research:
Education and educational research
Education, scientific disciplines
Education, special
Psychology, educational
The education and educational research category might not seem problematic but it does not cover all resources ranging from theoretical to applied research. According to Laurens et al. (2010), macro-level delineation hardly results in a satisfying relation of precision and noise for broad or interdisciplinary fields of research. Thus, a heterogeneous field such as educational research requires more sophisticated field delineation than the insufficient macro-level delineation. In this study, we made use of a hybrid citation–lexical delineation as recommended in Zitt and Bassecoulard, (2006) and Laurens et al. (2010). This approach does not only work on the basis of relevant journals, but on the basis of articles implying that articles from non-educational journals are also retrieved and included.
In a first step, we drew on existing journal lists compiled by experts. In the Science-Metrix journal classification, 284 Education journals are listed. 9 The European Reference Index for the Humanities provided another list of journals. As of July 2013, 531 journals are listed within the category Educational Research. 10 Finally, the SCImago journal list was used, where the category Education comprises 573 journals (July, 2013). 11 These three lists of journals are overlapping to some extent. The match of the journals in WoS resulted in 436 active journals. Since not all of the journal publications deal with educational research, a lexical approach was required to narrow the publication set and to increase precision. Valuable lexical terms were generated on the basis of the ERIC Thesaurus. 12 Additional lexical terms were taken from German projects on Educational Research that were published in journals covered by the WoS. Altogether, 560 lexical terms (see Appendix) proved to be valuable for a final retrieval in the WoS. Article titles and keywords attached to articles were scanned via Structured Query Language queries for valuable lexical terms. In a first step, this was done within the set of potentially relevant educational research journals. The search for relevant educational research publications across the whole WoS database (besides the potentially relevant educational research journals) was processed in a stricter lexical match, because of the missing context. The strictness grows with the number of lexical terms to occur either in the article title or the keywords provided. To ensure that publications with short article titles or missing keywords do not go unnoticed, the citation-based approach retrieved the publications that are either co-cited or bibliographically coupled with the seed publications in educational research (see Zitt and Bassecoulard, 2006).
In contrast to the field delineation described in Botte et al. (2015), for this publication an additional blacklisting was applied to eliminate articles that most of all deal with health, nursing, and medicine. The decision on whether or not to include these latter publications may depend on the definition of educational research. Our goal is to work with a rather narrow definition of educational research to prevent false-positive hits that result from the lexical term ‘education’, as it is used in many articles on physical education, nurse education or AIDS education which usually do not represent a publication culture rooted in educational research.
The process of field delineation was informed and controlled by experts of the German Institute for International Educational Research. In terms of information retrieval, the final set has a precision rate of 99%. It includes 92,820 journal publications from the SCI, SSCI and A&HCI within the period 2002–2013.
In order to operationalize ‘internationalization of educational research’ we use the following characteristics of publications:
the number of publications in the WoS;
the number of transnational co-authorships;
the share and development of English language publications; and
the number of citations as a measure of reception of publications.
Results and discussion
The following presentation of results has to be interpreted in the light of the previously mentioned restrictions:
The WoS does not cover all relevant educational research publications, but it represents a selection of peer-reviewed publications which were evaluated as important to the international community. As these publications are mostly written in English, they generally have a better chance of being recognized outside the country of origin than most other publications in the field.
Bibliometrics are based on quantitative statistical data. It is obvious that measurements of numbers (‘productivity’) generally depend highly on the size of a country and its number of publishing (educational) researchers.
Countries with English as the national communication language are advantaged when we compare publishing in international proliferating journals.
Numbers of citations depend on the coverage of the WoS and are gathered for the delineated set of educational research publications.
Co-authored publications in the WoS are used as a proxy of collaboration. However, co-publishing is only one form of scientific cooperation. Hence, the following results are restricted to the microcosm of WoS, but we assume that this microcosm is a biased yet meaningful representation of the international scholarly communication in educational research.
Publication output and growth
This section deals with the first research question about the development of national publication outputs in educational research in the WoS. We focus our analyses on the most productive European countries in terms of WoS publications in educational research, as the numbers of publications for other countries are not resilient enough. Note that the identification of the most active countries not only depends on their general publication activity, but also on Thomson Reuters’ journal policy. Since the Institute for Scientific Information, the predecessor of the WoS, has its origin in the US, the majority of publications in the database as well as in the delineated field of educational research are of US-origin. The number of publications involving contributions from US researchers amounts to 44.1% of all publications in the years 2002–2013. The US is followed by the UK (12.3%) research landscape, which ranks first among all European countries. Other important non-European countries that contributed to the journal research output in educational research between 2002 and 2013 are Australia (6.1%) and Canada (5.5%).
Table 1 lists the 20 most productive European countries in the delineated set of educational research publications in 2002–2013. The countries are displayed in descendant order according to their total number of journal publications during the whole period (last column). The UK ranks first and is followed by Germany, Spain, Turkey, and the Netherlands which all have more than 3,000 publications in educational research.
Publication counts of
Besides the predominance of the English-speaking UK the order of countries is expectedly influenced by their population size and/or their economic power. But there are exceptions to this rule. According to the criteria of this bibliometric study the Dutch publication outputs are most remarkable. As a relatively small country, the Netherlands outperforms other non-English speaking countries in respect of all indicators in our study. Across all four periods Dutch educational researchers publish about four times more in the WoS than, for example, French researchers. The fact that the Dutch research community is much more internationally oriented than the bigger French community may not come as a surprise to experts. As both countries are disadvantaged by a lack of ‘regional’ educational research journals in the WoS (see Figure 1 and compare with Diem and Wolter, 2013: 104), the magnitude by which these different publication cultures are reflected in the WoS is astonishing.
(a) Web of Science (WoS) language distribution of publications in the WoS of the 20 most productive European countries in percentages; and (b) share of English-language publications in percentages for 12 countries publishing in languages other than English.
Table 1 also presents publication figures in three-year periods to show how they developed over time for these 20 countries. All 20 European countries show an extraordinary growth in publication output over the 12 years, which reaches much beyond the growth in the number of journals in this period (about 1,600 new journals from a total of about 12,000 WoS journals in 2014).
It is striking that Turkey, Spain, Belgium and most Scandinavian countries increased their publication numbers above average. The countries showing extreme growth, Spain and Turkey, have obviously profited most from the incorporation of new regional journals into the WoS after 2007 (see Figure 1). As this effect only counts for a small amount of the tremendous growth in the case of Turkey and Spain, the growth is probably based on a national strategy of internationalization of educational research. Also, there is a significant difference between the publication numbers of Switzerland and Austria, countries which are comparable in terms of population and culture. But the smaller the absolute figures are the more arguable are generalizing interpretations.
Besides special efforts of some countries, we can assume that the EU’s policy towards more harmonization and coordination in education evoked strong motives for more transnational research activities which were taken up by all the 20 European educational research communities listed in Table 1. Whereas the share of European publications in our set of WoS publications increased, the share of US publications declined from 57% in 2002 to 39% in 2012.
The ranking was limited to most productive countries to avoid that single outliers may have non-representative influence on the comparison of countries. Nevertheless, our data support the conclusion that European educational research communities strived successfully for more publications in the internationally visible journals of the WoS, at least since the year 2002.
Internationally collaborative publications
After looking at the general trend of educational research publications in the WoS the next two sections deal with the second research question on the development and effect of internationally cooperative publications in the field.
It is one of the special features of WoS that collaboration in terms of international co-authorships can be measured. The countable indicator of international collaboration is a paper with affiliations from more than one country. Table 2 illustrates the growing ratio of papers that were internationally co-authored. The results are based on the 20 European countries (see Table 1) with the highest numbers of publications in WoS. It is significant that not only the number but also the share of internationally collaborative publications has increased over the years. Even if the share has not risen continuously, there is a clear tendency towards internationalization starting in 2002 with 14.1% internationally collaborative publications leading to 21.7% in 2013.
Growth of internationally collaborative publications of the 20 most productive European countries between 2002 and 2013 in the Web of Science.
Since collaboration habits differ among countries, Table 3 provides an overview of the evolution of internationally collaborative publications for the 20 European countries. The countries are displayed in a descendant order according to their overall share of internationally co-authored publications within the 2002–2013 period.
Growth of the share of internationally collaborative publications between 2002 and 2013 in percentages in the Web of Science. Countries are ordered according to their share of internationally co-authored publications.
Switzerland ranks first, where nearly every second publication related to educational research is international. It is followed by Austria, where the share of international publications has rapidly increased over the years. Given the low numbers of publications the values fluctuate strongly over the period. The high shares of internationally co-authored publications for Austria and Switzerland are mainly due to collaborative publications involving the neighboring country Germany (see Table 4). In general, smaller countries show a high rate of collaborative publications (see Moed, 2005). Turkey’s publication numbers indicate that the share of internationally co-authored publications decreased over the years. Since the absolute number of Turkish publications has grown rapidly from 2005 onwards due to the integration of regional Turkish journals into the WoS, we can argue that the international cooperation could not keep up with the pace of national Turkish-language publications.
Affinity of co-authorships in the Web of Science among European countries for the 2002–2013 period. Cells are left empty for countries with fewer than 5 collaborative publications. High values express high affinity.
For the first part of the second research question, we can conclude that there is a clear trend towards international co-authorship according to the educational research publications of the WoS, but that the share of ‘international publications’ differs significantly between countries.
Collaboration preference
Schubert and Glänzel (2006) already noticed that trans-national asymmetries can exist in the relation of co-operating partners. To present a comprehensible comparison, an indicator of specific co-authorship affinity can be expressed as the ratio of two percentage values. The indicator considers a country’s absolute number of publications, its collaborative publications, and the number of co-publications with a country of interest. This ‘affinity value’ of the 20 educational research communities towards each other is presented in Table 4. If the number of joint publications is less than 5, the values are not indicated. Table 4 can be read as follows: each cell represents the importance of the country in the columns for the countries in the rows.
The highest ‘affinity value’ can be found for Germany (row) and Switzerland (column). Generally, the data confirm the expected effect that neighboring countries tend to show higher rates of cooperation. The UK, on the other hand, co-authored with all the other 19 countries and preferably with the Netherlands (3.6) and Germany (2.9). Also, a very widespread and even higher rate of international cooperation is featured by the Netherlands. Possibly, a very strong international cooperation rate might be one essential element of the international publication strategy practiced in the Dutch community.
The analysis of the ‘affinity value’ suggests that the UK and Netherlands dispose of a relatively widespread net of international cooperation in the field of educational research and that smaller countries may tend more often to cross borders for cooperation.
Language
This section deals with the third research question on the role of English as a publication language. In the past, non-English-language journals only stood a small chance to be included in WoS (Archambault et al., 2006). Van Leeuwen (2006) found that 95% of the publications in the SSCI were English-language publications. According to Figure 1, prior to 2005 more than 90% of the articles were published in English. Only articles published in German accounted for a visible share of non-English articles. The change of the journal selection policy from 2005 onwards becomes evident in the emergence of languages other than English in the following years, when especially Spanish and Turkish publications contributed substantially to educational research literature. While at first the underrepresentation of non-English publications declined until 2010, the three-year period from 2011 to 2013 shows an increase in the share of English language publications.
Figure 1b provides some insight into this somewhat confusing development by showing the share of English-language publications of countries that also publish in non-English language journals covered in WoS. After 2006, the absolute growth of journals enhanced the share of non-English languages, but after the progress of including new journals was decelerated, the underlying tendency that more and more researchers from non-English countries publish in English becomes evident. The last time period in Figure 1b indicates for all non-English countries a growing share of English publications in comparison to the earlier period.
As WoS did not integrate more German language journals after 2005 for German-speaking countries a more continuous increase is visible for the share of English-language articles. Whereas between 2002 and 2004 researchers from Germany published 42% of publications in English the share amounts to 71% in the 2011–2013 period, even though the absolute number of German-language publications also rose albeit only slightly (Botte et al., 2015: 150). Austria and Switzerland show similar trends.
A different evolution of the share of English-language publications can be observed for Spain, Portugal and Turkey. The large-scale inclusion of national journals into WoS from 2005 on caused a striking relative decrease of the share of English-language publications. After 2010, however, the share of English-language publications increased. Apparently, the following conclusions can be drawn:
There is a general trend of European authors in the field of educational research towards more publishing in English journals of the WoS.
This trend was overlaid by publications in the national language at the time when regional journals were integrated into the WoS – but publications in English are the dominant trend causing more European contributions to the analyzed set of educational research publications.
A possible reason for this ongoing trend might be trans-national awareness and citation frequency.
Citation impact
The next two sections deal with the fourth research question about the reception of research publications in terms of citations. Whereas the numbers of publications and indicators based on these data might be widely accepted as measures of research activity, citations are an object of controversy. It is hard to tell what citations exactly indicate since the reasons to cite are various (Moed, 2005). For example, negative citations might indicate disagreement or finding fault with another paper.
Nevertheless, it is generally assumed that citations are a measure of response to a research paper and that numbers of citations are an accepted indicator of the impact of publications. At least at the level of countries with a strong publication record, statistical validity can be assumed. Hence, the following figures provide an overview of two standard citation indicators for the field of European educational research publications on the level of countries: citations per paper (CPP); and the field normalized citation rate (FNCR). CPP is a basic indicator that expresses how often publications from a given year have been cited on average within a fixed citation window. The window was set to three years, that is, the most recent publications are from 2011, and citations received in 2011, 2012 and 2013 were considered.
As previously mentioned, citation behavior differs among disciplines. The FNCR considers specific publication behaviors and is calculated within the same field. The FNCR-value is not only normalized on the basis of the field but also on the publication year and the document type. Values above 1 signify a citation impact above average, whereas values below 1 indicate that the performance is below the average citation rate in educational research.
Figure 2 demonstrates that for most of the countries, the CPP grew from the first period to the second period under study. The highest citation rate in both periods can be found for the Netherlands, where on average each publication received 3.36 citations within the first three years after publication. The least cited publications from the 20 selected European countries are those from Turkey published between 2002 and 2006. However, Turkey increased its CPP remarkably in the second period. Note that citation distributions are extremely skewed. A small number of highly-cited papers can exist next to a majority of uncited publications, immediately affecting the CPP.
Citations per paper (CPP) and field normalized citation rates (FNCR) for the 20 most productive European countries in the 2002–2006 period and the 2007–2011 period in the Web of Science.
The FNCR indicator is more robust against outliers (highly-cited publications). We can see that Belgium, Finland, the Netherlands, Norway and the UK show an above average citation rate throughout the 2002–2011 period. Countries that perform below-average in both of the time periods are Denmark, Ireland, Portugal, Slovenia, and Spain. Countries that experienced an increase in their impact and show values above average within the 2007–2011 period are Austria, Cyprus, France, Germany, Italy, Sweden, and Switzerland. This development is accompanied by an increasing share of English-language publications of these countries.
For the second period, Greece and Turkey have a FNCR value of 1 and thus match the average citation rate in educational research. On the other hand, there are countries that faced a decrease in their impact in recent years. Apparently, the inclusion of regional journals had an immense effect on publication and citation counts of Croatian authors in educational research. Whereas prior to 2006 no journals from Croatia were included in the database, from 2006 onwards regional journals from Croatia were covered in WoS. These journals publish on educational research but are low in impact and probably not yet visible to the international scientific community. The inclusion of regional journals also had an effect on the impact of Turkish publications, as Turkish-language publications receive citations. This effect is obviously stronger for the larger Turkish community than for the smaller community of Croatia.
In the preceding section, we addressed the role of publication language against the background of criticism of WoS that it almost exclusively focuses on English-language publications. Given the competitive pressure of Scopus, WoS has also incorporated journals publishing in regional languages other than English. Citation frequencies are a strong indicator of perception and a premium feature of bibliometric indexes. After more than five years of inclusion of regional journals, it is interesting to look at the effect of this inclusion in terms of citation behavior.
The overall picture is clear: considering the 20 most productive European countries, there are 19,678 English-language publications that were published between 2002 and 2011. 2,713 publications (2002–2011) in the field of educational research were published in another language than English. English-language publications reach a CPP of 2.48 within a three-year citation window. This means that they are cited three times more often than non-English language publications with a CPP of 0.81. Also, the FNCR shows that English-language publications perform much better (with a value of 1.22) at the expense of non-English publications (0.81).
Major conclusions are:
Citedness and non-citedness correlate very much with publication language, English being the preferred language of transnational communication and exchange.
Publications in languages that are spoken only in small national communities will hardly improve their international perception even when they are included in WoS.
Publications in more popular languages such as German or French show higher citation rates, but as these higher frequencies can in most cases be ascribed to citations by a few regional communities, it seems convincing that even German or French authors need to publish in English, if they want to raise their international awareness and citations.
Another indicator that is connected with citedness is international co-authorship. Figure 3 presents the indicators CPP and FNCR for international publications on the one hand and for domestic publications on the other. In this context, ‘domestic publications’ refers to all publications with corporate addresses from only one country. By ‘international publications’, we mean those where two or more countries appear in corporate addresses.
Citations per paper (CPP) and field normalized citation rates (FNCR) for the 20 most productive European countries in the 2002–2013 period in the Web of Science.
The citation rates are calculated for a three-year-citation window for a country’s total number of publications from 2002 to 2011. The CPP for nearly all of the countries in Figure 3 indicates that international publications are far more frequently cited within the same time frame than domestic publications. Turkey is the only country that shows a lower citation impact for participating in international publications which might be referred to the relatively high number of new Turkish journals in the WoS which open the window for intra-national citations. These can have more relative weight if the number of international co-authorships is low.
Obviously, international co-authorship correlates with higher citation rates. Of course, the citation rates of international publications profit from the fact that they are mostly published in English. Other reasons for higher citation rates of international publications may reside in international co-authorship sui generis or in the topics addressed by these publications, which may evoke more international interest.
Highly-cited publications
Another bibliometric indicator expressing the citation impact of publications is the PPtop10%. The PPtop10% is the proportion of the publications of a country in a field that belong to the top 10% most frequently cited (Waltman and Schreiber, 2013) compared with other publications in that field. Thus, this indicator quantifies the share of a country’s total publications that belongs to the 10% most highly-cited publications of the same type, year and field. Here, the field is defined as the delineated set of educational research publications within the chosen period.
Figure 4 shows the share of the top 10% highly cited papers for the 20 European countries. The red line represents the ‘average’ 10% threshold assuming that on average 10% of a country’s publications belong to the upper 10% of the most frequently cited publications. The red line visualizes whether a country is above or below the expected value of 10%. The indicator was calculated according to the algorithm proposed by Waltman and Schreiber (2013). Since the PPtop10% indicator is a relative indicator, countries can perform well only at the expense of countries that show low performance.
Percentage of publications belonging to the 10% most highly-cited publications in educational research (PPtop10%) for the 20 most productive European countries in the 2002–2006 period and the 2007–2011 period in the Web of Science. A citation-window of three years was applied. Journal publications of the type article, letter and review were considered.
When we look at the top-10%-highly cited publications the absolute numbers for some countries tend to be quite low and skewed. Interpretations have to recognize these constraints. For example, the high share of highly-cited publications (28%) for Croatia in the 2002–2006 period is striking. In the following period Croatia performs strongly below average. The high share of highly-cited publications is due to a low number of internationally co-authored publications that were frequently cited. Generally, we can see that the Netherlands produced highly-cited publications throughout the time period just as Belgium and Germany did. The share of German publications belonging to the top-10%-highly-cited has increased steadily and comprises around 14% in recent years. We can see that Cyprus, Greece, Ireland, Portugal, Slovenia, Spain, Sweden and Turkey show a below-average share of highly-cited publications throughout the 2000–2011-period. Austria and Switzerland increased their impact immensely, whereas Croatia (see above) and Denmark suffered a loss in highly-cited papers. While Turkey had no single publication (2002–2006) that belonged to the top-10% highly-cited publications, their share increased to 7% in the latter period. Taking the previous results into consideration we can conclude that although the CPP of Turkish publications from 2002–2006 shows average values, none of the publications is highly cited.
Generally, we can conclude that highly-cited publications are a problematic indicator for the field of educational research. Only for countries with high numbers of cited publications this indicator may be consulted to look for confirmation or explanation of tendencies detected by the analyses of the more basic CPP indicator. For the smaller countries, the publications which account for the share of highly-cited publications are possibly random.
Figure 5 shows the top-10% indicator distinguished by international and domestic publications. Evidently, international collaborative publications have a high propensity to belong to the top 10% highly-cited publications in educational research. With the exception of Turkey, each of the European countries has an international PPtop10% higher than 10%. Publications from Austria, Germany, the Netherlands, and Switzerland seem to gain a special bibliometric profit from international cooperation. More than 20% of their international publications belong to the most frequently cited 10% of publications in educational research. Domestic publications from Belgium, Germany and the Netherlands prove to be likewise of high impact and visibility, whereas most of the other countries suffer a loss in visibility of domestic publications.
Percentage of publications belonging to the 10% most highly-cited publications in educational research (PPtop10%) for the 20 most productive European countries in the 2002–2011 period in the Web of Science. A distinction is made between internationally collaborative and domestic publications. A citation-window of three years was applied. Journal publications of the type article, letter and review were considered.
Again, we can conclude that international publications reach high visibility in the educational research community and are cited far more often than domestic publications.
Conclusions
This paper aimed at presenting a broader view on European countries active in educational research on an international level, here represented by journal articles covered by the WoS. Besides this specific publication culture reflected in the WoS, there are other publication cultures in the field of educational research, bearing considerable national or regional impact. Nevertheless, the WoS journals are genuinely important for the international awareness of research activities, which makes the following results of analyses meaningful for a European international publication culture:
Many European countries increased their publication output through the journals of the WoS considerably. The grade of this movement towards international awareness of publications is significantly different across national communities. The Netherlands and some Scandinavian countries are represented beyond their country size. Some countries (e.g. Turkey and Spain) show an extraordinary growth during the last decade.
The integration of more non-English journals into the WoS since 2005 has fostered the spread of WoS authorships beyond the English-speaking countries. Nevertheless, the availability of more publications in other languages than English is overlaid by a continuous general trend towards English as the international language of publication.
There is a strong increase of international co-authorships.
European educational researchers have a growing international impact in terms of citations if they publish in English. Data show that non-English language journals have only a small chance of being cited internationally. That means that the inclusion of non-English ‘regional’ journals in the WoS offers only a limited way to publish with aspirations of international response.
Especially in a European context, language presents an impediment for bibliometric indicators. Articles published in languages other than English are less accessible to some researchers, which in turn influences their citation patterns.
What do results of this study tell about the success of ‘European and national policies’ to support internationalization of educational research? Especially the fact that the growth of publications goes in line with more transnational cooperation in the form of co-authorships provides evidence that ‘political’ challenges of European and of many national administrative bodies to promote research with an international orientation seem to be effective. Nevertheless, this is only an indirect conclusion: our study does not provide any direct evidence, that European or national policies may have caused this effect.
At least for the communities of the 20 European countries analyzed in this study, it is made visible that the trend towards internationalization of educational research encompasses small countries as well as bigger countries with strong traditions in nationally oriented funding and research (e.g. Germany and France). The trend is also identifiable for research communities with a longer history of internationalization starting on a relatively high level already in the first period included in this study (e.g. the Netherlands and Finland) on the one hand, and for ‘newcomers’ (Croatia, Slovenia, and Turkey) on the other hand.
Clearly, the extent to which internationalization takes place differs between countries. Only a country with a certain national tradition of educational research will be able to make successful progress at the international level. Also, certain subfields and methods of educational research seem to have advantages when striving for internationalization. An analysis of the top-15 journals which contribute to our corpus of WoS publications shows that journals with a background in science, medicine and psychology are the strongest providers of articles (Botte et al., 2015: 136). Other studies support the assumption that internationalization is a stronger trend in empirical social sciences than in the more hermeneutic social sciences and humanities (cf. Engels et al., 2012: 381; Moed, 2005: 148).
International ‘orientation’ can manifest itself in the form of direct cooperation or in the form of striving for international awareness, which includes that research contents, methods and results reach international relevance. Both criteria may constitute the measurement of a new quality of publications, which will be assessable along with traditional peer review criteria.
Methodologically, this study revealed chances and limitations of a bibliometric approach:
Generally, the analysis of WoS publications is a fruitful approach to follow the development of national publication cultures striving for international awareness. The inclusion of more non-English regional journals since 2005 has increased the outreach of the WoS indexes. Bibliometric indicators of scholarly publications can confirm or detect possible developments in international publication cultures. Like all statistical data, they need high levels of aggregation and long periods of observation to be valid. Bibliometric analyses lose reliability when smaller countries with low publication rates are considered.
The analysis of citations can be a suitable approach to a deeper understanding of the forms of international awareness. But as the share of cited publications in educational research is still low, analyses have to be careful with cases of noise and bias caused by non-representative numbers and outliers. The indicators of highly-cited publications seem to be not suitable for country analyses in the field of educational research publications.
Qualitative or hermeneutic research is required to interpret interactions and to explain motives.
Bibliometrics might be regarded as an imprecise methodological instrument because bibliometric databases are limited in coverage and statistical metrics do not necessarily reveal qualitative aspects. Nevertheless, bibliometrics offer a powerful set of tools to analyze aspects of an international publication culture. Our findings suggest a growth in the overall amount of publications in educational research in our data set. This means that educational researchers in many European countries have slightly changed their publication habits towards publishing in journals with international impact. Evidently, there is a concentration of highly productive countries, among which the UK, the Netherlands, Germany and Sweden play a special role. Spain and Turkey are upcoming countries with an obvious national strategy of internationalization.
The WoS data seem to remain a relevant source for the observation of such a development. Alternatively, web-based communication of research and among researchers is said to be also an important – at least future – indicator of the international impact of research. These alternate metrics are capable of indicating scholarly significance independent of a citation database such as WoS. Altmetrics encapsulate the digital collection of mentions of scholarly articles from newspapers, blogs, social media, and equivalent sources. The field of altmetrics as a prospective additional approach to traditional bibliometrics has already created indicators and even commercial tools 13 . For educational research, the question about which groups of scholars already prefer the web as an approved source of communication (e.g. open access publications) is still to be examined 14 . Except for Google Scholar, the sources for altmetric analyses of educational research still do not seem to present a viable alternative to the WoS.
We can conclude that bibliometric analyses provide information on how research is conducted and received, but the data do not account for regional idiosyncrasies. Bibliometric measures may come across as neutral and valuable, but as Tight puts it, ‘bibliometric data like this does tell us something […] but it clearly does not tell us everything’ (Tight, 2009: 64).
Footnotes
Appendix: lexical terms
| academic_achievement | coeducation | course selection |
| academic_degree | collaborative learning | creative activities |
| academic_failure | collaborative_problem_solving | creative teaching |
| academic_freedom | collaborative_task_solution | creativity |
| academic_persistence | college administration | credential |
| academic_probation | college admission | credit |
| academic_rank | college attendance | cross_age_teaching |
| academic_record | college choice | cultural enrichment |
| academic_standard | college credit | day school |
| access to computer | college environment | degree |
| access to education | college governing council | democratic education |
| achievement | college instruction | developmental dyscalculia |
| admission | college outcomes assessment | developmental dysgraphia |
| adult education | college planning | developmental dyslexia |
| affective state | college preparation | developmental studies program |
| agricultural college | college program | developmental timetable |
| analogical thinking | college role | diagnostic teaching |
| andragogy | college school cooperation | diary |
| articulate | community education | direct instruction |
| associate degree | community school | disadvantaged school |
| attendance | comparative test | discussion |
| automaticity | compensatory education | disregard |
| bachelor’s degree | comprehension | distance education |
| background knowledge | comprehensive program | diversity |
| bilingual education | comprehensive school | doctoral degree |
| bilingual school | compulsory education | doctoral program |
| black college | computer attitude | dress code |
| blended learning | computer literacy | dropout |
| boarding school | computer use in education | education_work_relationship |
| boards of education | concept map | educational administration |
| bully | conceptual change | educational attitude |
| career | consolidated school | educational benefit |
| case method | continuation education | educational certificate |
| case record | continuing education | educational change |
| catholic school | continuity | educational concept |
| certification | continuum | educational cooperation |
| charter school | conversation | educational demand |
| church related college | cooperative learning | educational development |
| class activity | cooperative program | educational diagnosis |
| class average | corporate education | educational environment |
| class organization | correctional education | educational history |
| class rank | correspondence school | educational improvement |
| class size | county school | educational innovation |
| classes | course content | educational legislation |
| classroom | course evaluation | educational malpractice |
| cluster college | course objective | educational method |
| co-education | course organization | educational mobility |
| educational need | grading | intergroup education |
| educational objective | graduate study | international education |
| educational opportunity | graduation | international school |
| educational philosophy | grandparent | interschool communication |
| educational planning | group activity | junior_high_school |
| educational policy | group decision | kindergarten |
| educational practice | group discussion | laboratory school |
| educational principle | group instruction | language acquisition |
| educational problem | grouping | language learning |
| educational program | high achiever | law school |
| educational quality | high_school | learner engagement |
| educational research | higher_education | learning activity |
| educational resource | home program | learning center |
| educational responsibility | home schooling | learning environment |
| educational service | home study | learning from picture |
| educational specification | homework | learning from text |
| educational standard | implicit knowledge | learning paradox |
| educational strategy | implicit learning | learning science |
| educational supply | impression_formation | learning skill |
| educational technology | improvement programs | learning strategy |
| educational therapy | independent study | learning_difficulty |
| educational trend | individualized_ program | lecture method |
| electronic learning | individualized_education | lesson |
| elementary education | individualized_instruction | library school |
| elementary school | informal education | lifelong learning |
| elementary_secondary_education | informal learning | longitudinal study |
| enrollment | information literacy | magnet school |
| entering school | information seeking | majors |
| episode teaching | inquiry | mandatory_continuing_education |
| ethnic_minorit | institutional autonomy | master’s degree |
| excellence | institutional cooperation | master_’s degree |
| exchange program | institutional environment | master’s program |
| experiential learning | institutional evaluation | master_’s program |
| experimental college | institutional mission | mastery learning |
| experimental program | institutional role | material development |
| experimental school | instructional design | mathematic |
| experimental teaching | instructional development | mental addition |
| expertise | instructional effectiveness | mental calculation |
| expulsion | instructional improvement | metacognition |
| faculty | instructional innovation | microteaching |
| flexible schedule | instructional leadership | micro_teaching |
| fluid intelligence | instructional material | middle school |
| folk school | instructional program | migrant education |
| fosterage | instructional systems | migrant_adult_education |
| free school | intellectual freedom | military school |
| giftedness | intelligence | minimum competence |
| governance | intercultural communication | mono_education |
| government | interdisciplinary approach | montessori |
| grade | interest | motivation |
| multicampus | professional learning | school organization |
| neighborhood school | professional training | school orientation |
| nonformal education | professional_continuing_ | school performance |
| nongraded_student_evaluation | education | school planning |
| nonschool_education | professional_development | school policy |
| non_school_education | professional_development_ | school prayer |
| nontraditional education | school | school registration |
| non_traditional education | progressive education | school responsibility |
| number sense | proprietary school | school restructuring |
| nursery school | public college | school role |
| one_teacher_school | public education | school safety |
| open university | public relation | school security |
| parent_ student_relationship | public school | school setting |
| parent_school_relationship | reading | school shop |
| parent_teacher_conference | reciprocal teaching | school size |
| parent_teacher_cooperation | reference group | school space |
| parents as teacher | reflective teaching | school supervision |
| parochial school | regional school | school turnaround |
| partnerships in education | remedial instruction | school uniform |
| pass_fail_grading | remedial program | school visitation |
| pedagogical_content | research paper | school_entrance_age |
| peer teaching | research university | schools of education |
| performance contract | residential school | secondary education |
| performance factor | response to intervention | secondary student |
| performance technology | role of education | secondary_school |
| persuasion | rural education | self_regulated learning |
| phase of learning | rural school | sequence learning |
| phonological awareness | scholarship | serial learning |
| phonological processing | school activity | service learning |
| place_based_education | school administration | sex_difference |
| politics of education | school attendance | short_term_memory |
| poor reader | school attitude | shyness |
| popular education | school begin | single_sex_classes |
| postdoctoral education | school bonding | single_sex_college |
| postsecondary education | school catalog | single_sex_school |
| practicum supervision | school choice | slum school |
| precision teaching | school closing | small class |
| preschool education | school community | small college |
| pre_school education | school council | small school |
| preschool evaluation | school demography | social change |
| pre_school evaluation | school district | social exchange |
| primary education | school effectiveness | social intelligence |
| private college | school expansion | social promotion |
| private education | school holding power | sociomathematical norm |
| private school | school involvement | special class |
| problem_based_learning | school law | special degree |
| process education | school location | special program |
| professional autonomy | school maintenance | special school |
| professional education | school music training | state college |
| state school | study skill | teacher supervision |
| state university | study stress | teacher transfer |
| student adjustment | suburban school | teacher welfare |
| student attitude | teacher association | teaching assignment |
| student attrition | teacher attendance | teaching condition |
| student behavior | teacher attitude | teaching experience |
| student certification | teacher background | teaching load |
| student characteristic | teacher behavior | teaching method |
| student development | teacher burnout | teaching model |
| student distribution | teacher certification | teaching pattern |
| student diversity | teacher characteristic | teaching program |
| student education | teacher collaboration | teaching skill |
| student empowerment | teacher competence | teaching style |
| student engagement | teacher discipline | test anxiety |
| student evaluation | teacher dismissal | test preparation |
| student exchange | teacher distribution | test wiseness |
| student experience | teacher education | textbook |
| student government | teacher effectiveness | thinking |
| student improvement | teacher empowerment | time pressure |
| student interest | teacher evaluation | time_on_task |
| student journal | teacher exchange | traditional school |
| student leadership | teacher improvement | transfer program |
| student mobility | teacher influence | transitional program |
| student motivation | teacher judgment | truancy |
| student need | teacher leadership | tutorial program |
| student organization | teacher learning | tutoring |
| student participation | teacher militancy | two_year_college |
| student placement | teacher morale | university |
| student problem | teacher motivation | urban education |
| student project | teacher orientation | urban school |
| student promotion | teacher participation | urban teaching |
| student reaction | teacher persistence | urban university |
| student record | teacher placement | virtual university |
| student recruitment | teacher promotion | vocational education |
| student responsibility | teacher qualification | vocational high_school |
| student right | teacher recruitment | vocational orientation |
| student role | teacher response | vocational school |
| student school relationship | teacher responsibility | women education |
| student subculture | teacher right | word_problem |
| student transportation | teacher role | writing assignment |
| student_centered | teacher selection | writing exercise |
| student_college | teacher student ratio | |
| student_teach | teacher student relationship |
Declaration of Conflicting Interest
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article. This work was supported by the Federal Ministry of Education and Research (BMBF) under the grant number 01PQ17001.