Abstract
Abstract
Quality assurance is about getting people to think and behave in ways that may be different from what they are used to. In other words, quality assurance is about changing individuals and through them the institutions with which they are associated. Different countries and different higher education institutions approach the processes and the politics of effective quality assurance differently.
I draw out major issues that help or hinder effective quality assurance. I point to information that needs to be provided about quality assurance, and also emphasise the need for evaluation of how quality assurance is undertaken. I emphasize the importance of recognizing the people and processes of quality assurance that lead to positive outcomes. I argue that whereas there is no single way to approach quality assurance, there are principles that need to be understood and applied in ways that are appropriate to the individuals, institutions and countries involved.
Introduction
This is a personal reflection on quality assurance in higher education, based on more years than I care to count and a large number of involvements in quality assurance activities across a range of countries. I make these comments from the perspective of someone who has conducted external quality assurance of institutions in different countries, as well as someone who has been responsible for quality assurance within universities in Australia.
By way of introduction, I wish to make four points: Depending on the understanding of these points, quality assurance can be helped or hindered; it can go well or not go well.
First, sometimes words matter, and sometimes they don’t. Quality assurance is sometimes called quality audit, quality improvement, quality assessment, and so on. The term quality assurance is also sometimes used interchangeably with terms such as standards, accreditation, regulatory compliance, external review, and so on. Although there are genuine differences among some of these terms, the important issue is to know when those differences exist and matter and when they don’t. At its heart, quality assurance is about asking one very simple question: “How are things going?” There is a whole international industry around framing that question, obtaining and interpreting the answer, and then implementing the changes that are needed.
Second, quality assurance in higher education is, or should be, a system of peer review. Across most of higher education, peer review is a well-recognised and well-accepted approach to assessing and improving scholarly activity. Quality assurance is no different. It should be an open system of peer review, even though that is not always the way that governments see it in various countries.
Third, quality assurance should be part of the normal process of, rather than an isolated event in, the life of an institution. We hope and encourage our students to engage in continuous learning, with assessment simply sampling that learning. And so it should be for quality assurance. It should be part of the culture and fabric of an institution, with external quality assessment simply sampling that culture and fabric. Unfortunately, however, that is not always the way that institutions approach quality assurance.
Fourth, the most difficult part of quality assurance is putting in place the changes that are inevitably needed to improve. There can always be improvement. Usually, there is no shortage of ideas and suggestions in universities and in quality assessment reports. But there is sometimes a shortage of the commitment and action in universities that is needed to make those things happen. Turning possibilities for improvement into a reality is not only the most difficult part of quality assurance, but also the most critical part.
So, what are some of the ways in which higher education institutions and individuals can engage successfully in quality assurance? And, what are some of the problems to avoid? I address these questions by drawing out some issues concerning information and preparation, visit and report, and implementation and evaluation. I will then make some concluding comments.
Information and Preparation
The information and preparation needed for external quality assurance depends in large part on the model or framework that is being used for the quality assurance. Most countries and agencies have a particular model or framework for quality assurance, and there can be major differences across them. It is essential that all of those involved understand the particular model or framework under which they are operating.
Although it is usually the case that the institution and the assessors understand the particular framework under which they are operating, sometimes they do not. I have seen well-meaning assessors who want to bring their own framework, whether ideological or cultural, to bear on a quality assurance activity. Similarly, I have seen institutions participate in quality assurance without any real understanding of the framework of the country in which they are operating.
The focus of the quality assurance activity can vary widely across countries and agencies. Some quality assurance activities consider all academic and administrative functions within an institution, others focus largely on the academic aspects, and others focus on identified themes, such as governance and management or international relations. Many things are connected within universities, and it is sometimes difficult to disentangle activities, but clarity about focus is essential in quality assurance.
Irrespective of the model and the focus, the materials that are provided in advance typically serve as the basis of the quality assessment. It is essential that these written materials be open and honest and be made available in a timely way to all involved. Although most institutions seek to engage responsibly in terms of the provision of materials, I have seen some institutions provide materials that are obviously inaccurate or prepared solely for the purpose of quality assurance. One issue that sometimes needs to be considered is the language of the materials. Whereas the materials will be in the language of the relevant country, it may be that not all assessors can read that language.
I have also seen some institutions express reluctance to provide certain materials and/or provide those materials at the last minute, often expressing concern about possible breaches of confidentiality. In my experience, most people who engage in external quality assurance across the world genuinely wish to understand and assist the institution they are assessing, and are scrupulous about confidentiality. They are also, in my experience, very skilled at detecting inconsistencies, misleading statements, and plain nonsense in the written materials.
At the heart of the comments that I have made so far is effective communication and authentic commitment in two major domains. The first domain is within the institution that is being quality assessed by an external group. It is important that all faculty and staff are aware of the purpose and process of the quality assessment activity, and it is essential that all key individuals are committed to the activity. Ideally, the most senior people (Presidents, Vice-Presidents, and the like) should be the public and committed champions of quality assurance.
The second domain is between the institution and the external agency. It is important that this relationship is an open and direct one that is based on a constructive collaboration that is committed to the activity, rather than one that is based on an antagonism and a reluctance to engage. Any such antagonism during preparation creates suspicions and problems that are difficult to overcome.
Visit and Report
A major issue for any external quality assessment is whether a formal site visit is undertaken. There is substantial variation across countries and quality assurance agencies in this respect. An external quality assessment visit can range from none happening with the activity based on a consideration of written materials, to a selective or sampling visit that focuses on aspects of the institution, to a comprehensive visit that interviews a large number of people, views a wide range of facilities, and takes place at all relevant locations of the institution. Each of these types of visits requires particular material to be prepared and other, often complex, logistics to be determined and arranged.
On the assumption that a comprehensive visit is involved, detailed interviews usually occur during such a visit. Typically, a sample of faculty, staff and students as well as graduates, employers and community members are interviewed either individually or in small groups. The essential aspect of these discussions is to investigate and triangulate the information contained in the written materials and the information provided by other interviewees. As with the written materials, if the interviews involve people from different language backgrounds, then care must the taken to ensure that verbal communications are accurate and understood.
It is in these discussions that external assessors get a real sense of what is actually happening in the university, rather than what the policies say should be happening or what senior people say is happening. Mostly, in my experience, there is reasonable convergence between policy and practice, but sometimes there is not. In those instances, it is usually not the case that senior people are being deceptive. Rather, it is usually the case that nobody has told them that the place is not operating in the way they believe.
An amusing aspect of a site visit is that it is sometimes apparent that faculty, staff and students have been told to say certain things by senior people. This rarely lasts long. A simple “Okay, now that you’ve said what you’re supposed to say, what do you want to really tell us?” usually leads to nervous laughter, and then to a wealth of information being provided. This works across almost all countries and cultures, as well as across all levels of people within an institution, if done properly.
In my experience, those involved in quality assessment are particularly skilled at the pleasant, disarming question that allows information from multiple sources to the checked and verified or not. Similarly, in my experience, such experienced colleagues rarely accept at face value the accuracy of any information, whether written or verbal, but rather seek to get a deep and independent understanding of the information they are given.
Following consideration of the materials and the discussions occurring during the site visit, a formal report is prepared. Most quality assurance agencies have a report format or template that is aligned with the model of quality assurance that they are using. In some instances, this may concern the extent to which the institution meets specified accreditation or regulatory standards or criteria, and the report format may leave little scope for suggested improvements. In other instances, the report may focus more on how the institution is performing against its own stated goals and targets, and the report format may encourage suggested improvements.
Whatever the case, the preparation of the report typically goes through a number of drafting and fact-checking phases before it is made available to the institution. This takes time, and sometimes the time that passes is so long that the potential impact and of the report is lessened. If a report is provided too long after a quality assurance visit, then the institution may have lost the impetus for change. Indeed, there may be scope for the institution to argue that the report is out-of-date. Unfortunately, the time involved in providing the report is a process issue that many quality assurance agencies around the world have not been able to deal with effectively.
The extent to which the formal report is subsequently available beyond the institution, in the sense of being a public report, varies greatly around the world. Let me say that it is my view that most of the time quality assurance reports should be open and public. Sunlight is a great motivator in my experience. The issue here is that, whatever the format of the report, it must be factual, defensible, timely, and ideally provided in an open and public way.
Of course, when a report is provided, the task of the external quality assessor is finished. In fact, it is one of the ironies of external quality assurance that those who conduct that activity walk away from it with more knowledge about and understanding of the university than any individual within the university. But they typically have no further involvement with the university.
Implementation and Evaluation
When a university receives a report, the initial reaction typically depends on the findings of the report and the consequences of those findings for the institution. If it is a largely positive report, then it is often the case that the institution will react along the lines that they knew this all along and the quality assessment did not add any value. Whereas that may be true in part, an external quality assessment that is largely positive provides an important and independent validation of the institution.
If, on the other hand, it is a mixed or largely negative report, then it is usually the case that the institution will react in one of two ways: they may indicate that the quality assessment was flawed, and the findings are wrong; or, they may indicate that the problems identified in the quality assessment were known by the institution and have been corrected or are in the process of being corrected.
These are all normal initial organisational reactions to an external assessment. The important thing, however, is that such initial reactions should not be allowed to either lead to complacency because all is well or lead to rejection because the quality assessment is said to be flawed.
Rather, and in the spirit that quality assurance is, or should be, a continuous process within an institution, the best considered response is that the quality assessment report is an external and independent indication of where effort needs to be continued and where effort needs to be initiated to improve the institution.
The quality assessment report is one piece of information in the strategic direction of a university, but it is an important piece. It is the case, in my experience, that most universities embrace the report and integrate the findings and suggestions that it contains into their own quality assurance activities.
But it is also the case that some institutions engage in excuse making and/or rejection of the report. Of course, this will depend in part on the consequences for the university of the quality assurance activity. It is a paradox that organisations, like individuals, often need to be convinced to improve, and such improvement sometimes only occurs when there are negative consequences associated with not improving and positive consequences associated with improving.
Whatever the initial reaction of the university and whatever the more considered response and actions that are taken, the quality assurance activity should be formally evaluated. This is simply because quality assurance activities themselves should be evaluated. Although such evaluation is not conducted in all countries or by all agencies, in my view it is an essential aspect of the overall quality assurance process.
Changes within an institution almost always flow from a quality assessment activity, but they do not always flow easily. This is not because faculty and staff are particularly resistant, but rather because change is not easy to achieve within conservative organisations such as universities. As an aside, let me say that one of the great historical ironies of universities generally is that they are internally conservative organisations that exert externally progressive influences.
In my experience within universities, the most effective way of achieving change is through recognizing the people and processes that are heading in the desired strategic direction. Most of us are influenced heavily by the actual or the potential consequences of our actions, and if certain actions are recognized and rewarded then our behaviour will move closer toward those actions. Put another way, for strategy to be effective, an appropriate work culture needs to be created and recognized within the institution.
Concluding Comments
I have tried to make a number of simple points based on my experience and observations. Let me emphasize that there is no single way to approach quality assurance, but there are principles that need to be understood and applied in ways that are appropriate to the individuals, institutions and countries involved.
I said earlier that at its heart, quality assurance is about asking one simple question: “How are things going?” The best way to approach that question is to engage in a way that is open and transparent and to involve independent peer review. And the best way to approach the answer to that question is to engage in a way that places the answer and the actions needed within continuous quality assurance in the institution.
This must involve recognizing and rewarding the people and the processes within the institution that are constructively aligned with the needed improvements. In other words, the key is to create a culture that aligns the quality assurance improvements with the strategic direction of the university. In that way, quality assurance will lead to positive change in individuals and in the institutions with which they are associated.