Abstract
Multi-rater feedback instruments have been an established way of providing feedback to employees in organisations for nearly three decades, yet it remains debatable whether the use of these instruments adds value for recipients or their organisations. In this paper we question some of the underlying assumptions behind the design of these instruments and challenge the way they are implemented. Our inquiry is based on critical reviews of the literature relating to multi-rater instruments and recent developments in theories of self and multiplicity of self, and an exploratory case study in an organisational context. Without questioning the value of the multi-rater feedback in principle, we identify prevailing assumptions that minimise its effectiveness and advocate for a different, more fruitful, approach.
Introduction
Multi-rater 360° feedback systems are used by many organisations in the USA, UK, and other countries (Bracken, Rose & Church, 2016; Church et al., 2019; Mabey, 2001) and by the majority of coaches practising in organisations (Mcdowall & Kurz, 2008). The term ‘360° feedback’ originated in the mid-1980s and the practice of 360° feedback became popular in the mid-1990s (Bracken et al., 2016, Fletcher, Baldry & Cunningham-Snell, 1998; Slater & Coyle, 2014). The increased popularity of these instruments can be attributed to:
A general and reasonable belief in organisations that employees must have access to feedback if they are to develop and grow (de Villiers, 2013).
Difficulties in creating an environment in which people feel comfortable giving each other direct, honest, and open feedback. One of the reasons is that people typically lack the skills required to deliver feedback effectively and/or are worried about the repercussions of providing feedback (Maxwell, 2017).
The perceived objectivity of multi-rater feedback (Facteau et al., 1998; Fletcher et al., 1998; Garavan, Morley & Flynn, 1997; Mabey, 2001).
Problematising 360° feedback instruments
Meanwhile, the debates among academics and practitioners continue with many decrying the lack of evidence as to the impact of 360° feedback or highlighting evidence that suggest its minimal impact (e.g. Atwater & Brett, 2006; Bracken et al., 2016; DeNisi & Kluger, 2000; Smither, London & Reilly, 2005). For example, Kluger and DeNisi (1996) conducted a meta-analysis of feedback interventions generally, defined as “actions taken by (an) external agent (s) to provide information regarding some aspect(s) of one's task performance”, and reported a decline in performance following feedback in over one-third of cases.
In the literature many reasons are suggested as to why 360° feedback may not deliver desired outcomes. We summarise these reasons in two groups: issues with raters and issues with instrument design and information processing. Then we describe what solutions may be offered to address these issues.
1. Issues with raters
Although there are good reasons to believe that the use of multiple raters is a step forward in terms of the objectivity of feedback, it is also generally recognised that raters do not score objectively. Their ratings are subject to phenomena such as the leniency effect and the halo effect (van der Heijden & Nijhof, 2011), their understanding of the data anonymity (Morgeson et al. 2005; Waldman, Atwater & Antonioni, 1998) and how the data is to be used (DeNisi & Kluger, 2000; Waldman, Atwater & Antonioni, 1998). The variations in their responses are subject to whether they find ratees pleasant to work with (Fletcher et al., 1998; Mcdowall & Kurz, 2008; Waldman, Atwater & Antonioni, 1998) and the age, gender and status of the person they are rating (Alimo-Metcalfe, 1998; Facteau et al., 1998; Fletcher et al., 1998; van der Heijden & Nijhof, 2011). Even when raters genuinely wish to give as objective feedback as possible, other complicating factors come into play, such as a failure to recall events accurately, lack of opportunity to observe ratees in relevant situations, and different raters’ beliefs as to what good leadership looks like (Eckert et al. 2009; Garavan et al., 1997; Gentry & Eckert, 2012; Nowack, 2019; van der Heijden & Nijhof, 2011). Unfortunately, as van der Heijden and Nijhof (2011) point out, rater subjectivity cannot be managed out simply by increasing the number of raters.
2. Issues with instruments design
This group of issues includes factors such as the content of the actual instrument, how data are collected and interpreted, and various situational aspects. In terms of the design of the 360 instruments many authors comment on insufficient reliability, validity and content relevance of the questions to the role assessed, with some additional concerns about cultural sensitivity (e.g. Bracken et al., 2016; Fleenor, 2019; McCauley & Brutus, 2019; Sibley, 2007). Process factors are also problematic, starting with choice of raters and the flaws in their training, moving on to insufficient clarity of purpose and intended use of data, and finishing with issues around ratee ability to interpret data and feedback timing (e.g., Eckert et al. 2009; Fleenor, 2019; Maxwell, 2020; Nowack, 2019). Situational issues are concerned with organisational culture and levels of support provided for everyone involved, peer pressure and accountability (e.g., Bailey & Austin, 2006; Fleenor, 2019; Maurer, Mitchell & Barbeite, 2002; Nowack, 2019).
Coaching and 360° feedback
Analysis of the above issues has led various authors to offer best practice models (e.g., Campion, Campion & Campion, 2019; Church et al., 2019; de Villiers, 2013; Lawrence, 2015; Nowack & Mashihi, 2012). Most of these models recognise the importance of providing the ratee with the opportunity to make sense of their feedback through various forms of facilitation process. Some authors point specifically to the value of a coach (e.g., DeNisi & Griffin, 2001; Luthans & Peterson, 2003; Maxwell, 2017; McCauley & Brutus, 2019; Smither et al., 2003; Thach, 2002). Some writers go so far as to suggest that providing feedback is the fundamental role of a coach (Zeus & Skiffington, 2000).
Although the use of coaching for structuring and delivering the results of 360° feedback should be supported, we believe that it is important for coaches to review their practice in light of the issues we have summarised. They can benefit from challenging their approach to delivering 360° feedback on two counts: perspective on systems and view on the nature of self.
1. The coach’s perspective on system
With regard to the first challenge, the coach’s perspective on systems, we note that the adoption of typical feedback practices can be justified and reasonable if viewed through a behaviourist/objectivist philosophy. According to this philosophy the purpose of feedback is to reduce discrepancy against an external standard (Maxwell, 2017; Lawrence 2021). This perspective is consistent with simpler models of systems in which individuals have ‘to fit in’ in order to be effective (Stacey, 2012). In these models humans can be usefully compared to the components of a machine, which may have a complicated design, but the ‘faulty’ components can be identified and ‘fixed’. Good feedback allows a smooth ‘fitting in’ process. Overall, this position is based on four key assumptions:
The system can accurately and objectively determine how the individual should behave.
The individual should behave this way in all contexts in order to be maximally effective.
The system can accurately monitor how the individual is behaving.
The individual is compliant and amends their behaviour in line with feedback received.
Even a brief overview of these four assumptions reveals fairly obvious problems with this ‘fitting in’ philosophy. Through this lens, the skills of a leader may be compared to those of a plumber. The plumber examines the leaky tap, diagnoses the problem, then reaches into his toolkit to find the tool that will remedy the situation. The leader is similarly encouraged to diagnose the situation and deploy the appropriate competence to resolve the scenario. Stacey (2012) labels this philosophy ‘instrumental rationality’, contrasting it with the constructivist philosophy which is based on a belief that the world is much more complex and unpredictable. According to Stacey (2012: 5), “the inevitably uncertain and ambiguous processes of interaction between people, which produce an organisation, do not take a linear form and efficient causality does not apply.” In the context of organisations, instead of instrumental rationality Stacey offers ‘practical judgment’, which implies making judgments in the moment based on reflections upon their own previous experience and the perceived experience of others. Coaches delivering feedback would also recognise their own subjectivity, and the limitations of their capacity to determine how the ratee should behave. As Maxwell (2017: 320) suggests, the role of the feedback coach shifts to “… partnering the coachee to make sense of feedback received, scaffolding from their prior experiences to new understandings and knowledge.”
We would also argue that the constructivist position on feedback (Maxwell, 2017) indicates the complexity not only of the system, but also of the individual. It clearly requires designers of the instruments and coaches who deliver these to review their conceptualisation of self in order to allow for more complexity, agency and unpredictability of responses. This leads us to the justification of the second challenge - the need to reconsider perspectives on the nature of self.
2. The coach’s perspective on self
First, we noticed that none of the studies we referenced on 360° feedback consider theories of self and their relevance to feedback giving. Whilst some studies recognise that coaching makes a difference in the ratee’s capacity to interpret feedback and put new insights gained into action (Luthans & Peterson, 2003, Maxwell, 2020), no link has been made so far between the opportunity that coaches have to understand the person in more nuanced ways and better results of the feedback delivery.
Secondly, most of the studies, implicitly or explicitly, take the self to be a unitary entity. The role of feedback is therefore to help the recipient better understand his or her (single) self. For example, Avolio (2005: 320) suggests that to reflect on feedback means “to know oneself, to be consistent with one self…”. In this paper, we would like to offer a different conceptualisation of self that reflects not only a contemporary view on self that challenges the behaviourist/objectivist position on feedback, but also provides new insights for coaches on the best use of 360° feedback instruments.
Challenging assumptions with a perspective on the self as multiple
The dominant narrative in the literature is of a single, continuous, agentic self. For example, the leadership literature usually frames authenticity as a desirable leadership attribute and most often defines the quest for authenticity as the search for a greater understanding of that unitary single self. Feedback systems, such as 360°, are used in a consistent way with that narrative, with rater results often aggregated, and ratees being discouraged from trying to work out how individuals may have rated them and invited to gather new insights about that single self.
Although this view of self may feel intuitively accurate and is partially supported by our use of language and some theoretical positions, it is certainly not the only theoretical position today, at a time when there is more recognition and embracing of complexity in many aspects of the world, including ourselves (Bachkirova, 2022; Lester, 2010). In this section we give a brief overview of the theories that conceptualise self or, in other words, identify what we mean by self. Then, we give examples of theories that position self as multiple. Multiplicity refers to the various expressions of self: how we appear to feel and act in different situations, and how we ‘tell stories’ about ourselves.
What we mean by self is not often a topic of discussions in applied disciplines, such as leadership or coaching. It has been, however, a topic of philosophical debate for a long time. Amongst philosophers there have been disagreements not only about the self’s singularity or multiplicity, but also about its entire existence. From being identified as a soul in early traditions, the self was seen as psyche or mind in later discourses. In the 16th century Descartes argued that the mind is distinct from the body and that the mind is singular in nature. “… when I consider the mind, or myself in so far as I am merely a thinking thing, I am unable to distinguish any part within myself; I understand myself to be something quite single and complete” (in Kenny, 1985:59). However, the idea of multiplicity also dates back even earlier, to the time of Plato, more than 2,000 years ago. Plato defined the psyche as comprising three parts: the rational self, the appetite, and the spirit (in Hommel, 2019). In contemporary literature the debate as to what the self is continues, with numerous disagreements between phenomenologists, neuroscientists and social psychologists (e.g., Gallagher, 2013; Menary, 2013; Strawson, 2018; Zahavi, 2014; Bachkirova, 2022).
In an attempt to reconcile some of these disagreements, Gallagher (2013) conceives different aspects of self not as independently existing but as the same ‘ingredients’ organized in certain patterns. He suggests calling each of these patterns a ‘self’ if they have a sufficient number of characteristic features to justify the status of a particular self (Gallagher & Daly, 2018). Some examples of selves include embodied self (core biological); minimal experiential; affective; behavioural; intersubjective interaction; psychological/cognitive; reflective; narrative; extended/situated and normative. This view of self embraces a pluralistic attitude that honours the legitimacy of different perspectives.
Bachkirova’s (2022) ‘Development of Self in Action’ (DSA) theory for coaching shares this pluralistic attitude but concentrates on three dimensions essential for coaching. These dimensions of self are self-as-locus-of-experience, self-as-actor, and self-as-story, with each having support from solid conceptual work and research (e.g., James 1890; Strawson 1999; Gallagher & Zahavi, 2008; Gazzaniga 2012). Multiplicity of self is described in this theory as related mainly to the self-as-actor. It is about different expressions of self in different situations, that demand a particular ‘mini-self’ to come to the fore and to be dominant when it is required by the situation. It is about multiple roles of the self in different situations – a mix of feeling/acting and telling a story.
This idea of the multiple recognisable selves in action has been shared by many thinkers who propose specific mini-selves. For example, Freud wrote of a self comprised of ego, superego and id, and Jung described autonomous partial subsystems existing within the psyche (Rowan, 1990; Schwartz, 1995; Lester, 2010). Other ‘multiplicity theories’ include the work of Swift, Gergen, Mair, Martindale, Redfearn and Crabtree (Altrocchi, 1999). Rowan (1990) lists the work of Lewin (subregions), Perls (topdog and underdog), Watkins (imaginal objects), McAdams (imagoes), Tart (identity states). Eric Berne’s (1964) theory of multiplicity is well-known through the popularity of his writing. Schwartz (1995) writes of internal family systems, a schema that places as much emphasis on relationships between aspects of self as it does on the parts themselves.
However, the philosophy of a ‘monolithic’ self persists, at least implicitly, across the broader organisational development domain in which most coaches are operating. This philosophy underpins approaches that frame the task of the coach as being to establish the best possible relationship with a single self in service of helping the individual to think differently about the challenges they face. Such coaches may over-privilege the role of psychometric instruments in characterising the nature of the ratee’s single true self in service of helping the individual choose how to think and behave differently.
The distinction between the single and multiple self is important and relevant to the discourse about 360° feedback systems. If the person comprises multiple selves, and these different selves appear in different scenarios, then people’s ratings of these selves would differ, as raters usually encounter different versions of this person’s ‘self’. Of course, many 360° feedback systems provide ratees with data specific to different rater groups, but they tend not to interpret this data in a way that helps the ratee understand how they are differentially perceived in different situations. If, for example, one aspect of the ratee is strongly identifiable when they work with clients, that self might not be witnessed by some of their direct reports, some of their peers and their line manager. In summarising the data this data might be marginalised or even treated just as an anomaly. In leadership meetings another aspect of the ratee might be witnessed by their peers and line manager, as at social events - by multiple individuals from multiple rater groups. These aspects could be given priority in the overall description and diminish the richness of information potentially available.
Moreover, many coaches encourage ratees to determine what their 360° data is telling them about their authentic self. The ratee is supported in pushing away or correcting aspects of the self that are seen to be unhelpful, usually with reference to a generic set of leadership competencies. However, there are many issues even with understanding what authenticity is. If it is not just a ‘discovery’ of a ‘hidden true self’ but the way individuals are familiar with their various selves and allowing these selves work better together or the most effective self stepping forth in a given situation (Bachkirova, 2022; Carter, 2008; Lester, 2010; Rowan, 1990, 1993; Schwartz, 1995), then the traditional approach does the ratee a disservice. In the same way, a coach who actively discourages the ratee from exploring every aspect of their self, might be encouraging them to think simplistically about the nature of leadership. They might be missing the opportunity to help the ratee think about the complexity of their nature and therefore adapt to different situations accordingly.
To examine a different way of delivering 360° feedback, free of the assumptions we have critiqued so far, we designed a practical experiment in a real situation of delivering 360° feedback, through which we began to explore how receptive both raters and ratees might be to exploring 360° feedback through a multiplicity of self lens. The experiment was designed with the aim of testing the validity of our assumptions and to see if a different approach might usefully change the ratee’s experience of receiving feedback.
Case study
In this study we collected feedback from four different raters on the same leader, asking those raters to describe how they experienced the leader in different situations, and therefore potentially in different roles. Each rater had previously completed an online 360° feedback questionnaire for the ratee. The study comprised five steps.
Step one: interview design
We designed a simple rater interview protocol designed to elicit how raters experienced the ratee in different situations. The interviewer began each interview by explaining the purpose of the study, confidentiality, and the use of data. The coach then asked each ratee:
Can you help me broadly understand your ratings with regard to each of the categories? Intended as a warm-up question, designed to help them recall their participation in the online 360° feedback exercise.
In how many different contexts/situations do you experience (leader’s name)? Interviewer to further explain ‘contexts/situations’ if necessary.
For each of those contexts/situations can you describe how you experience (leader’s name) specifically in that context? Interviewer to probe as appropriate.
What aspects of (leader’s name) behaviour do you appreciate in this context? How might (leader’s name) be more effective in this context? Interviewer to probe as appropriate.
Step two: sourcing participants
We spoke to the HR Director of a data/technology company and explained that we were looking for a volunteer to participate in an experiment, the purpose of which was to explore the use of 360° feedback instruments. We asked that the volunteer had recently completed a 360° review. We explained how the process would work and that the volunteer would benefit by having access to further perspectives on how he/she was experienced by others. The HR Director nominated ‘John’ (not his real name), telling us that he was keen to participate. John was an executive manager leading a client-facing team of more than 80 people.
Step three: interviewing ratee
The coach met with John and explained both the purpose and the process. We explained that we were particularly interested in understanding how people might act differently in different contexts. He was informed that the process would entail conducting 30–45-minute interviews with raters, with the content of the interviews being confidential, and following these – another interview with him. John provided the coach with the contact details of four raters willing to participate and the coach made direct contact with them to set up appointments. The four raters included peers and direct reports.
Step four: interviewing ratees
The coach interviewed the four raters individually, adopting the protocol detailed above. Each interview was audio-recorded and transcribed.
Step five: collating contexts
After completing the interviews, we used the transcripts to independently identify seven contexts in which one or more of the raters had nominated during the interview. We then organised feedback from all the raters against each context. A high-level summary of the feedback data is presented in Table 1.
Contexts of feedback and characteristics of the relevant selves
Step six: presenting data to ratee
The coach met again with John to discuss the feedback received. The coach played back each of the role descriptions/feedback in turn, inviting John to ask clarifying questions and make sense of the feedback received. John responded to the first four sets of feedback data by considering why he behaved in those different ways in different scenarios and by describing his thoughts and feelings when in those situations. In responding to the fifth scenario, presenting to clients, John spoke for the first time about the discrepancies in his self and questioned his ‘authentic self’. The following extracts from the conversation with John illustrate how he makes sense of the apparent multiplicity of his self. The numbers that John refers to correspond to the numbers on Table 1and the order they were discussed in this conversation.
Unprompted, John has identified two selves; an authentic self and a commercially aware self and identified another context in which he operates (number zero, not in work). The coach then invited John to elaborate on his authentic and commercially aware selves:
John is struggling to define what ‘authentic’ means, to distinguish authenticity from the notion of being “open and transparent” and to reconcile being authentic with the need to act differently in different scenarios. He goes on to say that “the client context is a different context and therefore you need to think about operating differently because of that”. The coach then prompts John to expand on data from the other contexts.
Later in the conversation, John describes himself in the seventh context, the team leader context, in the following terms:
At the end of the conversation, the coach asks John to stand back and think about the way he experienced what has been discussed.
The conversation overall shows that John has not only begun to delve deeper into the different selves that he has identified but he has reflected on how a multiplicity perspective has shifted his experience of 360° feedback. He has recognised that the traditional perspective is about finding generic weaknesses, whilst the multiplicity perspective helped him develop a better sense of who he is in his multiple roles.
Discussion and practical implications
Our intention in this paper was to review the use of 360° instruments and to question some of the underlying assumptions behind their design and deployment. Following the critique of these instruments in the literature, we identified that systems theories and theories of self can provide insights into some of the issues that undermine the effectiveness of these instruments. As we positioned ourselves as more aligned with a complexity view on systems and the theories of self as multiple, we conducted a case study in an organisational context – to deliver multi-rater feedback in a way that is compatible with these theories. We interpret the results of this experiment as positive because it has increased self-understanding of the client, which was valuable to him, and such value for the client is in line with our position on the benefits of coaching.
At the same time, we can anticipate a potential argument that there is no point in self-understanding if there is no focus on what abilities need to be improved. This challenge is fair if the critics do not share our philosophical and theoretical positions in relation to coaching. To demonstrate how these different philosophical positions on complexity and the nature of self may affect our views on development of people in organisations and different means of supporting them we developed the following Table 2. It draws on many sources (e.g., Stacey, 2012; Maxwell, 2017; Bachkirova, 2022), and our own practice.
Aspects of feedback practice through the lens of two philosophical positions
In practice, the differences between these two perspectives are likely to be more complex and less polarised. However, we contrast them deliberately in order to highlight issues that we believe are important to consider for all involved in the use of 360° feedback systems. These differences in perspectives are not unique for multi-rater feedback – they are reflected in many contrasting views and even paradoxes in various theories of learning and development. For example, some traditions in psychology and philosophy are in line with ‘the paradoxical theory of change’ (Perls, 1969), which asserts that a real change in the self is only possible when a person accepts oneself as one is. So, the value of feedback for change would require first self-understanding and self-acceptance rather than a prompt for change. We would argue that such self-understanding and acceptance is better served by the view of complexity and multiplicity of self rather than the linearity and singularity of objectivist/behavourist view.
To demonstrate this argument in practice we can compare the two approaches to cases like the one we described. Typically, many feedback instruments provide their data in aggregate form. John, for example, in debriefing 360° feedback, may have been told to speak up more, engage more with others and to be more flexible in his thinking, because that is what the aggregate data suggested. This would not have been very useful feedback, since there are several contexts in which John is already demonstrating those behaviours, and there are some contexts in which John might usefully be more reserved and approach relationships with caution. Instead, we advocate a different approach that would encourage the ratee to think about how they act in different contexts and recognise the aspects of self that are dominant in these situations. Someone like John would be encouraged to identify contexts in which he is open and engaging and in which - more guarded. The coach would support John’s inquiry to get to know each aspect of his self, and to be more purposeful when each self is responding to the demands of the situations.
By comparing two different baselines for delivery of multi-rater feedback and describing our approach to practice we hope that we have presented a compelling case for coaches to question their philosophies on complexity and the nature of self. We strongly believe in the value for coaches to pursue more in-depth explorations of these ideas in order to articulate more clearly the philosophies that underpin their practice (Bachkirova, 2016; Jackson & Bachkirova, 2018; Lawrence, 2018). By describing these implications for coaches in the first place we acknowledge their ‘front line’ role of the delivery of multi-rater feedback and therefore, potentially - most influential role. However, this process begins much earlier with other stakeholders, such as coaching educators and organizational sponsors, who are also in the position to influence this practice.
In terms of implications for the educators of coaches, we believe that awareness and articulation of these perspectives can make a difference if they are recognised and discussed in the process of coaches’ development. If educators of coaches stimulate discussions of different philosophical positions and different theories of learning and of the nature of self, they would promote curiosity and encourage diversity of coaches in styles and approaches. It is unlikely that there will be soon a definitive answer as to the nature of self for them to present as evidence-based. There are no objective means by which to determine the nature of our self. However, learning about different perspectives expands the minds of coaches and promotes experimentation rather than fixed views.
Thinking about organisations that apply traditional approaches to 360° feedback, there might be more challenging implications. For example, these approaches may be useful when the ratee is working in a relatively simple environment that places few demands on their capacity to adapt to different situations. In this case, a directive approach to leadership development that Stacey (2012) called ‘coercive persuasion’ may work, but even then, it can “foster dependency and … block questioning and reflective thinking”. However, it is unlikely that such leaders can fit into today’s complex, dynamic, and unpredictable world of organisation. They will be bringing different aspects of self to different situations of leadership. Different behaviours and style may be required for different challenges, such that specific behaviours may be useful in some contexts but detrimental in others. A constructivist and multiplicity of self approach is more likely to serve the ratee to be better able to manage such complexity.
Conclusion
In this paper we have challenged the traditional deployment of 360° feedback in organisations. Without questioning the value of the multi-rater feedback in principle, we have identified assumptions that minimise its effectiveness. By juxtaposing two wider philosophical perspectives, we argued for and tested in practice a different approach that coaches can adopt when facilitating 360° feedback. This approach acknowledges and appreciates the complexity of self that is barely acknowledged in contemporary coaching literature and practice.
We believe that approaches based on the appreciation of multiplicity of self have a significant potential to be useful for leaders and their organisations. Leaders are increasingly being asked to be more adaptable in managing the challenges of today’s complex world. If organisations are to help leaders further enhance their levels of adaptability and agility, then these organisations need to support systems of feedback that help leaders understand how they are perceived differently in different contexts. Then coaching that explores such feedback can help leaders to become more aware as to the processes that are involved in their multiple self-expressions and actions. This, in turn, could enable leaders to be more purposeful and effective in their endeavours.
We hope that future research into the deployment of 360° feedback will help to explore various approaches to this practice with an appreciation of how philosophically different their foundations might be. We would like to see more studies explore the experiences of ratees who are at the receiving end of different approaches to 360° feedback. It would be valuable also to explore how raters approach their task and what factors make their experiences of rating difficult or easy. Such research should enrich our understanding of feedback associated practices in organisations and potentially contribute to the theories of self in work environment. We hope that our case study into ‘360° feedback based on the multiplicity of self theory’ has paved the way to further research into this complex phenomenon.