Learning From a Failed Mixed Methods Child Art-Therapy Research Project

Abstract

This paper critically describes a mixed methods study focused on the development of a measurement instrument and the effectiveness evaluation of an art-therapy intervention for children. The project combined an outcome research, conducted for the validation of a psychotherapy program for children with behavioral and emotional problems, with a mixed qualitative-quantitative study for the development and validation of an observation sheet for the analysis of video recordings in the assessment of children’s self-image. Even if the quantitative analysis showed that both the instrument and the program were adequate, the research program cannot be considered a success because of several shortcomings. We describe the strengths and the gaps of the qualitative study, as well as the likely motives for the failure of the entire project. We also include lessons learned and recommendations.

Using Video Data in Analyzing Interactions in Education and in Psychotherapy Research

Ethnography involves the use of participant observation in natural contexts, and videography, an interpretive research utilizing video technology in context, is an emerging observational approach in ethnography (Lynch & Stanley, 2018). Qualitative visual methods can provide useful tools through which to generate data and to augment observational studies with children (Lynch & Stanley, 2018) and they are particularly useful with young children, because these rely more on nonverbal communication than adults (Curtin, 2001). Video data analysis is a complex qualitative research task because it involves abstracting and creating new knowledge, but it is also a reliable research instrument since the information is recorded and can be transferred for later study and analysis, and may be repeatedly examined at any convenient time (Xiao et al., 2004).

Another systematic method to analyze verbal and nonverbal aspects of the interactions in educational and occupational settings using video data is the context analysis or “microanalysis” method (Erickson, 1992), that involves collecting simultaneously information about movement, body language, facial expressions and performance in various activities (Miller Scarnato, 2019). This process has five steps that need to be followed after collecting the video data: (a) examination of the entire session without pausing or slow motion and taking notes; (b) highlighting the three phases of the event (beginning, focus of the activity and conclusions) by locating major shifts in activity and then playing and replaying the video back and forward for closer examination; (c) highlighting the skeletal structure of the event, the organization of each of the three phases and the linkages between them; (d) transcriptions of the statements and nonverbal communication of the participants, guided by the purpose of the analysis; (e) comparation of the analyzed segments with the remainder of the video data in order to determine the degree of representativeness and examination of the entire video record for exceptions (Camic et al., 2003).

In more recent studies, the analytic induction process is applied on video recorded data, with or without Erickson’s microanalysis method, with the help of computer-based programs specially designed to make it easy to extract transcripts and to produce categories. For example, Rostvall and West (2005) designed a video study for the analysis of teacher-student interaction in a music lesson focused on the dynamic aspect of the interactions, different from the static description of the spoken language which is traditionally used in video analysis. The analysis was divided in three levels: a detailed description of how teachers and students act and interact, a systematic analysis of the patterns of interaction, and an interpretation on a macro level of why they are interacting in the way they do. The transcription of the video was automatically generated using The Analyzing and Reporting Transcription Tool (ARTT) (Rostvall & West, 2005). Later, they designed TATOO, a video analysis software tool specialized in transforming the systematic analysis of video material on teaching–learning interactions in an explicit and open process. The program identifies syntactic sentences by variations in the prosody of speech as well as by gesture, gaze, or longer events, which then are coded into categories and lead to a complex understanding of interaction at several levels: individual, interpersonal, institutional and societal (West, 2007).

The integration of a reflexive attitude into experiential qualitative psychology is supported by two arguments: first, since we are social beings, our experiences must be understood within the context in which they happen; second, we experience and interpret the world from a particular perspective (Shaw, 2010).

Video data can be particularly useful in analyzing children’s behavior, emotions and attitudes in social contexts, since it studies children as they like to behave: in a playful and imaginative way (Yamada-Rice, 2017). The activity of constructing a “world” in a creative way has the capacity to provide visual and verbal metaphors that enhance the child’s self-understanding (Mannay et al., 2017) and the exercises that are created in order to induce such activities are tools of elicitation leading to the analysis of children’s attitudes about themselves and the world around them. This is a complex social-interaction activity and audio-video recordings are a suitable way to collect this type of data (Heath et al., 2010); it has been recently argued that innovation in visual methods is an important key in driving visual research with children forward (Stirling & Yamada-Rice, 2015).

The analysis of video data in ethnographic research usually involves a grounded theory approach, when the video content is converted into written text for analysis, and the observed events are coded into a checklist of objects or processes. Since the video is the most effective way to capture the context, more qualitative strategies of video data analysis need to be developed, in order to maximize the positivistic tone of the process.

Analyzing and interpreting the children’s interactions in videotaped social and education settings is a complex task and a large number of studies focused on this issue. One approach is to “give the voice” to the children themselves and to allow them to express the meaning of their art productions. Lomax (2012) used a group of 14 children as co-researchers in a project using photography and video data, and the analysis revealed the way children respond to, organize themselves and talk about their creative projects. In the current study, this was done in the both pre and post-test sessions, and also in the intervention sessions, by inviting the children to express their thoughts and feelings regarding the drawings and other art work they had created; these sequences were filmed and were part of the video recordings, so that the children’s points of view had to be taken into consideration by the experts who analyzed the recordings.

Children can be reliable in giving valuable information about their own experiences through creative activities and it is important to develop research methods that involve child participation, so-called child-friendly methods (Groundwater-Smith et al., 2015), such as drawing and clay-modeling. This type of setting was previously used in studies of non-directive psychotherapy with children. For example, in psychoanalytic psychotherapy, Mannay and her colleagues (2017) used video-taped session of “sandboxing” in order to allow children to create scenes representing their images about school or their projections about their future (Mannay et al., 2017).

Involving children as research informants also brings with it some threats, and delineates possible sources of failure: the researcher may wrongly assume that children are less competent participants and may avoid to include them as active contributors; the adult-child relationship may affect the children’s comfort in expressing their thoughts and emotions; children may have different communication tools than adults and may rely more on nonverbal language and silence (Curtin 2001).

Combining Outcome and Process Research in Psychotherapy

Art-therapy techniques have a long history in child psychotherapy, because they provide children with an environment and materials familiar with their main activities in school and in play (Oaklander, 1988). Art is used in such techniques “in a supportive setting, to facilitate growth and healing; it is a process of discovering ourselves through any art form that comes from an emotional depth” (Rogers, 1993, p. 29).

Psychotherapy research has debated for quite some while now if outcome or process methods are more adequate for the validation of intervention programs (Norcross, 2011). As a possible solution to this debate, combined process-outcome approaches have been proposed, i.e. approaches that both measure process variables and test if they relate to psychotherapy outcomes (Timulak, 2008).

We report on a study that was based on this latest trend and that combined outcome quantitative research with process qualitative research, in order to validate an art-therapy program for children with behavioral difficulties, and to develop a process research measure associated with this domain, for the measurement of self-image in psychotherapeutic interventions with children.

This combined qualitative-quantitative research approach is particularly suited in child psychotherapy, especially with children who manifest behavioral problems associated with emotional impairments. Studies focused on the ability of children to recognize emotions in art, especially in drawing, show that children are able to recognize and label positive emotional expressions earlier than negative ones. Also, emotions like happiness, sadness and fear are easier recognized for young children, while anger is more difficult to detect and understand (Misailidi & Bonoti, 2008).

When children manifest behavioral and emotional impairments, parent responses to the child’s emotions are often inadequate (Greenberg, 2015); this is important because children learn to identify and regulate their emotions based on their parents’ responses, and family discussions of emotions shape this important process. Also, perceived parental approval is positively associated with children self-reported expression of their emotions (Scherr et al., 2019). When children are not able to regulate their negative emotions, therapy may provide a corrective emotional experience through which children may learn how to identify and label their difficult emotions, like anger.

The Study (Methodology)

The initial study had two goals: to assess the effectiveness of and art-therapy program for children with behavior problems and to generate validity evidence on an assessment instrument developed with this occasion—an observation sheet for the evaluation video recorded behavior for such children. By constructing an instrument to analyze these video-recorded psychotherapy sessions, the focal study intended to add to the current literature dealing with how to analyze video on the usage of arts in education and therapy, as well as to that dealing with the involvement of children in qualitative research as participative informants.

The study was approved by the Research Ethics Committee of the University of Pitesti, Romania. Written informed consent was signed by the parents of the children involved in the research.

The decision to develop and validate an art therapy program for children with behavioral problems was based on several assumptions. First, using art-therapy to help children regulate their emotion is a widely used method. For example, McLachlan and her colleagues (2009) validated an experiential emotional regulation training program for elementary school children (aged 6 to 8), that helped children to learn to identify emotions, deal with emotions and practice expressing emotions in a healthy way, using drawings, puppets and other experiential play therapy methods (McLachlan et al., 2009). Other studies showed that the ability of children to infer others’ emotions from situational cues continues to develop throughout primary school years (Fine et al., 2006), and children who have a better ability to name emotions and recognize emotions in others have also an increased ability to function effectively in social relationships (Miller et al., 2005).

Second, children with behavioral problems also have important emotional difficulties, especially a low ability to regulate their negative emotions, particularly anger. Most parents are shocked when their child expresses anger, become themselves anxiously insecure or too harshly controlling, and respond inadequately to the child, which may increase the child’s hostility and may lead to even more aggression (Greenberg, 2015). When children have such a pattern of interactions with their parents or other significant adults (e.g., teachers), they become unable to regulate their anger. Experiential art therapy may provide these children with an opportunity for corrective emotional experiences and also with an opportunity to learn emotional regulation.

In order to maximize our capacity to capture the context and interactions in children group therapy sessions, we decided to develop an observational analysis instrument. The development of this instrument is important for both our study (i.e., enabling us to capture the change that resulted from the art therapy program), and for later usage by therapist in their professional work, in similar domains as our study. Self-image is an important domain that is and should be assessed when working with children: we know that, in general, people with a positive self-image exhibit more positive behaviors than people with a negative self-image (Scheidin & Armelius, 2008), and that youths with both conduct disorders and depressive symptoms present self-image disturbances compared to normal children and adolescents (Koenig, 1988).

Connected directly to our study, the assessment instrument was meant to be used by therapists, after watching video recordings with the focal child’s therapy sessions. Therapists would analyze the video data directly, without a transcription, and would manually code their observations using a pen and paper observation sheet. If a success, this could have become a valuable process research instrument for child psychotherapy, since an important part of interventions with children rely on play therapy and art therapy techniques, where children express themselves less with words and more in nonverbal and metaphorical ways. Also, self-image issues are present in a majority of psychological conditions in children and are a major focus in psychotherapy processes, regardless of the theoretical approach embraced by the therapist. Given the fact that eliciting data from video recordings of activities performed by children, for usage in clinical settings, was not particularly well studied at the time of our research project (2007), nor where there any other tools widely available to assist in this task (computerized or otherwise), we considered that the development of such an instrument would be worthwhile, and that grounded theory and Erickson’s microanalysis method would be frameworks for this approach.

Our research design involved four different phases. First, we developed the observation sheet with the help of groups of experts, following two steps: (1) six psychotherapists generated items under the form of observable behaviors relevant to self-image; (2) another group of 10 psychotherapists generated the answer choices for each item, as behavioral anchors on a 5-point Likert-type continuum.

Second, we created two dynamic group exercises called “The Orchard” and “Our World,” that were based on classical projective techniques (the draw a person test and the tree test), with the purpose of producing projective data on children’s self-image.

In the first exercise (“The Orchard”), children were seated around a table, where a large piece of paper and colored modeling clay were available. They were invited to choose a piece of modeling clay and, with the eyes closed, to interact with it by modeling and in the same time listening to a therapeutic imagery suggestion. They were instructed to model a small tree, and then to open their eyes and place their tree on the white paper. Then, they were asked to look at their little tree and imagine that it would like to be completed in order to become more beautiful than it is, and to add more features so that it may become a happier tree. The children voices were used now: each of them was asked to tell something about his/her little tree, but imagining that they actually were the tree, self-presenting itself to all the other children. Then they were invited to imagine that the table is an orchard, and to place the little trees where it feels best for them and to draw the rest of the orchard. Next, the therapeutic suggestion followed the idea of seasons change over the orchard. The trees passed all four seasons, spring, summer, autumn, winter and spring again. In every season, every child described what was happening with their tree and how their little tree was feeling about it. Finally, each child was asked come up with a name for the whole drawing.

For the second exercise (“My World”), children were also sitting around a large table and materials such elements of nature (shells, pieces of colorful plants, plastic wildlife animals and Lego bricks) were made available for them. The children were instructed to choose a drawing instrument that attracted them more (crayon, pencil, brush etc.) and then to draw a little man as “you would like yourself to be” on a small piece of paper. After everyone finished drawing, each child stuck his/her drawing on his/her own chest. Next, children were encouraged to talk about the drawing, to present themselves in front of the group: “What is your name? What do you like to do most? What do you usually do? What do you like mostly about yourself? What do you dislike about yourself?” Then, children were encouraged to unstick the drawings from their chests and put them on a large sheet of paper, and then they were invited to further build a small world around these characters. In the end, each child told the others what he/she felt like in this world that they had built.

Third, we implemented a pre-posttest experimental design, with 12 first and second grade children, aged 7–8, separated into two groups: seven in the experimental group (children with behavioral problems) and five in the control group (typical children). Such a small number of children is often used in this type of complex qualitative research, since the amount of data elicited from video recordings is very large and the time for analyzing is usually quite extensive. For example, Yamada-Rice (2017) used a small group of seven children aged 3 to 6 in a study aimed at exploring their interactions and their understanding of a number of images (Yamada-Rice, 2017).

In the pretest phase we conducted one session for each group with the exercise “The Orchard.” Then the children from the experimental group received an intervention under the form of 10 sessions of experiential psychotherapy. In the posttest phase we have conducted one session for each group with the exercise “Our World.” Both the pretest and the posttest exercises were video recorded. One important decision when using videos in social-settings research is whether to have a fixed or a handled camera (Heath et al., 2010). In the present study, since the participants were children actively engaging in mobile activities, the researchers choose to have a mobile camera handled by one of the researchers, a psychotherapist, who could direct the focus to those images that best represented important psychological content.

The therapeutic plan consisted of 10 sessions (20 hours) involving children in a series of provocative exercises specific to humanistic-experiential approach, based on art-therapeutic techniques (drawing, sculpture, modeling, dance-therapy, music-therapy), psycho-dramatic techniques (drama, role-playing, playing with puppets) and metaphoric techniques (metaphoric scenarios, that create an analogy with real-life situations). Each session started with the usage of such provocative techniques, followed by a group discussion involving all children and aimed at facilitating identification of negative emotions, their likely causes and effects, and at helping the children to discover new alternative modes of interaction and self-regulation.

Fourth, 33 psychologists watched the videos in a group session and scored separate observation sheets for some of the children present in the video. Each child was screened by five psychologists independently, who watched the video of the group session, but who focused only on one child at the time.

Measurement Qualities of the Observation Sheet

Reliability results

Inter-rater reliability was medium to strong (Spearman’s ρ = .50, p = 0.00), which indicates a high inter-rater agreement. Analyzing the data separately by group and type of exercise, we noted that Spearman’s correlation coefficient ranged from .54 to .65. Internal consistency (Cronbach’s α coefficient) indicates a very good internal consistency (.91), which allows us to say that the designed instrument offers reliable measurement.

Validity results

Content validity was ensured by the fact that experts were involved in developing the exercises, as well as in writing the items and response options. In order to assess criterion validity, we used the t test for independent samples, which revealed a significant difference between the group of children with behavior disorders and the group of typical children in the global score obtained on the observation sheet (t = 2.29, p = 0.03, d = 2.49). The observation sheet successfully discriminates between the two groups of children, which is an indicator of a good validity.

The good reliability and validity results obtained for the observation sheet are important for our study, reflecting how much reliance we can place on the reported results, but are also important for the continued usage of this instrument in other studies and by practitioners. While this is not a sophisticated psychometric tool, and the current study did not attempt to report complicated psychometric analyses, it is important to know that the qualities of the instrument are very good, and that the instrument can be used with confidence in professional practice.

The Effectiveness of the Intervention Program

The construct targeted directly by our art psychotherapy program with children with disruptive behavior disorders was self-image. To test the hypothesis regarding the efficiency of our program we measured this criterion through the total score obtained by participants on the observation sheet. These scores were captured both at the beginning and at the end of the therapeutic program, and participant scores in these two instances (before-after) were then compared. Due to the small sample size and the asymmetry of the distributions, we decided to use nonparametric statistics; more specifically we used the Wilcoxon paired sample test in order to evaluate the changes induced by the psychotherapy program in children from the experimental group. We found improvement of self-image with a statistically significant difference between the two measurements of the total self-image score before and after the intervention: z = −2.37, p = .02. We concluded in this context that the therapeutic program had the expected result, leading to an improvement of self-image in participating children.

The qualitative analysis of the video data also revealed interesting results regarding the perception of participating children about themselves and regarding their evolution in therapy. For example, at the initial testing moment, the chromatic of their artistic constructions was precarious or large and disharmonious, whereas at the final testing it was mainly large and harmonious; at the initial moment of testing the children rarely explained what they were doing, whereas at the final testing they tended to do this oftentimes; at the initial testing moment children reflected apathy or violence, whereas at the final testing they tended to be calm or need-centered but non-violent; at the initial testing most of the children were either rejected or isolated, whereas at the final testing most of them were accepted and even liked; at the initial testing children tended to have no role inside the group, or tended to imitate certain roles or combine personal elements with imitation; at the final testing they tended to imitate less, although the assumed roles were simple.

Reflections and Recommendations

This study adds to the literature on how to analyze video data on children interaction and on the usage of arts in education and therapy, as well as to the perspective of involving children in qualitative research as participative informants. We consider this research project a failure in spite of the apparently encouraging results, and attribute the reasons for failure to the fact that the project was not well prepared. We have made an assumption at the inception of our project, without fully contemplating the consequences of that decision—we assumed that the observation sheet developed by the group of experts will be usable in the pre-posttest assessments of our experiment and will offer credible evidence as a stand-alone measure on which to validate the intervention program. We furthermore did not develop the measure before the actual intervention project and did not conduct a previous a pilot study but only relied on the experience with children that the various experts had had, basically developing the measure during the actual intervention. This makes our results describing the program’s effectiveness less credible, even if the statistical results are indicative of a success. Unfortunately, the results of the quantitative study are not reliable and cannot be generalized; more than anything they are less credible and thus less likely to be reported and to be considered as a contribution to scientific discourse. The outcome study was therefore compromised.

There are also three limitations based on elements of faulty design in the qualitative part of the study. First, we intended to follow the inductive approach in qualitative research, but the observation sheet was constructed prior to watching the videos, which is a breach with the principles of the inductive approach. The subjectivity of the experts who have developed the observation sheet could have affected the structure of the instrument, since they only relied on their previous experience in working with children, but the situation created in the experiment is not entirely typical for a normal therapy session. In this context we point out that “although it is possible to use preexisting categories of behavior and impose them on video data, qualitative research historically has emphasized the emergence and descriptions of categories inherent to the research context” (Camic et al., 2003). The researcher’s ability to set aside assumptions about what is important in a given session and to be open and experience the fluid wholeness of the video record is a key value in this type of qualitative research (Collier & Collier, 1986). The correct approach would have been to have videotaped an initial session with children and to have asked our experts to generate items and finally the observation sheet only based on watching the video records of this session.

Second, we failed to use a clear methodological framework for data analyzing. For example, we didn’t follow the steps prescribed by the grounded theory approach in developing the initial form of the observation sheet, since we didn’t construct axial and core categories; the instrument only generates a global score for self-image.

Third, we have induced bias through our decision to analyze the audio and video tracks of the recordings simultaneously. While there are authors who argue for this approach, emphasizing the unity of the analysis (Collier & Collier, 1986), others recommend separate analysis; Erickson & Schultz, 1982, since visual cues are different from audio cues and it’s oftentimes difficult for the evaluator to pay attention to both types of information at the same time.

We Didn’t Do Everything Wrong

Our study aimed to cover a methodological gap on instruments for measuring self-image in children. Usually, self-report questionnaires and in-depth interviews are used to capture the person’s sense of self, but children are less capable to understand their own feelings and to verbalize things like what they think they look like, how they see their personality, what kind of person they think they are, what they believe others think of them, how much they like themselves or they think others like them. For adolescents there are some self-report tests such as the Offer Self-Image Questionnaire Revised–OSIQ-R (Offer, 1972), but younger children need adult-report tests to cover this kind of issues. Psychotherapists who work with children are aware of the importance of the self-image in defining and understanding the children’s emotional and behavioral difficulties, and direct assessment of the child’s self-image provides useful data when appraising the outcome of the therapeutic intervention (Hughes, 1984). When the purpose of the observation is to unveil unspoken events or feelings with personal signification, videos are especially effective, since they capture authentic responses and embodied expressions that could be otherwise overlooked (Li & Ho, 2019).

We used multiple analysis performed by 33 psychologists who coded the video separately and filled in the observation sheet—this permitted triangulation, which is an important way in which to establish validity in qualitative research. Also, multiple analysis increases the credibility of the qualitative research.

Conclusions

Our study reports on an attempt to validate an intervention program in humanistic-experiential psychotherapy for children with behavioral problems. These children have difficulties in regulating their negative emotions, especially anger. The chosen therapeutic approach helps children to identify their own and others’ emotions and offers a corrective emotional experience, through which the child learns how to cope with negative emotions and to express them in a healthy way. Most of the psychotherapy programs to that date were cognitive-behavioral programs, that tend to ignore or work less with dysfunctional emotions that form the basis of problematic behaviors. We have developed a program that offers children corrective emotional experiences. A drawback is the fact that we did not have a sufficiently large group of children on which to collect data, but the conclusions both on the qualitative approach and on the instrument used are encouraging. In order to collect data, we have developed an instrument to analyze video recordings of the therapy sessions, and to capture both verbal and nonverbal cues (body language, face expressions, tone of voice and emotions expressed through voice, movement etc.). Previously this analysis was only done by transcribing, coding and categorizing behavior, and ignoring the nonverbal aspects of communication, which are extremely important in communicating emotions. We thereby provide therapists with an instrument that facilitates the analysis of video data in therapy and education. We have, however, based our approach rather on clinical observation than on a clear qualitative research framework.

In her review of the scientific literature about the use of video in social science research, Rosenstein (2002) outlined some basic rules for all methods of observation: one must choose to observe an activity representative of other activities or principles within the frame of reference of the observation; the chosen to be observed activity should easily be broken down into manageable and meaningful units of analysis and then easy to be reassembled after the analysis; the method of observation must allow reliability and validity checking; the registrations of the observations needs to be concrete and understandable and the researcher should avoid vague and overgeneralized language (Rosenstein, 2002).

Then, when analyzing the video data, a collaborative approach would probably be more suitable than the individual reflections of each expert separately, in order to enhance insight and extract significant psychological meaning. In 2019, Li and Ho did a study close to ours when they analyzed data from play therapy groups with old patients diagnosed with dementia. The authors used collaborative analytical approach in the data analyzing phase, meaning that the researchers, professional practitioners and volunteer patients (who participated to the intervention) were watching the video together in group sessions and they shared insights and interpretations of the events (Li & Ho, 2019). This kind of group analysis is known as co-inquire group (Banks et al., 2017).

However, it’s important to note that, at the time we did the original research, no framework was published about how to systematically analyze video data. In a paper published 2 years later, in 2009, Marie Jeanne McNaughton wrote “When I began to work on the analysis of the numerous actions and interactions caught in the video recordings, I found useful advice from a number of sources (…) but no one text that would give me an overview of the steps I needed to take nor the analytical tools I would have to employ” (McNaughton, 2009, p. 28). In her research, she adapted several processes from discourse analysis in order to analyze data obtained from video recordings of video drama lessons.