The Oral History of Evaluation: Joan LaFrance,Indigenous Evaluation,and Attention to Culture in Evaluation

Introduction

Since 2003, the Oral History Project Team has conducted interviews with individuals who have made substantial contributions to evaluation theory and practice. Many of the previous interviews were conducted with individuals who were present at the founding of evaluation as a distinct field. This oral history highlights an individual who was involved in the development and fostering of evaluation involving an important, and all too often ignored population—that is, members of tribal nations and the organizations intended to serve them.

This interview with Dr. Joan LaFrance continues the effort represented in a recent oral history of Edmund Gordon (Mark et al., 2023) to capture the reflections of individuals who have made important contributions to broadening the perspective of evaluation in ways that are more culturally responsive. Three long-time members of the Oral History Project Team (Valerie J. Caracelli, Melvin M. Mark, and Robin Lin Miller), along with Nicole Bowman, interviewed Dr. LaFrance on September 13, 2023. ¹ The authors have edited the interview transcript for clarity, length, and content. Dr. LaFrance reviewed and approved the final product prior to its submission to the American Journal of Evaluation.

Joan LaFrance, a member of the Turtle Mountain Band of Chippewa, owns and operates Mekinak Consulting, a management and evaluation service specializing in educational program evaluation, research, and management studies. Dr. LaFrance received a master's degree of public administration from the University of Washington and a doctorate from Harvard University's Graduate School of Education. She has served as an adjunct faculty member in graduate programs at the University of Wisconsin-Green Bay, University of Washington, Western Washington University, and The Evergreen State College. Dr. LaFrance has extensive experience in the evaluation of programs in Tribal Colleges and Universities, in tribal and Indigenous communities, and for nonprofits. She is a founding member of the Indigenous Peoples in Evaluation (IPE) Topical Interest Group (TIG) of the American Evaluation Association (AEA). Dr. LaFrance is best known among evaluators for her ground-breaking work on evaluation in tribal contexts. She was the lead author of the 2009 book, AIHEC Indigenous Evaluation Framework: Telling Our Story in Our Place and Time, which was prepared in conjunction with the American Indian Higher Education Consortium (LaFrance et al., 2009). Dr. LaFrance has described and expanded on this framework in several other articles, chapters, and presentations, and she has shared the framework and her experiences in numerous workshops, webinars, and other training venues. We heard echoes of Dr. LaFrance's and her colleagues’ and followers’ work in the Biden White House's memoranda on Indigenous knowledge in policy making, research, and evaluation (Executive Office of the President, 2021, 2022).

In this interview, Dr. LaFrance describes her pathway into evaluation, such as the transition from her earlier work administering a program, as well as her experiences at Harvard. At Harvard, LaFrance studied with Carol Weiss, a well-known figure in evaluation. Notably, Dr. LaFrance discusses the Indigenous evaluation framework, its key foundations, and how it came into being. She also talks about her experiences in AEA and as a sole proprietor of an evaluation consulting business.

Interview with Joan LaFrance

Mel: It is nice to have you here. As a major thought leader with respect to Indigenous evaluation, you’ve been on our list for a long time. We’re delighted to be able to talk with you today. Thank you very much. I wonder if we could begin by tracking your pathway into evaluation, starting with how you came to have evaluation as part of your focus at Harvard. Your path to the doctorate may be a good starting place.

Joan: Well, I was working as a program administrator in the late 1990s, directing a program at the United Indians of All Tribes Foundation here in Seattle. I had gotten a BA in history that really hadn’t prepared me to do the work in nonprofit public service administration. That's what I ended up doing, though, through resources that became available to the Indian community here in Seattle. Someone was needed to direct what we can call a war on poverty program. This is basically how I got started at the Indian Center and I continued to do that kind of work over the years. Also, I was doing some training on multiculturalism when it was in fashion in Seattle public schools. I always knew that I needed to go back and get a master's degree and for some reason assumed it would be in the field of education. In fact, I even went so far as to get the application and started filling it out.

At the program I worked on, we were doing curriculum development. The project grant had set out a work plan that was unworkable. It assumed parents could develop usable curricula without providing them adequate training and support. So, we ended up doing a completely different take on that program. When I was told that we were going to get an external evaluator, I had all the negative reactions. I felt like they were going to blow our cover. They’re going to see what we’re doing to make this grant work. But we had to do it the way we were doing it.

Anyway, we had the evaluation. I learned so much from the evaluator, Marc Lindenberg. He was such a delightful faculty member at the University of Washington [and later Dean of its Daniel J. Evans School of Public Affairs]. I just raised my hand and said, “I want to learn how to do this.” I had been thinking about going back to school. I hadn’t looked at public administration, but Marc was in that field. I ended up going to the University of Washington and got an MPA. And then to further my abilities, I applied to Harvard. I was lucky that at that time the Graduate School of Education had a grant for Indian people. It happened to end right when I applied. From that same Office of Indian Education in the Department of Education, though, I got an individual award which allowed me to go to Harvard, where I met Carol Weiss. She ultimately became my advisor.

When I came home, I acted more on my public administration credential. I got a job working in the city of Seattle in the budget office, and I was there for 15 years. But I was able to negotiate a part-time position towards the last few years after I got my doctorate so I could hang up my shingle, to use an old expression, and start to work as an independent evaluator. Initially I thought I needed to get an academic appointment to do evaluation professionally, but I had no interest in leaving Seattle or the Pacific Northwest where I am deeply rooted. So, I realized I could just go out on my own as an evaluator.

I slowly created a little niche in Indigenous evaluation. Over the years, I got to know Carrie Billy (long-time President and CEO) at the American Indian Higher Education Consortium (AIHEC). I had met Richard Nichols, who was also an evaluator, while I was in school. He was in DC at that time but he also went home, to Santa Clara Pueblo. After we had both been doing evaluations in tribal settings, we were musing about how we altered our practice from what we were taught when we did our graduate work. We were interested in documenting those practices when the stars aligned. I mean everything we needed came together to let us go beyond the conversations Richard and I were having, and to work on developing the Indigenous framework. At AIHEC, Carrie was increasingly interested in doing professional development in evaluation, because the tribal college people she worked for identified it as a real need. They got so many grants that required evaluation, which was annoying for them to deal with. So, Carrie was interested in having more professional development about evaluation. And the National Science Foundation was putting out a grant program. I can’t remember what the exact category was, but Elmima [Johnson], ² who was very active at that time, encouraged us to go for the planning grant in that category and then eventually an implementation grant. So, Richard and I and AIHEC were able to get the resources to really look at what Indigenous evaluation would look like. Through focus groups, through our own research, and through our own considerations, we developed the AIHEC product.

Mel: This is a wonderful overview of a good part of your career. You mentioned aspects of your roots in Seattle, as well as the work you were doing with Indigenous groups even before you got into graduate school. I wonder about how your roots as an Indigenous woman, how issues of identity and commitment, might have influenced your pathway to and through Harvard—or after, if that's more relevant.

Joan: Actually, before Harvard is relevant, so I’ll start there. My parents worked for the Bureau of Indian Affairs. My mother was very light-skinned. My father was identifiably Indian. My mother grew up wanting nobody to know that we were Indian if that was possible. The message she gave me was, “You don’t have to tell anybody.” My father was very different, very proud of it, but also a very quiet force in our home. When I got to Seattle University in the early 1960s, I wanted to get more involved in an Indian community than just being the daughter of Bureau people and knowing Indians only socially. So, I went to the Indian Center and talked with them about how I might be involved. There was an organization called the American Indian Women's Service League, which in fact was the sponsor of the Indian Center. The director at that time obviously encouraged me to join it, and I volunteered to be a tutor at the center. I was kind of unique at that time being a college student and I was really embraced by the women in the American Indian Service League as a young member going to college. All that encouragement helped lead to Harvard. I had quite a career before I went to Harvard. In 1965, immediately after I graduated from college, there was a job for me at the Indian Center. Then there was a job for me in the school district to do multicultural training. Then there was another job to be a social worker, which is the one I hated the most. I worked on what's now called TANF (Temporary Assistance for Needy Families), then called Aid to Families with Dependent Children (AFDC ³ ), in our public welfare organization. Even there I specialized with Indian clients. We did fun things. My AFDC caseload formed a little club through a partnership with Seattle Community College. A faculty member had a grant so she could support evening get-togethers for us, and we developed some programming and that sort of thing. So, there was always this interest and ability on my part because of opportunities that opened up. I knew I had to go back to school, as I said, and the faculty member who evaluated the curriculum development project I managed was from the graduate school of public administration. Once he knew of my interest in graduate education, he promptly recruited me. From there, I was able to go on to Harvard for more training, without any idea who Carol Weiss was or that she would one day become my advisor and a friend.

Mel: You mentioned your reaction to an early evaluation of a program you were working on, where you had the kind of anxiety that evaluators often see, but then had a very positive experience with the evaluator who was there. Is that when your interest in evaluation as a pathway developed, or did you have it earlier? Could you talk more generally about that development and your interest in evaluation?

Joan: Yes. I had no understanding of what evaluation was until I met Marc Lindenberg. It was Marc who came to United Indians to do our evaluation and, as I said, the way he did the evaluation impressed me. He walked into an organization that was in chaos really, and I kind of worried about what might happen because of that. But he very deftly worked around some thorny political issues. In particular, I wondered how he would deal with the fact that the director of the organization had basically spent out my grant doing things that the grant didn’t require or authorize, and I was fussing with how to save the program. This was why I was so worried, not only about the term evaluation, but also about the situation that Marc was entering and what that would do to us. As I say, he so deftly worked through it, showing impressive skills and ability in managing a program evaluation within an organization in turmoil. And in the process, he gave me some very good feedback and a good evaluation of our program. Even though we weren’t doing it the way the proposal had said we should do it, we were doing some unique things. We were supposed to develop curriculum and we were supposed to use a group of members of the community, like parents of Indian children. But they were not able to really develop curriculum that theoretically would be used in the Seattle public schools.

So, what we did instead was to form a little group of kids, the children of the adults the grant intended us to involve. They became “editors” of a short magazine called the Daybreak Star. ⁴ We put out this little magazine and we sent it out to all the Indian education programs across the country. It was a very novel thing at the time. This was the 1980s, and we were just having a lot of fun doing it too. I think Marc (the evaluator) saw the potential in that and the imagination in our creating such a thing to make our grant work, and he just really recruited me to evaluation. That's how the idea of going into evaluation kind of opened up for me. There was an opportunity for funding at the time, and I was ready to make a change and to move out of the Northwest for at least a while. I was thinking of going to work for my congressman. One of the instructors or professors at the University of Washington at that time was Bill Demerett, a Tlingit, who had been the first director of education for the U.S. Office of Indian Education in the Department of Health, Education, and Welfare. He was on some kind of exchange to be at the school when I was there. He had gone to Harvard and pushed me in that direction. As I said, the funding opportunity was there. My daughters and I moved, spent four years in the Boston area and came back.

Mel: You mentioned Harvard and of course, as you mentioned earlier, Carol Weiss was your advisor there. Any comments about Carol, the way she influenced your thinking, observations about her as a mentor, the way she shaped your evaluation? Anything you’d like to say about her?

Joan: The first thing I’m going to say is that I had two major influences, one being Sara Lawrence-Lightfoot, who's not an evaluator, but is a wonderful scholar at Harvard. I think she got a MacArthur Foundation Genius Award. ⁵ In terms of the approach I took, Sara was just then doing her portraiture work and that had a lot of influence on me. You can think of portraiture as an abbreviated ethnographic approach you can use to try to get to the essence of the school or whatever you’re characterizing.

Carol's influence on me was her wonderful openness. I was a mother of two children, a single mother, living off the small amount of grant money that was getting me through school, and away from a large extended family that I have out here. My whole mother's clan moved out to the Pacific Northwest from Turtle Mountain [Reservation in North Dakota]. So, I have an array of aunts and uncles and cousins that weren’t around me anymore, and when I would walk into Carol's office, obviously very much anxious about the progress I wasn’t making or whatever, the first thing she would say to me when I sat down was, “How are your daughters?” She just brought me into a sense that you’re a human being and you’re not an academic thing. It just brought me down to earth. It would take away my anxiety.

As it turned out, I probably did three dissertations in one (LaFrance, 1990). But first some background. A big national study of tribally controlled schools was commissioned by the House of Representatives, by a committee person, actually by a staffer of a person on the education committee who was relatively hostile to the growing movement of tribally controlled schools. She commissioned this study, and it was assigned to the Department of Education. I happened to be fairly well connected. I knew the employee in the Department of Education who was administering the study. It was put out for a bid, and Abt Associates located in Cambridge, MA won the contract. I got hired by Abt Associates to be on the data collection team for the national study. So, I used the contacts that I was able to make through the Abt work to collect data for my dissertation. I decided to send a survey to tribal schools to find out their priority areas for evaluation. I also had this portraiture notion from the work of Dr. Lawrence-Lightfoot and decided to conduct case studies to complement the survey research.

I visited three different schools to do the kind of qualitative portraiture work Sara was developing. I combined the qualitative and quantitative pieces, which obviously Carol was very open to, and I wrote my dissertation. Sara wrote lovely commentary about it in her written review of the dissertation. Lovely. If my dissertation ever became a book, Sara's comments are what you would want for a blurb on the back cover. Both of us were very interested in how Carol was going to take my dissertation, with the portraiture piece, even though there was a survey in it. Well, Carol loved it too. I passed, I got my dissertation approved. Carol's advice then was to take the survey piece and submit a manuscript based on that portion to AJE (American Journal of Evaluation).

Unfortunately—and this was a good lesson to me later—neither Carol nor Sarah knew how unaware I was of how to do something like that. I didn’t come from people who had a college education. I didn’t have a clue about how to get published. And I was back in Seattle by then and my friends couldn’t help. So, I never submitted my dissertation for publication. At some point I realized that Carol and Sara and others just made assumptions. I think I found that the American Indians were kind of a different breed at the Ed School at Harvard, coming from the backgrounds that we did. I don’t believe any one of the Indian students I met there came from a family of college educated people. We were the first to go to college and obviously the first to get into graduate school in our generation.

Mel: Were there practices or experiences that seemed especially helpful or especially harmful to your development and perhaps also to what you think reflects sound evaluation practice with tribal nations?

Joan: I can remember that one of the very first projects I got was with the early childhood education program up on the Lummi Reservation north of here. I went to visit with them, and I was there to meet with the woman who ran the program as she met with the program parents and the community of advisors. When she introduced me as the evaluator, it was just in her tone that she could not understand why I was there. People who are running programs feel like they’re doing a fairly good job and don’t really care to have an evaluator come in. The most important thing is that later this woman embraced my visits. So, the first thing I would say is to take time to work through that negative or fearful element—even like I had when I was running a program, and yet [the evaluator] Marc Lindenberg became somebody I treasured. We call that relationship building. You have to build that relationship, and there are protocols to follow. Good evaluators are trained to be kind of assertive and ask questions and all of that. When you enter Indian country, that's not what you do. You take time to listen, and you don’t present yourself as a credential. You present yourself first as a person; where you live, you have children, what you’re there for. That's not taught so much, I think. We don’t get information on how to enter the scene when we learn about evaluation. At least I don’t recall that. In fact, I don’t think Carol would have discussed this in our classes. I remember that Karen Kirkhart, who's a good friend, interviewed Carol and in Karen's multicultural lens asked Carol some sort of question about her background. Carol really didn’t think it was a necessary question at all. That maybe would have been a limitation of Carol at that time. But in any case, I don’t think we’re told how to enter in a relationship with the programs that we evaluate so much as we’re taught how to do an evaluation.

Robin: Part of your dissertation highlighted the need to transform education to be more culturally responsive and set the stage for a lot of the work you would do subsequently. I wondered if you could talk a little bit more about some of the things you learned in that dissertation project that sensitized you to thinking about transforming settings to be culturally responsive.

Joan: I’ll set the context a little bit for the House of Representative's directive to do the evaluation of tribally controlled schools. The Bureau of Indian Affairs was moving out of having any kind of Indian schools in the northern tier ⁶ of the country. So, Indians in reservation areas were being sent to the public school in the town bordering the reservation. In response to bad experiences with public schools, tribes were taking advantage of legislation that let them run their own programs and not have the Bureau of Indian Affairs do it. Tribes could contract to do it, to actually develop their own schools. That was the setting, with these tribally controlled schools being relatively new. As I said, there was a kind of hostile reaction from a staff person for a committee member of the House Committee on Education. The staff of senators and representatives can have a lot of power, so she ordered that study. ⁷ The study was designed to be quantitative. Indians going to public schools in a community were to be compared to Indians going to the tribal school in the same community on basis of attendance, academic achievement, and there was supposed to be a cost factor.

The evaluation plan was weighted against tribally controlled schools because Indians who tended to go to those schools were dropping out of the public schools. The tribal schools also don’t have the economies of scale that public schools do. And it was really clear that was what was going on. I think the employee who got the study on his desk at the Department of Ed was a fairly sensitive guy. When Abt Associates got the grant, the first person in charge of it was very quantitative, and he just bought into these measures. But he left Abt and the person who took his place was very qualitatively oriented and immediately saw that this was not the way you wanted to look at tribal schools. Instead, she pushed the quantitative aspect and the qualitative aspect of these schools.

I felt that was my first experience of understanding that you can’t just use numbers to tell a story that has to be told within the context it's set. I mean, you had to know why these schools came to be and who they served to have those numbers make sense, if you were doing that kind of comparison. I think that was my first introduction to how important that context is. Contextualizing. I don’t have a strong awareness of, “This is how I took my evaluation training and did it in Indian country.” I think it was just such a normal process that you take time to build the relationship, you look at context, you make sure you’ve got the story right.

My late friend Richard Nichols and I kind of happened onto a conversation that we were making these kinds of adjustments, changes from what we learned in graduate school to what we learned to do while evaluating in Indian country. And as I said earlier, the stars had just aligned in the sense that, at the same time, AIHEC, with Carrie Billy's leadership, and the National Science Foundation funding came together for the two of us to explore how we would define this approach to Indigenous evaluation. So, we spent several years putting it together.

Val: You made the segue to the Indigenous evaluation framework and the support that you had received from the National Science Foundation, with the work that you conducted with Richard Nichols resulting in your book AIHEC Indigenous Evaluation Framework: Telling our Story in our Place and Time. Let's start with how this came about, and then we’ll dig deeper into the framework.

Joan: As I said, it came about because funding from NSF and a sponsor in AIHEC came together for Richard and me, who were independent evaluators working with tribal colleges. Carrie (Chair and CEO of AIHEC) had invited me to talk about evaluation at one of their college gatherings. College staff were there, and they were negative on evaluation. I tried to point out to them that they depend on grants, and this sets them up as researchers. I said that in a grant basically you say you’re going to do X and as a result Y is going to happen. I explained it as a research issue. You’re saying you’re doing these things, and because you’re doing these things, you expect these outcomes. And that's where evaluators come in. They help you capture that story by tracking what happens because of what you do. By describing program directors as researchers who are busy doing X so Y can happen, I could talk about how they need evaluation to capture the story of how that happened. I could explain that this means evaluation is integral to understanding and growing your program. So, Val, I think tearing down that barrier where evaluation feels imposed was one of the first and most important awarenesses to make evaluation feel comfortable. Carrie liked that talk so much that it built the relationship that I have with her still today. I described the early childhood director earlier who did not appreciate me coming to evaluate her program. But after a while in our relationship, I would go up to interview her and she’d just pause and would say, I love these conversations because they make me reflect on what I’m doing.

I guess the first and maybe most important thing Richard and I struggled with is: Is there another word we can use for evaluation? As it turns out, evaluation is not an easy word to translate into different Indian languages. We had one person who said, “Well, you’ll know tomorrow what you learned today.” That was his way of working out how he would express the idea of evaluation. Evaluation is such an integrally useful thing, but the historical experience with evaluation and with research has been negative, especially I think for marginalized populations who don’t have power. They don’t have the power of the purse strings that are funding evaluations and that sort of thing. The disconnect is especially strong when the evaluator doesn’t understand the cultural setting of the program. An aim of the development of the Indigenous Evaluation Framework was to make evaluation feel comfortable for Indigenous folk involved in an evaluation. Make it feel like it's ours, which it had always been. I mean, we’ve always been evaluators. After all, evaluative thinking is part of existing.

Val: Joan, you help the people involved in your evaluations reframe. That's what it sounds like to me. Thank you for that background. I know we don’t have time to do justice to your entire framework. When I asked before what you saw as a couple of key points in the framework, one you clearly are mentioning is relationship building.

Joan: There are four foundations in the framework. I think what's important is starting with what we call foundations or values; not to start with methodology, but to start with valuing Indigenous knowledge. The framework includes (1) valuing our being people of a place, (2) recognizing our gifts, (3) honoring family and community, and (4) respecting sovereignty. The framework puts in the forefront valuing community and family, valuing a sense of place, valuing our strengths, and valuing sovereignty. That's what we call the foundations.

When I’ve done some presentations saying these are the foundations of our Indigenous evaluation, I ask “What does using these foundations look like in practice?” Then we talk about sovereignty, for example. For evaluations, sovereignty includes data sharing agreements, IRBs, those kinds of elements. In evaluation, honoring our gifts refers to showing respect for each individual and the gifts that they bring, so that measurement is honoring and not reducing. When it comes to community, a key issue is community engagement in evaluation. I love the ideas that came out of Cousins and Whitmore (1998) that we’ve used a lot, about engaging community either through advisory committees or engaging college students and staff roles in data collection, and other ways to engage community.

Indigenous knowledge gets more into understanding context and the place of people you are with. I love the quote from an elder that we found in Vine Deloria’s (1999) work: “You can’t mis-experience anything.” What you experience is what you experience. And then you interpret that experience to get your understanding of it. We use that notion to open the door to the idea that what people are doing in the program is of value, but people can always learn and grow. Take the evaluator's judgment out of it. What we’re doing is storytelling. Evaluation is for learning. It's not our judgment. I hear that idea a lot when I’m at evaluation conferences, that we’ve got to get that sense of the evaluator's judgment out of the scene. Karen Kirkhart and I have had quite a few discussions on this, about whether in evaluation you are judging or learning. I say, “you’re learning.”

By the way, Karen has had some wonderful experiences on projects in Indian country. I know she was notetaking once and somebody asked her to stop notetaking. She really had a sense—and it wasn’t that he didn’t want her to take notes—that he didn’t want her to note take on that particular item he was talking about. Of course, Karen was apologetic. But overall, I say that you can make a lot of mistakes. I mean Indians are very forgiving. So, if you own a mistake, you’re going to be forgiven.

Val: Can I ask if you would advocate any change to the framework at this point? Is there anything you would add or wished you might have said differently?

Joan: One thing we learned when we did a project—and the writing of this is still in progress—stemmed from applying the framework in three different college settings. At Northwest Indian College, the framework was pretty much applied the way it was set up to be applied. In another setting, the framework didn’t work at all. They simply didn’t follow through with it at that college. The third college took the framework and adapted it. The framework was developed for evaluation, of course, but they used it in developing a student assessment tool. I find it very interesting to look across the three settings. The framework seemed to give permission to apply it in ways other than what we had intended. One college adapted it to be responsive to indigenous ways of knowing when developing an assessment tool. The framework seemed useful even when focusing on something other than evaluation.

So, we had three completely different experiences with the use of the framework. In one way, I think that the framework is kind of a stimulus for people to look at and think about. And, I’m going to say “apply the framework” using quotation marks, because whoever is using it is going to do it in unique ways. So, I think the framework is kind of a stimulus to think, well, we can do evaluation with an Indigenous process. We can make the evaluation, or whatever you’re working on, responsive in the sense of thinking about it in an Indigenous way.

Val: You’re answering another question I was about to ask. The framework would seem to suggest that some important commonalities exist across the communities the framework is meant to cover; but, at the same time—consistent with what you’ve just been describing—you’ve advised caution about expecting the same exact thing to work across all Indian nations. So as an evaluator, how do we thread that needle? Is that the idea of putting the framework out there as a stimulus and just letting the evaluator run with that, with the community or…

Joan: Yes, I would say that's somewhat the way it is. I think you need to frame the key elements of the framework a little bit. For example, we identified the core values, and we identified these as core values across Indian country—that community and family and sovereignty and Indigenous ways of knowing should be valued. But you should have a conversation about those values and the values that the community would like to articulate, and that should influence the way you evaluate. It's not easy to have this conversation, because communities aren’t used to people encouraging it. The foundations are just so implicit that to articulate them and make them explicit is a process.

I think that's where we run up against a real dilemma, because that conversation about values takes time and evaluations have bounds from their funding. But, it's important to take the time for this kind of conversation and for other front-end work. That's really a key part of what the Indigenous evaluation framework is trying to stimulate—doing the important front-end work. The framework doesn’t change a lot about how you collect data and analyze data, which are the fundamental elements of evaluation. The framework really focuses on that front-end work, where you make the evaluation fit the context, and where you talk about how the evaluation reflects key values as you go about doing things.

I should mention an expansion of the original framework that was stimulated by Karen [Kirkhart]. As you know, Karen has done important work on validity (e.g., Kirkhart, 1995). She mentioned that we never said anything about validity in the framework. So Richard and I talked about that. We also had a conversation with our elders on what validity means. I talked to a couple of Anishinaabe speakers and Richard talked with his Tewa speakers. We came away with them answering in different ways that mean the same thing. What you and I call validity, to these people was about doing things in the evaluation in the right way. An evaluation is valid if you do it in the right way. So again, front-end work is needed to understand, “What does the right way mean here?” The right way in Indian country probably includes gifting protocols—that's pretty common (for gifts to be given)—and gifting sets a tone. The right entry protocol sets you up to do other things in the evaluation in the right way. Do you start meetings with a prayer? Find out if that's the case and, if so, you make sure that happens, that somebody is asked to start any meeting with a prayer. As discussed in a chapter (Nelson-Barber et al., 2005), as well as in a paper with Richard and Karen (LaFrance et al., 2012), doing it right also involves evaluating in a way that's consistent with the framework's foundations, like valuing place and community.

Val: To me, this leads to the scaffolding concept, from the book with Richard Nichols, which is the design of an actual evaluation. You explained design in a broader way that people can understand, not just the X's and O's that we see in one of our methods books, but totally bringing the reader in to understand what happens if you do something in an evaluation one way versus another.

Joan: I think that the realization has been growing among evaluators that we’re storytellers. And if you look at evaluation as a story, just as if you look at our legends as stories, they both provide lessons. You learn from them. Using that idea somewhat metaphorically, our evaluation story is a story that we learn from. The trickster [a common and important character in most Indigenous oral traditions] teaches lessons. Our evaluation work will teach us lessons. So, I think that's the main thing you’re trying to do in building the scaffolding. You want to tell a story. Now there are certain structures that you use to get at that story, but essentially you’re just telling a story to learn from.

Val: Can you talk a bit about the role that you see for non-Indigenous evaluators practicing evaluation in Indigenous contexts, for those of us who might actually be tasked to do an external evaluation?

Joan: I think that there's a lot of room for that role because we don’t have enough Indigenous evaluators to meet the requirements for external evaluation in Indian country. I’m heartened by the fact that when I do training, I see a lot of non-Indians in a session at the evaluation conference or at CREA. ⁸ There's an interest in learning. Again, I would advise any tribe or Indian organization or tribal college that's going to have an external evaluator to invite them to sit down and talk with tribal people before they start. I do think an advisory council would be very good, so the evaluator has people in the community or in the organization that can help them navigate the context. And the non-Indian evaluator should be open to that kind of learning. You really want to get somebody who wants to succeed at being good at evaluating in Indian country.

Robin: I’d like to hear you talk a little bit more about the decision to found your own consulting company after you left Harvard, the things that influenced you to make that decision, and what it was like to get your company off the ground.

Joan: As I mentioned, I had assumed that I would follow the model of working in a university and doing program evaluation on the side. The truth of the matter was if you want to work in a university, you’ll go where the job is, and I’m pretty deeply rooted. And there was no real niche for me in a university setting right here in Seattle. Or at least I didn’t take the time to try to figure out how to create one. Again, that relates to the fact that I wasn’t even sure how to publish. I had that weakness. But, I realized that when Marc Lindenberg came and evaluated my program, it wasn’t because he was a university professor. He was doing it as an independent contractor. We didn’t pay the university, we paid him.

So, I decided I would incorporate. I learned how to do that as a sole proprietorship. I needed a name and went to an elder. I was home at Turtle Mountain, and I went to an elder who was an Ojibwe speaker. She told me to use Mekinak, which is Ojibwe for Turtle. So that was it. I incorporated under that name. I negotiated the ability to work part-time with the City of Seattle. I was lucky—a program at Makah Reservation needed an evaluator. And the person who had been running the program knew me personally. In fact, her mother had been the director of the Indian Center that I went to years ago to join the American Indian Women's Service League. So, she hired me.

In the early 2000s, I went to one of the evaluation conferences and made a presentation about evaluating in Indian Country. The Principal Investigator for a rather large program that included five universities was in the audience. The consortium was a Center for Learning and Teaching funded by the National Science Foundation. The PI, who was from one of the schools in the consortium, Montana State, was there. They were very unhappy with their evaluator. Because they were a university program, they had fellows who were students that they wanted involved in the evaluation, and the evaluator they had just didn’t seem to work with this model at all. So, the person at Montana State contracted with me. I got a nice big contract, which was great since I had cut my City of Seattle earnings in half. I worked with the student fellows, evaluating that five-university consortium. I’m proud to say that four of those five fellows have evaluation as one of the hats they wear in the work that they do.

That project made it financially feasible for me to work as an evaluator. And then one thing led to another. One of the five schools was the University of Portland, and a faculty person there hired me to do some of her evaluations. You just create this niche market. I’ve never had to market myself. I don’t even have a webpage. Now I have to think about how many times I will say “yes” when asked to evaluate a project. Most of the time what I get now is through word of mouth that's occurred over the years. People contact me when they’re writing a proposal. I will write their evaluation section and, if they get funded, then I will become their evaluator. So, that's how I make my business work. I’m at the point now where I’m close to maybe retiring. I may be taking on another five-year project, and that might be it. But I never marketed, unlike Gail up in Canada with her wonderful book on how you run your own business (Barrington, 2011). I just never had to do that. I’ve had such a niche market that I’m not a very good person to ask, “How do you start a business?”

Robin: What would you say are the highs and lows of running your own business?

Joan: Especially as a sole proprietorship, I don’t have colleagues. It gets lonely. That's a low. Also, the fact that it's not salaried. Your cashflow is not regular, which means you’ve got to be very careful. The highs are that you’re doing what you want to do. I mean, I love doing this kind of work. For some people having a lot of travel can be a low, but I would say that for me a high in the past was visiting a lot of communities. I’m doing less of that now, though. My recent projects have mostly been closer to home. I can drive up the road. Now I have only a couple of distant projects. Fortunately, or unfortunately, they’re out in Micronesia. So, it's a bit of a travel, when you do it.

Robin: As a sole proprietorship, do you have opportunities to engage younger evaluators, the next generation, in some of the kinds of projects that you take on?

Joan: I’ve thought about that. And I try to get a student if I’m doing something with a tribal college. I’ve thought about talking with some people at the University of Washington to see if they can assign a student on a project. It's just that I haven’t totally followed through on my plan. The problem with the tribal colleges is that they’re 2-year institutions. The level of analysis in most evaluations fits better, I think, with a 4-year college or somewhere that has graduate students.

Robin: You mentioned your first conference at AEA. At that point it might have been ENET or ERS [the Evaluation Network or the Evaluation Research Societies, the two predecessor organizations to the American Evaluation Association]. But you’ve been involved with the association for a very long time. I wonder if you could reflect a little bit on your early experiences of stepping into professional associations.

Joan: AEA was a wonderful experience because, as I said, it is lonely in a sole proprietorship. It was wonderful to go and meet with people at a place where evaluation was front and center and everybody knew what you were talking about. In fact, with Elmima's help, we were able to recruit Indian evaluators to join AEA. Elmima [Johnson of NSF] and I wanted to get more Indian people to go to the evaluation conference. Quite a few Indian academics went to AERA [American Educational Research Association]. Elmima had access to some money through an organization, maybe the American Institutes for Research. They had a grant from the National Science Foundation, and Elmima made the connection with them to host a breakfast at AERA aimed at the Indigenous Peoples of Americas Special Interest Group of AERA, inviting them to come to a breakfast if they were interested in evaluation. This was at an AERA meeting in Chicago in 2003, I believe.

That breakfast is where we got a critical mass of Indian evaluators, at this AERA meeting. We told them about AEA and encouraged them to join. Then, when we met at the [2003] AEA meeting in Reno, again with Elmima's help, we had a luncheon. Of all places, it was in the Captain Cook room. Again, we invited people who were Indian or worked in Indian Country to come to that luncheon. We had a fairly good turnout even though we used word-of-mouth advertising to get people to go to that conference. Reno was a good location to get some people to attend from the western part of the United States who normally didn’t go to AEA.

We asked people at the luncheon what they would like, and they said they would like their own TIG. This upset Elmima a little bit, because she really wanted them all to just join the Multiethnic Issues in Evaluation TIG. Instead, we formed our own Indigenous Peoples in Evaluation TIG in 2006. That luncheon meeting in Reno started the momentum to get our own TIG within AEA. That just slowly built the presence of Indigenous people at AEA. It started with that breakfast in a Chicago AERA conference with a special invitation to AERA people to come and learn about how to get involved in AEA as Indigenous people.

Robin: I’d love you to start helping us think a little bit about the future as you think about the work that still lies ahead. How could educational pathways and professional development for evaluators be improved, especially to better support evaluations with Indigenous people and first or tribal nations?

Joan: Well, I have a wonderful opportunity every two years through the University of Wisconsin at Green Bay to teach a specialized course in Indigenous evaluation. The students might not all be Indigenous, but most of them are in a doctoral program that's aimed at preparing educators to work in Indian country. One of the most rewarding experiences I have is really being able to take the Indigenous framework and work it as a textbook. As to what the future is, I would like to see specialized evaluation courses on working in Indigenous communities or in other people of color communities, where class members can really talk about and imagine how to contextualize their evaluations in cultural ways. I would like to see more of that—not just that such communities are mentioned or that you read a few articles, but that you really have enough course time to get serious about this kind of work. There may be more opportunity for courses like this in graduate evaluation programs. I don’t know where programs like up in Kalamazoo [Western Michigan University] or Claremont [Claremont Graduate University] are in terms of actual courses that focus on cultural communities, not just multicultural, but cultural communities and what you do in them. But I’d like to probably see more of that based on my experience with that course at UW-Green Bay. A course like that really allows you to talk about things in a deeper way.

Also, like Nicky, I’ve done professional development sessions, but I’m not seeing new sessions or instructors come up recently. So, I’m a little concerned about where the leadership of our Indigenous evaluation community is going, whether things will evolve with them doing training or being trainers of trainers. I’m not seeing that happen yet. And I think maybe this is worthy of a conversation for us to have within our [IPE] TIG.

Robin: What needs to happen for professional associations to have greater capacity and competence to support professional development and evaluations with scholars who are Indigenous or for people who will practice in Indigenous settings?

Joan: As I said, I’m worried that we’re not seeing leadership from within the IPE TIG emerge that wants to do training on Indigenous evaluation. I think it would be worthwhile to look into this kind of training as an organization, maybe with current [IPE] TIG members. And I wish that the universities that are training evaluators would do more on culturally specific groups and evaluation.

Robin: What would you like to say to young Indigenous evaluators at the beginning, or middle of their careers? What advice would you offer them as early career and mid-career professionals?

Joan: First of all, I would really want to honor early and mid-career professionals for helping us move forward because they’re bringing their youth into it. And I would encourage them to think about what it is they’re learning in a way that they can share, so that they do workshops like Nicky [Bowman] and I have done. Younger Indigenous evaluators should use their organizational membership opportunities to pass on what they’re doing and learning. I really think that Indigenous evaluation practice is evolving and that we’re constantly going to get new insights from people's experiences. In Hawaii they call it “talking story.” You just “talk story.” And I’d like to hear these younger evaluators’ stories: Where they’re working—are they working in their own communities? What are the advantages of being able to do that? What are some of the disadvantages? As I said, our practice is evolving. We need to know more about what we’re learning, and we learn from telling our stories. So, building as many network opportunities as we can is what I hope we can do moving forward. When we have our [AEA] conference, we have our TIG meeting, but we need more opportunities for whatever it is that draws people to a TIG to come together and talk, other than just the TIG meeting. You have to do business at those meetings, you have to introduce people, and so on. I know it's hard because the conference program is rich, but we need more time to talk with each other.

Mel: Joan, this has been wonderful. At this point I’d like to loop back to some things you’ve talked about and ask a couple of follow-up questions. You mentioned Marc Lindenberg, the evaluator who turned around your view of how valuable evaluation could be. At another point you mentioned the time and the investment of oneself needed in the entry, in the front-end relationship building an evaluator should do. Given Marc was probably on a contract where this wasn’t built in—at least it wouldn’t have been standard at the time he was doing that evaluation—I wondered what it was that he did that made up for that. Might there be some broader lessons for evaluators when they’re working without the kind of funding or time frame that facilitates that front-end relationship building?

Joan: I think Marc in fact did do some relationship building. Even though he came on as an evaluator, I don’t remember him asking me anything about evaluation. I just remember him asking me about what I do, and I was able to just talk to him. So, I think that quality of being a listener is what he brought in. You build relationships by hearing what people are saying to you and providing space for that. And I think that is how he did it. As I said, I was very worried because we were doing something that was rather novel, but not what our grant proposed. Marc didn’t come in with the preconceived notion of “these are your objectives, these are your outcomes.” He just came in as a listener and let you describe what you are doing and what's behind that. That's advice I would give, because when people believe they are doing good work, they don’t understand why somebody has to evaluate it. Marc was very mild mannered and just a good listener who built good relationships.

Mel: You’ve mentioned your colleague, collaborator, and friend Richard Nichols a couple of times. Anything else you’d like to say about him, about the kind of work that the two of you did together, or the ways that he influenced you beyond what you’ve said already?

Joan: I remember we had the opportunity with our planning grant to have these focus groups and we did put tobacco at everybody's place as a thank you for the wisdom they were going to give us ⁹ . I would not have thought about smudging the room to prepare for the work that was going to be done. I learned that from Richard. He took time to prepare for smudging before people were in the room. There are certain risks with smudging—in particular, setting off an alarm because it is smoky. To minimize that risk, it's better to use sweet grass than sage. But the key lesson I learned from Richard was about just doing things in the right way for the community you’re working with. Richard was definitely reservation raised while I wasn’t, so he would kind of put the brakes on and help me do things in the right way, like gifting tobacco and smudging.

Val: I wondered, with COVID over the last couple of years, whether that impacted your work in any way. Have you done more online now than before? You were saying how visiting and onsite observation is a very big part of the framework, so I’m just curious.

Joan: I don’t think Covid had a big effect on me mainly because I just didn’t have that many projects that I was working on, and none that required travel. Covid would have made more of a difference earlier in my career, when my projects required time on site. But I am winding down. I’m facing my 80th birthday in a few weeks, so I guess it's time to wind down a little bit.

Val: Totally amazing, Joan. Congratulations.

Joan: In fact, I was caring for an uncle who was 102. He was living with his sister. He never had any children, and I became kind of his child and caretaker. I moved him from Kansas City when he was in his late nineties up to his sister in Olympia. So, during most of Covid I was driving to and from Olympia very often. My uncle did pass during Covid time. He was born during the 1918 epidemic, and he died 102 years later in another epidemic. I kind of didn’t notice Covid because I was busy until after he passed. And then I realized, oh, I’m kind of stuck at home. But no, it didn’t have a big impact.

Mel: Joan, there was a point where you said that “evaluation is a story that we learn from.” I love that quote and feel it also applies quite well to this discussion. You’ve given us multiple stories and I think those of us who were able to participate in this session as well as those who read the interview in AJE will learn from your stories. So, thank you so much!

Joan: Miigwech to you for including me.