Where User Perceptions of Charities and Independent Nonprofit Ratings Meet: Inside the Black Box of Nonprofit Performance

Abstract

Scholars have pointed to the multidimensional nature of nonprofit performance, but some contend that user perspectives are often overlooked. To the extent that users rate their experiences and interactions with nonprofits on public online platforms, such as Google, this content presents an opportunity for tapping into user perspectives of charities and considering whether these perceptions align with those of independent nonprofit rating agencies. Do online user reviews of nonprofits converge with the performance assessments of independent charity raters? We answer this question through a mixed-methods study involving a quantitative analysis of ratings data from Charity Navigator and Google user ratings for 228 nonprofits, followed by a qualitative content analysis of online reviews conducted on a subset of these cases. We find a small but positive association between user and independent ratings, although service users tend to base their ratings on different factors than ratings agencies. Our findings carry a number of important implications for those interested in nonprofit performance. For nonprofit leaders, online user ratings and reviews offer a window into the subjective impressions of service users and can serve as a complementary source of performance feedback to more traditional types of data. For funders and donors, our findings discourage exclusive reliance on independent ratings in assessing nonprofit performance and point to the need to account for user perspectives in the calculus of performance.

Keywords

nonprofit performance ratings user ratings Charity Navigator Google ratings

Get full access to this article

View all access options for this article.

References

Bardach

N. S.

Asteria-Peñaloza

Boscardin

W. J.

Dudley

R. A.

(2013). The relationship between commercial website ratings and traditional hospital performance measures in the USA. BMJ Quality & Safety, 22(3), 194–202. https://doi.org/10.1136/bmjqs-2012-001360

Benjamin

L. M.

(2013). The potential of outcome measurement for strengthening nonprofits’ accountability to beneficiaries. Nonprofit and Voluntary Sector Quarterly, 42(6), 1224–1244. https://doi-org.proxy.cc.uic.edu/10.1177/0899764012454684

Benjamin

L. M.

Campbell

D. C.

(2015). Nonprofit performance: Accounting for the agency of clients. Nonprofit and Voluntary Sector Quarterly, 44(5), 988–1006. https://doi-org.proxy.cc.uic.edu/10.1177/0899764014551987

Campbell

D. A.

Marty

K. T.

(2022). Participants not observers: Involving beneficiaries as co-assessors in performance analysis. Journal of Health and Human Services Administration, 44(4), 260–273. https://doi.org/10.37808/jhhsa.44.4.1

Campbell

D. A.

Marty

K. T.

Bronstein

(2012). In the eyes of the beholders: Feedback motivations and practices among nonprofit providers and their funders. Public Performance and Management Review, 36(1), 7–30.

Carman

J. G.

Fredericks

K. A.

(2008). Nonprofits and evaluation: Empirical evidence from the field. Nonprofits and Evaluation: New Directions for Evaluation, 119, 51–71.

Charity Navigator. (2022a). About us. https://www.charitynavigator.org/index.cfm?bay=content.view&cpid=8658

Charity Navigator. (2022b). Methodology. https://www.charitynavigator.org/index.cfm?bay=content.view&cpid=5593

CharityWatch. (2012). About us. https://www.charitywatch.org/about-charitywatch?slug=mission-goals

10.

Cnaan

R. A.

Jones

Dickin

Salomon

(2011). Nonprofit watchdogs: Do they serve the average donor? Nonprofit Management and Leadership, 21(4), 381–397. https://doi.org/10.1002/nml.20032

11.

Creswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed methods research (2nd ed.). Sage.

12.

Daskivich

T. J.

Houman

Fuller

Black

J. T.

Kim

H. L.

Spiegel

(2018). Online physician ratings fail to predict actual performance on measures of quality, value, and peer review. Journal of the American Medical Informatics Association: JAMIA, 25(4), 401–407. https://doi.org/10.1093/jamia/ocx083

13.

Davison

Price

(2009). How do we rate? An evaluation of online student evaluations. Assessment & Evaluation in Higher Education, 34(1), 51–65. https://doi.org/10.1080/02602930801895695

14.

Filieri

(2016). What makes an online consumer review trustworthy? Annals of Tourism Research, 58, 46–64. https://doi.org/10.1016/j.annals.2015.12.019

15.

Glaser

B. G.

Strauss

A. L.

(1967). The discovery of grounded theory–Strategies for qualitative research. Aldine.

16.

Gordon

T. P.

Knock

C. L.

Neely

D. G.

(2009). The role of rating agencies in the market for charitable contributions: An empirical test. Journal of Accounting and Public Policy, 28(6), 469–484. https://doi.org/10.1016/j.jaccpubpol.2009.08.001

17.

Huang

Meschke

Guthrie

J. P.

(2015). Family firms, employee satisfaction, and corporate performance. Journal of Corporate Finance, 34, 108–127. https://doi.org/10.1016/j.jcorpfin.2015.08.002

18.

Kahlke

R. M.

(2014). Generic qualitative approaches: Pitfalls and benefits of methodological mixology. International Journal of Qualitative Methods, 13(1), 37–52. https://doi-org.proxy.cc.uic.edu/10.1177/160940691401300119

19.

Landers

Brusso

Auer

(2019). Crowdsourcing job satisfaction data: Examining the construct validity of glassdoor.com ratings. Personnel Assessment and Decisions, 5(3), 6. https://doi.org/10.25035/pad.2019.03.006

20.

LeRoux

(2009). Managing stakeholder demands: Balancing responsiveness to clients and funding agents in nonprofit social service organizations. Administration & Society, 41(2), 158–184.

21.

LeRoux

Medina

(2022). Bending the arc of nonprofit leadership toward justice: Impacts of racial representation and organizational publicness on diversifying executive leadership. Public Administration Review, 83, 103–116. https://doi-org.proxy.cc.uic.edu/10.1111/puar.13534

22.

LeRoux

Wright

, (2010). Does performance measurement improve strategic decision-making? Findings from a national survey of nonprofit social service agencies. Nonprofit and Voluntary Sector Quarterly, 39(4), 571–587. https://doi.org/10.1177/089976400935

23.

LeRoux

Wright

N. S.

(2021). Performance and public value in the hollow state: Assessing government-nonprofit partnerships. E-Elgar Press.

24.

Lincoln

Y. S.

Guba

E. G.

(1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Evaluation, 30, 73–84. https://doi-org.proxy.cc.uic.edu/10.1002/ev.1427

25.

Ling

Gordon Neely

(2013). Charitable ratings and financial reporting quality: Evidence from the human service sector. Journal of Public Budgeting, Accounting & Financial Management, 25(1), 69–90. https://doi-org.proxy.cc.uic.edu/10.1108/JPBAFM-25-01-2013-B004

26.

Lowell

Trelstad

Meehan

(2005). The ratings game. Stanford Social Innovation Review, 3(2), 38–45.

27.

MacIndoe

Barman

(2013). How organizational stakeholders shape performance measurement in nonprofits: Exploring a multidimensional measure. Nonprofit and Voluntary Sector Quarterly, 42(4), 716-738. https://doi-org.proxy.cc.uic.edu/10.1177/0899764012444351

28.

MacQueen

K. M.

McLellan-Lemal

Bartholow

Milstein

(2008). Team-based codebook development: Structure, process, and agreement. In Guest

MacQueen

K. M.

(Eds.), Handbook for team-based qualitative research (pp. 119–135). AltaMira Press.

29.

Masaoka

(2015, April 14). An open letter to Charity Navigator. CalNonprofits. https://opnlttr.com/letter/open-letter-charity-navigator

30.

Morley

Vinson

Hatry

(2001). Outcome measurement in nonprofit organizations: Current practices and recommendations. Urban Institute.

31.

Nowell

Albrecht

(2019). A reviewer’s guide to qualitative rigor. Journal of Public Administration Research and Theory, 29(2), 348–363. https://doi.org/10.1093/jopart/muy052

32.

Pandey

S. K.

Coursey

D. H.

Moynihan

D. P.

(2007). Organizational effectiveness and bureaucratic tape: A multimethod study. Public Performance and Management Review, 30(3), 398–425.

33.

Park

Kim

Almanza

(2016). Yelp versus inspection reports: Is quality correlated with sanitation in retail food facilities? Journal of Environmental Health, 78(10), 8–13.

34.

Petrovsky

Xin

(2023). Job satisfaction and citizen satisfaction with street-level bureaucrats: Is there a satisfaction mirror? Journal of Public Administration Research and Theory, 33(2), 279–295. https://doi.org/10.1093/jopart/muac022

35.

PowerReviews. (2021). Survey: The ever-growing power of reviews. https://www.powerreviews.com/insights/power-of-reviews-survey-2021/

36.

Ranard

B. L.

Werner

R. M.

Antanavicius

Schwartz

H. A.

Smith

R. J.

Meisel

Z. F.

Merchant

R. M.

(2016). Yelp reviews of hospital care can supplement and inform traditional surveys of the patient experience of care. Health Affairs, 35(4), 697–705. https://doi.org/10.1377/hlthaff.2015.1030

37.

ReviewTrackers. (2022, January 9). Online reviews statistics and trends: A 2022 report by ReviewTrackers. https://www.reviewtrackers.com/reports/online-reviews-survey/

38.

Saldaña

(2013). The coding manual for qualitative researchers (2nd ed.). Sage.

39.

Sowa

J. E.

Selden

S. C.

Sandfort

J. R.

(2004). No longer unmeasurable? A multidimensional integrated model of nonprofit organizational effectiveness. Nonprofit and Voluntary Sector Quarterly, 33(4), 711–728. https://doi.org/10.1177/0899764004269146

40.

Stork

Woodilla

(2008). Nonprofit organizations: An introduction to charity rating sources and cautions in their use. International Journal of Applied Management and Technology, 6(4), 1.

41.

Szper

(2013). Playing to the test: Organizational responses to third party ratings. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 24, 935–952. https://doi.org/10.1007/s11266-012-9290-0

42.

Thomson

D. E.

(2010). Exploring the role of funders’ performance reporting mandates in nonprofit performance measurement. Nonprofit and Voluntary Sector Quarterly, 39(4), 611–629.

43.

Woodroof

P. J.

Howie

K. M.

Peasley

M. C.

(2021). Nonprofit quality: What is it and why should nonprofits care? Journal of Philanthropy and Marketing, 26(1), Article e1682.