Restricted accessResearch articleFirst published online 2025
Beyond technical adequacy: A holistic framework for evaluating artificial intelligence systems through the scaling responsible artificial intelligence mentorship approach
Artificial intelligence evaluation practices face a fundamental challenge: traditional technical measures cannot adequately capture the complex socio-organizational impacts of artificial intelligence systems in real-world contexts. This research bridges the critical gap between technical evaluation and holistic evaluation by exploring how insights from evaluation theory can inform artificial intelligence system assessment. Using a mixed-methods case approach inspired by developmental evaluation, realistic evaluation, and empowerment evaluation principles, we analyzed 12 artificial intelligence implementations in organizations within the scaling responsible artificial intelligence mentoring program. Data collection consisted of semi-structured participant interviews, participant observation, and data extraction from various institutional databases, all guided by specific evaluation theoretical frameworks. Our analysis reveals five generative principles for a holistic evaluation of artificial intelligence: epistemic pluralism, democratic authority, contextual responsiveness, temporal sensitivity, and reflexive critique. These principles transcend procedural criteria to encompass fundamental epistemological changes in technology governance. This framework challenges technocentric evaluation paradigms and provides theoretical and practical guidance to organizations implementing artificial intelligence systems in diverse cultural contexts.
AithalPSAithalS (2024) Redefining experimental, empirical, and exploratory research in AI era. Poornaprajna International Journal of Emerging Technologies (PIJET)1(1): 90–136.
2.
AugensteinIBaldwinTChaM, et al. (2024) Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence6(8): 852–63.
3.
BarocasSHardtMNarayananA (2023) Fairness and Machine Learning: Limitations and Opportunities. Cambridge, MA: MIT Press.
4.
BhattDLMehtaC (2016) Adaptive designs for clinical trials. New England Journal of Medicine375(1): 65–74.
5.
BlackmanR (2022) Ethical Machines: Your Concise Guide to Totally Unbiased, Transparent, and Respectful AI. Brighton, MA: Harvard Business Press.
6.
CopestakeJ (2025) Anticipatory evaluation. Evaluation Journal of Australasia25(1): 9–22.
7.
CousinsJBWhitmoreE (1998) Framing participatory evaluation. New Directions for Evaluation1998(80): 5–23.
8.
CramF (2009) Maintaining Indigenous voices. In: MertensDMGinsbergPE (eds) The Handbook of Social Research Ethics. London: Sage, 308–322.
9.
DamschroderLJAronDCKeithRE, et al. (2009) Fostering implementation of health services research findings into practice. Implementation Science4(1): 50.
10.
DelgadoFYangSMadaioM, et al. (2021) Stakeholder participation in AI: Beyond “add diverse stakeholders and stir.” arXiv preprint arXiv: 2111.01122.
11.
FernandesSSheejaMSParivaraS (2024) Potential of AI for a sustainable, inclusive, and ethically responsible future. In: VedaADharSDharU (eds) Fostering Multidisciplinary Research for Sustainability. Allied Publishers, 196.
12.
FettermanDM (2001) Foundations of Empowerment Evaluation. London: Sage.
13.
FranaPL (2024) Assessing smart nation Singapore as an international model for AI responsibility. International Journal on Responsibility7(1): 2.
14.
GajendranTBrewerG (2012) Cultural consciousness and the effective implementation of information and communication technology. Construction Innovation12(2): 179–97.
15.
GreeneJC (2007) Mixed Methods in Social Inquiry. Hoboken, NJ: John Wiley & Sons.
16.
GreeneJCAbmaTA (2020) Responsive evaluation: New directions for evaluation. New Directions for Evaluation2020(167): 33–46.
17.
HanSKellyENikouS, et al. (2020) Reflections on artificial intelligence alignment with human values: A phenomenological perspective. In: Proceedings of the 28th European Conference on Information Systems (ECIS), An online AIS Conference, 15–17 June.
18.
HegazySElsnerCBoschJ, et al. (2025) Overcoming experimentation challenges in software ecosystems of large product and service organizations: A participatory action research study. Journal of Systems and Software230: 112550.
19.
HenryDTolanPGorman-SmithD, et al. (2017) Alternatives to randomized control trial designs for community-based prevention evaluation. Prevention Science: The Official Journal of the Society for Prevention Research18(6): 671–80.
20.
Hernández-OralloJ (2017) Evaluation in artificial intelligence: From task-oriented to ability-oriented measurement. Artificial Intelligence Review48: 397–447.
21.
HillM (2023) Hallucinating machines: Exploring the ethical implications of generative language models. PhD Thesis, Te Herenga Waka-Victoria University of Wellington.
22.
HinderksAMayoFJDThomaschewskiJ, et al. (2022) Approaches to manage the user experience process in agile software development: A systematic literature review. Information and Software Technology150: 106957.
23.
HopsonRK (2003) Overview of multicultural and culturally competent program evaluation: Issues, challenges and opportunities. In: DonaldsonSIScrivenM (eds) Evaluating Social Programs and Problems: Visions for the New Millennium. Mahwah, NJ: Lawrence Erlbaum Associates, 149–66.
24.
HouseERHoweKR (1999) Values in Evaluation and Social Research. London: Sage.
25.
JacksonDAllenC (2024) Enablers, barriers and strategies for adopting new technology in accounting. International Journal of Accounting Information Systems52: 100666.
26.
JayaprakashPPatiRK (2023) Sustainable information technology and information system: A systematic literature review, taxonomy, and agenda for future research. Asia Pacific Journal of Information Systems33(2): 468–512.
27.
JoshiN (2024) Emerging challenges in privacy protection with advancements in artificial intelligence. International Journal of Law and Policy2(4): 55–77.
28.
KatellMYoungMDaileyD, et al. (2020) Toward situated interventions for algorithmic equity: Lessons from the field. In: Proceedings of the 2020 conference on fairness, accountability, and transparency, Barcelona, 27–30 January, 45–55. New York: Association for Computing Machinery.
29.
KautzK (2011) Investigating the design process: Participatory design in agile software development. Information Technology & People24(3): 217–35.
30.
KingN (2012) Doing template analysis. In: SymonGCassellC (eds) Qualitative Organizational Research: Core Methods and Current Challenges. London: Sage, 426–50.
31.
McIntoshTRLiuTSusnjakT, et al. (2023) A culturally sensitive test to evaluate nuanced GPT hallucination. IEEE Transactions on Artificial Intelligence5(6): 2739–51.
32.
MadaioMAStarkLWortman VaughanJ, et al. (2020) Co-designing checklists to understand organizational challenges and opportunities around fairness in AI. In: Proceedings of the 2020 CHI conference on human factors in computing systems, Honolulu HI, 25–30 April, 1–14. New York: Association for Computing Machinery.
33.
MarkMMHenryGTJulnesG (2000) Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs. Hoboken, NJ: Jossey-Bass.
34.
MickunasAPilottaJ (2023) A Critical Understanding of Artificial Intelligence: A Phenomenological Foundation. Singapore: Bentham Science Publishers.
35.
MishraSRaoAKrishnanR, et al. (2024) Reliability, resilience and human factors engineering for trustworthy AI systems. arXiv preprint arXiv: 2411.08981.
36.
Montrosse-MoorheadBSchröterDBechoLW (2024) The garden of evaluation approaches. American Journal of Evaluation45(2): 166–85.
37.
MoserCDen HondFLindebaumD (2022) Morality in the age of artificially intelligent algorithms. Academy of Management Learning & Education21(1): 139–55.
38.
NovelliCTaddeoMFloridiL (2024) Accountability in artificial intelligence: What it is and how it works. AI & Society39(4): 1871–82.
39.
PallmannPBeddingAWChoodari-OskooeiB, et al. (2018) Adaptive designs in clinical trials: Why use them, and how to run and report them. BMC Med16: 29.
40.
PaolantiMTiribelliSGiovanolaB, et al. (2024) Ethical framework to assess and quantify the trustworthiness of artificial intelligence techniques: Application case in remote sensing. Remote Sensing16(23): 4529.
41.
PattonMQ (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guilford Press.
42.
PattonMQ (2011) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guilford Press.
QuinFWeynsDGalsterM, et al. (2024) A/b testing: A systematic literature review. Journal of Systems and Software211: 112011.
45.
RajiIDSmartAWhiteRN, et al. (2020) Algorithmic auditing: A practical approach to fairness assessment. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, New York, 7–9 February, 319–26. New York: Association for Computing Machinery.
46.
RakovaBYangJCramerH, et al. (2021) Where responsible AI meets reality: Practitioner perspectives on enablers for shifting organizational practices. Proceedings of the ACM on Human-computer Interaction5(CSCW): 1–23.
47.
RayPP (2023) Benchmarking, ethical alignment, and evaluation framework for conversational AI: Advancing responsible development of ChatGPT. BenchCouncil Transactions on Benchmarks, Standards and Evaluations3(3): 100136.
48.
SelbstADBoydDFriedlerSA, et al. (2019) Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta GA, 29–31 January, 59–68. New York: Association for Computing Machinery.
49.
ShenTLiJBouadjenekMR, et al. (2023) Towards understanding and mitigating unintended biases in language model-driven conversational recommendation. Information Processing & Management60(1): 103139.
50.
StephensonEMikolajczakGRyanM, et al. (2025) A framework for evaluating women’s leadership programmes. Evaluation31(1): 111–41.
51.
TianRGYingWZhongN, et al. (2024) Technological innovation capacity of MNEs in China: Studying the cross-cultural effects. Asia Pacific Business Review. Epub ahead of print 11 June. DOI: 10.1080/13602381.2024.2357657.
52.
WegerKYeazitzisT (2023) Conceptualizing a socio-technical model for evaluating AI-driven technology. Proceedings of the Human Factors and Ergonomics Society Annual Meeting67: 1639–44.
53.
WestSGDuanNPequegnatW, et al. (2008) Alternatives to randomized control trial. American Journal of Public Health98(8): 1359–66.
54.
WinnerL (1980) Do artifacts have politics?Daedalus109(1): 121–36.
55.
WrightJ (2024) The development of AI ethics in Japan: Ethics-washing society 5.0?East Asian Science, Technology and Society: An International Journal18(2): 117–34.
56.
XingBMarwalaLMarwalaT (2018) Adopt fast, adapt quick: Adaptive approaches in the South African context. In: Adaptive Approaches in the South African Context. Cham: Springer, 171–206.
57.
YanamalaRSuryadevaraNKalliR (2024) Privacy-aware evaluation frameworks for AI systems. Journal of AI Ethics and Privacy15(3): 245–67.
58.
YinRK (2018) Case Study Research and Applications: Design and Methods, 6th edn. London: Sage.
59.
YousefiY (2024) Approaching fairness in digital governance: A multi-dimensional, multi-layered framework. In: Proceedings of the 17th international conference on theory and practice of electronic governance, Pretoria, South Africa, 1–4 October, 41–7. New York: Association for Computing Machinery.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
