Digital transformation of nephrology POCUS education—Integrating a multiagent,artificial intelligence,and human collaboration-enhanced curriculum with expert feedback

Abstract

Background

The digital transformation in medical education is reshaping how clinical skills, such as point-of-care ultrasound (POCUS), are taught. In nephrology fellowship programs, POCUS is essential for enhancing diagnostic accuracy, guiding procedures, and optimizing patient management. To address these evolving demands, we developed an artificial intelligence (AI)-driven POCUS curriculum using a multiagent approach that integrates human expertise with advanced AI models, thereby elevating educational standards and better preparing fellows for contemporary clinical practice.

Methods

In April 2024, the Mayo Clinic Minnesota Nephrology Fellowship Program initiated a novel AI-assisted process to design a comprehensive POCUS curriculum. This process integrated multiple advanced AI models—including GPT-4.0, Claude 3.0 Opus, Gemini Advanced, and Meta AI with Llama 3—to generate initial drafts and iteratively refine content. A panel of blinded nephrology POCUS experts subsequently reviewed and modified the AI-generated material to ensure both clinical relevance and educational rigor.

Results

The curriculum underwent 12 iterative revisions, incorporating feedback from 29 communications across AI models. Key features of the final curriculum included expanded core topics, diversified teaching methods, enhanced assessment tools, and integration into inpatient and outpatient nephrology rotations. The curriculum emphasized quality assurance, POCUS limitations, and essential clinical applications, such as fistula/graft evaluation and software integration. Alignment with certification standards further strengthened its utility. AI models contributed significantly to the curriculum's foundational structure, while human experts provided critical clinical insights.

Conclusion

This curriculum, enhanced through a multiagent approach that combines AI and human collaboration, exemplifies the transformative potential of digital tools in nephrology education. The innovative framework seamlessly integrates advanced AI models with expert clinical insights, providing a scalable model for medical curriculum development that is responsive to evolving educational demands. The synergy between technological innovation and human expertise holds promising implications for advancing fellowship training. Future studies should evaluate its impact on clinical competencies and patient outcomes across diverse practice environments.

Introduction

Nephrology education has increasingly focused on bridging the gap between theoretical knowledge and practical clinical skills. Point-of-care ultrasound (POCUS) has recently emerged as a pivotal tool, providing real-time insights into patient conditions directly at the bedside.^1–3 Within nephrology, POCUS can enhance diagnostic accuracy, guide procedural interventions, and support comprehensive patient management.^4,5 Despite its immense potential, POCUS education faces significant challenges, including variability in the availability of standardized training programs, inconsistencies in faculty expertise, and a lack of structured assessment frameworks.^6–8 Additionally, disparities in trainees’ prior exposure to ultrasound, comfort with technology, and clinical experience contribute to variability in skill acquisition. For example, trainees from radiology or emergency medicine backgrounds may have greater familiarity with ultrasound imaging compared to those from nephrology-focused training. These confounding factors create significant barriers to achieving consistent proficiency among trainees.

Curriculum deficiencies remain a key contributor to this variability, as traditional approaches often fail to provide structured and adaptive learning pathways tailored to individual needs. Conventional POCUS training programs may rely heavily on faculty availability and in-person sessions, which are subject to resource limitations and time constraints. By improving the curriculum, specifically through the integration of adaptive technologies like Artificial Intelligence (AI), we aim to reduce variability by addressing these systemic gaps. A well-designed curriculum can serve as an equalizer, offering all trainees a consistent, high-quality educational experience regardless of their background. For instance, AI-enabled feedback mechanisms can provide trainees with immediate, objective assessments of their performance, allowing them to identify and address specific skill gaps in real time. Moreover, AI's ability to analyze large datasets can help identify common challenges faced by trainees and adjust the curriculum to address those areas systematically.^9,10

POCUS has transformed clinical practice in fields like emergency medicine, critical care, cardiology, pulmonology, gastroenterology, nephrology, vascular surgery, and other medical and surgical subspecialities. ² Traditionally, nephrology has relied on static imaging modalities such as ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI) for diagnostics. ¹¹ However, these methods often involve delays, logistical challenges, and limited accessibility, particularly in acute care settings. POCUS addresses these limitations by enabling just-in-time imaging at the bedside, allowing nephrologists to make immediate and informed decisions regarding patient care. ¹² The adoption of POCUS is driven by its applicability in a wide range of clinical scenarios, from assessing kidney conditions to guiding procedures like biopsies and dialysis catheter placements. Despite its growing recognition, the integration of POCUS into training programs remains inconsistent, leading to variability in skill acquisition among trainees and potentially impacting patient outcomes.^13–15 This variability calls for a more innovative approach to curriculum development, where AI can potentially play a crucial role in ensuring consistency and comprehensiveness.

AI has already demonstrated success in improving educational outcomes in other clinical domains. For example, AI-powered tools have been used in radiology training to enhance the interpretation of imaging studies, allowing trainees to achieve diagnostic accuracy comparable to experienced radiologists.^16,17 Similarly, in surgical education, AI-driven simulations have been shown to improve procedural skills and decision-making by offering individualized feedback and performance metrics.^18,19 These successes underscore AI's potential to revolutionize nephrology POCUS education by creating a dynamic, data-driven, and scalable training framework.

Incorporating AI into the development of a comprehensive POCUS curriculum offers a novel solution to the challenges faced by traditional curriculum design.^20–22 AI's ability to process vast amounts of educational content, identify gaps, and suggest improvements based on best practices is invaluable in creating an adaptable and comprehensive curriculum.^23–25 This AI-driven process is further enhanced by continuous feedback from human experts, who ensure that the curriculum remains clinically relevant and aligned with the practical needs of nephrology fellows. Utilizing AI in curriculum development allows training programs to more effectively address the variability in POCUS education, ensuring that all nephrology fellows receive a consistent and high-quality educational experience. ²⁶ By integrating AI into the development of a comprehensive POCUS curriculum, we aim to address these challenges and create an adaptive learning environment that minimizes variability in skill acquisition. This approach not only ensures that all trainees achieve foundational and advanced competencies but also prepares them to apply POCUS effectively in diverse clinical scenarios, ultimately improving patient care outcomes.

This study represents a significant step forward in nephrology education by demonstrating the effectiveness of AI-assisted curriculum development combined with human expert feedback loops. Utilizing advanced AI models, including GPT-4.0, Claude 3.0 Opus, Gemini Advanced, and Meta AI with Llama 3, we generated and refined a comprehensive POCUS curriculum tailored to the specific needs of nephrology fellows. ²⁷ The iterative process of feedback and refinement, incorporating insights from experienced nephrologists, ensured that the final curriculum was not only technically robust but also clinically relevant and practical. This innovative approach highlights the potential of AI in medical education, setting a new standard for curriculum development that can adapt to the evolving demands of clinical practice. By integrating AI and human expertise, this study provides a roadmap for creating a standardized POCUS curriculum that prepares the next generation of nephrologists to deliver high-quality, patient-centered care.

Method

Study design

This study was designed as a collaborative initiative to develop a standardized curriculum for POCUS training in nephrology, conducted within the Mayo Clinic Minnesota Nephrology Fellowship Program. The study utilized a hybrid methodology combining AI tools and human expert feedback. Specifically, it aimed to create a comprehensive, adaptable educational framework tailored to the unique needs of nephrology fellows.

This study represents a curriculum development and validation project. The primary objective was to integrate advanced AI models into the curriculum design process, leveraging their analytical and iterative capabilities to generate a robust foundational structure. Human experts subsequently refined the AI-generated content to ensure clinical relevance and alignment with nephrology-specific training requirements. This approach bridges the gap between traditional curriculum development and innovative digital tools.

Participants

The study engaged four advanced AI models and a panel of three experienced nephrology faculty members specializing in POCUS. Specifically, GPT-4.0 was responsible for generating the initial drafts of the curriculum; Claude 3.0 Opus provided iterative feedback focused on refining curriculum content, structure, and clinical relevance; Gemini Advanced assisted in enhancing educational methods and assessment strategies; and Meta AI with Llama 3 played a critical role in integrating feedback from all sources to produce a cohesive final curriculum (Figure 1). The human experts were blinded to the AI's involvement to minimize bias during the review process.

Figure 1.

Collaborative process involving AI models and human experts in developing the POCUS curriculum. The initial draft was created by GPT-4.0, followed by iterative feedback loops from three advanced AI models: Gemini Advanced (focusing on educational methods and assessment strategies), Meta AI with Llama 3 (integrating feedback from all sources), and Claude 3.0 Opus (refining curriculum content, structure, and clinical relevance). Each AI model iterated through multiple versions (v.1, v.5, v.10) before the curriculum was reviewed by human experts (v.12). The human expert feedback was then incorporated (v.13) to produce the final curriculum.

Rationale for choosing specific LLMs and role allocation for curriculum development

The selection and role allocation of ChatGPT, Gemini AI, Meta AI, and Claude AI for this study were driven by their distinct strengths and capabilities, allowing for a collaborative approach to curriculum development. ChatGPT was chosen for its exceptional natural language processing skills, enabling the efficient creation of structured and comprehensive drafts that formed the initial framework of the curriculum.^28–30 Gemini AI was selected for its expertise in integrating advanced educational strategies, including instructional design and assessment methods, which aligned with the study's goal of developing a robust learning framework. ³¹ Meta AI contributed by synthesizing and refining feedback from multiple sources, ensuring the curriculum's cohesiveness and alignment with educational and clinical standards. ³² Claude AI, recognized for its analytical precision and contextual refinement, was tasked with ensuring clinical relevance and structural coherence throughout the iterative development process.^33,34 Together, these LLMs worked synergistically, with specific roles tailored to their unique functionalities—ChatGPT focusing on initial drafts, Gemini AI on educational strategies, Meta AI on feedback integration, and Claude AI on iterative refinement. This strategic allocation of roles enabled a multidimensional and efficient approach to developing a comprehensive and adaptive POCUS curriculum.

Standardized text prompts for LLMs

We have explicitly detailed the standardized text prompts used to guide each LLM in its respective role. For instance, ChatGPT was tasked with creating the initial draft of the curriculum using the prompt: “Develop a comprehensive Nephrology POCUS curriculum tailored to a fellowship program. Include objectives, core topics, educational methods, and assessment strategies with practical and theoretical components.” Gemini AI, Meta AI, and Claude AI were provided similarly specific prompts aligned with their designated roles, ensuring consistency and focus throughout the iterative development process (Online Supplementary).

Protocol for incorporating human expert feedback

The process for integrating feedback from human experts followed a structured and systematic approach. Feedback was collected using standardized templates, categorized into specific domains (e.g., clinical relevance, educational structure), and iteratively integrated into subsequent drafts. Each iteration involved targeted revisions guided by feedback-informed prompts to the LLMs, ensuring alignment with both clinical and educational objectives. This iterative process continued until consensus was achieved among human experts and AI-generated content.

Determining the number of iterations for feedback

The number of feedback iterations was determined based on two primary considerations: the complexity of the curriculum and the need to ensure alignment with both educational and clinical objectives. An initial target of 12 iterations was set, allowing sufficient refinement cycles to address feedback from AI models and human experts comprehensively. This iterative approach ensured that the curriculum evolved systematically, incorporating insights at each stage to enhance its relevance, accuracy, and applicability. The process was monitored closely, and additional iterations were added as necessary to address specific areas of improvement highlighted during the feedback process. The iterative refinement continued until consensus was achieved among the AI models and human experts regarding the curriculum's final structure and content.

Curriculum development process

1. Initial curriculum drafting: GPT-4.0 generated the first draft of the curriculum, including core topics, learning objectives, educational methods, and assessment strategies.

2. Iterative feedback and refinement: The curriculum underwent 12 iterative rounds of revision, facilitated by the AI models, each contributing distinct enhancements to content, structure, and applicability.

3. Expert validation: Human experts reviewed the refined curriculum, providing critical clinical insights and recommendations for final revisions.

4. Finalization: The feedback from both AI and human experts was synthesized to produce a comprehensive curriculum ready for integration into both inpatient and outpatient nephrology rotations. Quality assurance protocols were also developed, emphasizing the recognition of POCUS limitations and the need for further imaging or interventions when necessary.

This study did not involve human or animal subjects, and no patient data or identifiable information was used. Therefore, Institutional Review Board approval was not required, and participant consent was waived. The curriculum development and refinement process adhered to institutional ethical standards for research without human or animal involvement, ensuring transparency and compliance with all applicable regulations.

Results

AI-assisted curriculum development and iterative refinement:

The AI-assisted curriculum development process began with GPT-4.0 generating the initial draft of the Nephrology POCUS curriculum. This draft included core topics, learning objectives, educational methods, and assessment strategies tailored to the specific needs of nephrology fellows. The draft covered essential topics such as ultrasound physics, renal ultrasound, bladder ultrasound, vascular access, volume assessment, and procedural guidance relevant to nephrology practice.

Following the initial draft's creation, the curriculum underwent 12 rounds of iterative feedback and refinement, facilitated by the three other AI models, i.e., Claude 3.0 Opus, Gemini Advanced, and Meta AI with Llama 3 (Online Supplementary). The AI models provided 29 communications, each contributing unique insights that enhanced the curriculum's structure, content, and relevance to clinical practice. Claude 3.0 Opus focused on expanding the curriculum's core topics, particularly emphasizing the inclusion of advanced POCUS techniques such as Doppler ultrasound and its application in special populations, including pediatric and geriatric patients. The model also recommended enhancements to the sections on patient communication and the interpretation of ultrasound findings, ensuring that the curriculum addressed the nuances of patient interaction during POCUS examinations.

Gemini Advanced contributed significantly to refining the educational methods and assessment strategies within the curriculum. This model suggested the incorporation of a variety of instructional techniques, including hands-on training sessions, simulation exercises, and case-based learning modules. Additionally, Gemini Advanced recommended the use of formative and summative assessments, such as quizzes, practical exams, and peer-reviewed performance evaluations, to ensure that trainees achieved the desired competencies throughout their fellowship. The model also emphasized the integration of telemedicine and remote POCUS applications, reflecting the growing trend of digital health in nephrology practice.

Meta AI with Llama 3 played a crucial role in synthesizing the feedback from the other AI models and ensuring that the curriculum was cohesive and well-aligned with the overall educational objectives of the nephrology fellowship program. Meta AI with Llama 3 ensured that all revisions were implemented effectively, leading to a balanced and comprehensive final curriculum (Online Supplementary).

Proportion of AI vs. human contribution to the final curriculum

To objectively assess the proportion of the final Nephrology POCUS curriculum shaped by AI versus human experts, we analyzed the key curriculum components before and after human intervention, as detailed in Table 1. The AI-generated content provided a strong foundational structure, covering essential topics, educational methods, and assessment strategies. However, the human experts significantly refined these elements by adding clinical relevance, practical insights, and specific enhancements tailored to nephrology practice.

Table 1.

AI vs. human contribution to nephrology POCUS curriculum.

Curriculum component	AI-generated	Post-human modification
Core topics	Ultrasound physics, renal ultrasound, bladder ultrasound, vascular access	Added emphasis on fistula/graft assessment, open-ended volume assessment, and advanced Doppler techniques
Educational methods	Hands-on training, case-based learning, quizzes	Integration of Qpath software, mentorship progression, and telemedicine applications
Assessment strategies	Formative and summative assessments, practical exams	Introduction of peer-reviewed performance evaluations and standardization per ASDIN guidelines

The AI models contributed significantly to the foundational structure, including topics, methods, and assessments. Human experts refined these elements by focusing on specific clinical relevance, such as fistula assessment and software utilization. This makes it evident that while AI laid the groundwork, the human experts added practical insights grounded in clinical experience.

Comparison with other POCUS curriculums

To objectively compare the AI-generated Nephrology POCUS curriculum with other established curriculums, such as those developed by Dr Abhilash Koratala and others,^{8,14,21,25,26,35} we analyzed key aspects as summarized in Table 2. This comparison highlights the unique contributions made by AI in curriculum development, particularly in the areas of iterative refinement, software integration, and open-ended approaches to clinical assessments.

Table 2.

Comparison of AI-enhanced nephrology POCUS curriculum with other curriculums.

Aspect	AI-enhanced curriculum	Koratala's curriculum 21	Other curriculums8,14,25,26,35
Training model	AI-augmented feedback loops with iterative refinement	Focus on hands-on training, simulation, and faculty-guided learning	Mixed methods with emphasis on traditional didactics
Image review	Integrated Qpath software for standardization	Utilization of standardized checklists and peer review	Basic review techniques, often less standardized
Volume assessment	Open-ended approach, the inclusion of lung ultrasound	Focus on IVC ultrasound and VExUS scoring	Traditional methods focusing on IVC and basic kidney imaging
Certification	AI suggested alignment with ASDIN standards	Emphasis on training that leads to recognized certification	Often lacking clear certification pathways

The AI-driven curriculum emphasizes a dynamic and flexible approach, particularly in the integration of advanced technologies such as Qpath software for image review and an open-ended strategy for volume assessment that includes lung ultrasound. This approach contrasts with more traditional, faculty-driven curriculums that may focus more narrowly on established methods like IVC ultrasound and VExUS scoring. The AI-generated curriculum also uniquely suggests alignment with ASDIN standards for certification, offering a structured pathway for achieving recognized qualifications, which is often not as clearly defined in other curriculums.

Human expert feedback

Once the AI models completed their iterative feedback process, the refined curriculum was reviewed by a panel of three expert nephrologists with extensive experience in POCUS. These nephrologists conducted a blinded review, focusing on the curriculum's clinical accuracy, relevance, and practical applicability to nephrology practice.

The human experts provided valuable insights that further refined the curriculum. Key areas of feedback included (1) emphasis on the importance of using POCUS for the assessment of arteriovenous fistulas and grafts, given their critical role in the management of dialysis patients, with a recommendation to expand the curriculum to include more detailed training on detecting complications such as stenosis and thrombosis. (2) They suggested making the volume assessment component of the curriculum more open-ended, allowing fellows to explore different methods of assessing fluid status, including the use of lung ultrasound and the evaluation of the inferior vena cava (IVC). This approach was intended to encourage critical thinking and adaptability in various clinical scenarios. (3) The experts recommended structuring the curriculum to guide fellows from novice to expert levels systematically. This progression would involve gradually increasing the complexity of POCUS techniques and clinical applications as fellows advanced through their training. The experts also suggested incorporating mentorship opportunities, where more experienced fellows could assist junior trainees in developing their POCUS skills. (4) They also advised the inclusion of specific software tools, such as Qpath, for mentored scans. ³⁶ Qpath software, commonly used for ultrasound image archiving and quality assurance, was recommended to facilitate the standardized review of trainee scans, providing consistent feedback and aiding in skill development. ¹⁰ And finally, (5) Experts questioned the inclusion of certain procedures within the curriculum and recommended pursuing certification through recognized bodies such as the American Society of Diagnostic and Interventional Nephrology (ASDIN). They also suggested that Mayo Clinic could adapt its certification standards based on ASDIN requirements, ensuring that the curriculum met or exceeded industry benchmarks.

Final curriculum

Incorporating the feedback from both the AI models and the experts, the final version of the Nephrology POCUS curriculum was completed. This curriculum features included: (1) Expanded core topics: The final curriculum included a comprehensive set of core topics covering all essential aspects of POCUS relevant to nephrology, including advanced techniques and applications in special populations. (2) Diverse educational methods: A variety of instructional techniques were incorporated, including didactic lectures, hands-on training, simulation exercises, and case-based learning. The curriculum also included opportunities for mentorship and peer-to-peer learning. (3) Robust assessment and feedback mechanisms: The curriculum integrated formative and summative assessments to evaluate trainee progress continuously. These assessments included quizzes, practical exams, and peer-reviewed performance evaluations, ensuring that fellows achieved the required competencies at each stage of their training. (4) Quality assurance protocols: Emphasis was placed on quality assurance, with protocols developed to ensure the accuracy and reliability of POCUS images obtained by trainees. The curriculum also addressed the limitations of POCUS and emphasized the importance of recognizing when additional imaging or interventions were necessary. (5) Integration into clinical rotations: The curriculum was designed to be seamlessly integrated into both inpatient and outpatient nephrology rotations, ensuring that fellows could apply their POCUS skills in real-world clinical settings.

Discussion

The AI-assisted curriculum development process, combined with expert nephrologist feedback, successfully produced a comprehensive Nephrology POCUS curriculum tailored to the needs of nephrology fellows. The curriculum included expanded core topics, diverse educational methods, and robust assessment strategies. The iterative refinement, facilitated by AI models and human expert reviews, ensured that the curriculum was both clinically relevant and adaptable to the evolving demands of nephrology practice.

In the development of core topics, the initial AI-generated draft included fundamental subjects such as ultrasound physics, renal ultrasound, bladder ultrasound, and vascular access. However, during expert review, nephrologists identified the need for more specialized content. For example, they recommended the inclusion of advanced Doppler techniques and a focused module on arteriovenous fistula and graft assessments. These changes were implemented to address critical clinical scenarios such as detecting stenosis or thrombosis, thereby ensuring that the curriculum not only covered basic imaging principles but also incorporated detailed, practical insights specific to nephrology practice. Regarding educational methods and assessment strategies, the AI-generated content initially proposed a structure based on hands-on training, case-based learning, and formative assessments. Expert intervention refined this framework by integrating additional components such as telemedicine applications, mentorship progression, and the incorporation of specialized software like Qpath for standardized image review. ¹⁰ Furthermore, the experts introduced peer-reviewed performance evaluations and aligned assessment criteria with recognized certification standards (e.g., ASDIN guidelines). This collaborative refinement process ensured that the curriculum provided a balanced and systematic progression from novice to expert levels, effectively combining the efficiency of AI-driven content creation with the depth of clinical experience.

One unexpected result was the feedback regarding the perceived overemphasis on the VExUS (Venous Excess Ultrasound) score.^37,38 While VExUS is commonly highlighted in POCUS literature for assessing venous congestion, it was suggested that it may be overrated, prompting a reconsideration of its role within the curriculum.^39–41 This prompted a more balanced approach, integrating alternative volume assessment tools and reducing reliance on a single scoring system. Another unexpected outcome was the questioning of the inclusion of procedures like thoracentesis and paracentesis in the curriculum. However, because these procedures are crucial for managing pleural effusions and ascites—important components of congestion scores, as demonstrated in the ADVOR trial—their inclusion was affirmed. This feedback led to a more thoughtful consideration of how these procedures are integrated into the curriculum, ensuring that they are relevant to nephrology-focused POCUS training.

The kidney ultrasound module, in particular, highlights the importance of addressing both basic and advanced diagnostic elements. ²¹ Beyond basic parameters such as kidney size, structure, cysts, masses, and hydronephrosis, the curriculum includes advanced components such as corticomedullary differentiation, medullary-cortical thickness, calcifications, Doppler resistance index, renal artery flow, and arterial turbulence.^{8,14,21,25,26,35} Nephrology POCUS should emphasize detecting bladder volume, identifying abnormalities such as thickened bladder walls or masses, evaluating urinary retention, assessing post-void residual volume, and recognizing obstruction-related changes. For vascular assessments, it should include evaluating aortic abnormalities such as aneurysms or dissections, which are critical in managing hypertension and advanced kidney disease. Limb arteriopathy, often coexisting with chronic kidney disease, should be integrated as a key competency to assess peripheral vascular disease and its impact on dialysis outcomes. Additionally, mapping the vasculature of the upper arm, including the cephalic vein and brachial artery, should be highlighted as a critical skill for optimizing arteriovenous fistula creation. Fellows should be trained to assess vessel diameter, wall integrity, and flow dynamics to ensure the successful creation and long-term patency of vascular access. These competencies prepare fellows for high-level clinical decision-making across a diverse range of nephrology scenarios.

The results of this study align with existing literature emphasizing the importance of POCUS in nephrology for enhancing diagnostic accuracy and guiding procedures. ⁴² However, the study's innovative use of AI in curriculum development is less commonly reported in the literature, making this approach a novel contribution to the field. The integration of AI was not merely a supplementary tool but as a vital component in the iterative development process. Through the application of advanced AI models to generate, evaluate, and refine curriculum content, this study demonstrates a transformative shift from traditional methods of curriculum design, which often rely solely on human expertise. The use of AI allowed for a dynamic and responsive development cycle, ensuring continuous improvements based on real-time feedback. Looking forward, the measurable learning outcomes incorporated into this curriculum also pave the way for future research on their impact. Studies can investigate how defined benchmarks influence trainee performance and whether these outcomes translate to improved diagnostic and procedural accuracy in clinical practice. Moreover, this structured approach to outcomes assessment could serve as a framework for evaluating other AI-assisted educational initiatives.

Future iterations of the curriculum should focus on integrating a comprehensive list of specific abnormalities that nephrology fellows must be trained to identify for each organ system, including key pathologies such as hydronephrosis, renal masses, bladder wall thickening, aortic aneurysms, vascular stenosis, and limb arteriopathy. Each organ system should also incorporate a checklist of minimum requirements that fellows must achieve to demonstrate competency, such as identifying normal anatomy, detecting subtle abnormalities, and interpreting findings within clinical contexts. Clear and measurable learning outcomes should be defined for each module to standardize expectations and enable both fellows and educators to track progress effectively. For instance, learning objectives could specify the ability to accurately measure kidney dimensions, assess vascular flow using Doppler techniques, and interpret bladder post-void residuals. These additions will provide fellows with a structured roadmap for their training, reinforce the practical application of POCUS, and facilitate structured assessments aligned with certification standards. By including these components, the curriculum will become more robust and practical, ensuring that fellows acquire the advanced skills needed to utilize POCUS confidently and effectively in nephrology practice. Future research should validate these learning outcomes, assess their impact on fellows’ performance, and evaluate their influence on patient care outcomes, further contributing to the refinement of nephrology POCUS education.

The iterative feedback process, involving both AI models and human experts, represents a significant advancement over traditional curriculum development methods, which often lack such dynamic refinement capabilities. The expert feedback that led to a more detailed and flexible approach to volume assessment and procedural training is also supported by literature advocating for adaptability in POCUS education, given the rapid evolution of ultrasound technology and its applications. The successful development of the curriculum can be attributed to the synergistic combination of AI's analytical capabilities and the clinical insights of experienced nephrologists. The AI models provided a structured, data-driven approach to content creation, while the expert feedback ensured that the curriculum was grounded in practical, real-world experience. The unexpected results, such as the reconsideration of the VExUS score and the inclusion of certain procedures, highlight the importance of human oversight in AI-assisted processes. While AI can generate comprehensive content, human experts are essential for contextualizing and refining that content based on clinical realities.

In light of the challenges posed by the American Board of Internal Medicine's (ABIM) current decision not to mandate POCUS education for nephrology fellows, ³ it becomes increasingly important to consider alternative strategies to enhance the adoption and effectiveness of POCUS training. One potential approach involves leveraging AI to streamline the curriculum, focusing on its most essential components to create a simple yet impactful educational framework. This AI-driven curriculum could emphasize key areas such as essential imaging techniques, particularly renal and vascular ultrasound, which are crucial for nephrology practice. Additionally, it could prioritize critical volume assessment tools, such as the IVC and lung ultrasound, which are vital for assessing fluid status. The curriculum might also focus on practical procedural guidance, particularly ultrasound-guided biopsies and catheter placements, which are directly relevant to nephrology. By simplifying the curriculum in this manner, it could become more accessible and easier to implement, particularly in programs with limited resources, while still achieving significant educational outcomes.

A primary limitation of our study is that it was conducted within a single institution, the Mayo Clinic Minnesota Nephrology Fellowship Program. Consequently, the curriculum may be specifically tailored to the unique needs and resources of this program, which might limit its generalizability to institutions with different patient populations, faculty expertise, or technological capabilities. Furthermore, we acknowledge that AI-generated content carries potential risks, such as inherent bias, misinformation, or content gaps. To address these concerns, we implemented a multi-layered mitigation strategy. First, we employed multiple AI models (namely GPT-4.0, Claude 3.0 Opus, Gemini Advanced, and Meta AI with Llama 3) to cross-validate the content and ensure a broader, more balanced perspective. Second, each round of AI-generated output underwent a rigorous iterative review by expert nephrologists who critically assessed its clinical relevance, accuracy, and completeness. This expert oversight was essential for identifying and correcting any instances of bias or misinformation. In addition, we used standardized prompts and feedback templates to guide the AI responses, thereby minimizing inconsistencies or omissions, and we implemented quality assurance protocols to align the curriculum with established clinical guidelines and certification standards such as those from ASDIN. Collectively, these measures reinforced the overall integrity and reliability of the final curriculum.

We also recognize the importance of developing clear metrics to enable an objective comparison between this AI-assisted curriculum and existing POCUS training programs. Future iterations will focus on standardizing benchmarks for skill acquisition, trainee confidence, and patient care outcomes. To provide quantifiable evidence of efficacy, we plan to incorporate objective measures such as time to competency, accuracy in image acquisition and interpretation, and the effective application of POCUS findings in clinical decision-making. Feedback from both trainees and educators will be systematically collected to evaluate the curriculum's impact on learning and teaching experiences. Additionally, we acknowledge that not having yet tested the curriculum with trainees limits our ability to demonstrate its practical efficacy in addressing the challenges outlined in the introduction. Future studies will therefore include pilot implementations with nephrology fellows across multiple programs, employing a multimodal assessment framework that encompasses pre- and post-training tests, performance evaluations in both simulated and clinical scenarios, and longitudinal tracking of skill retention and application. Comparative studies with existing curricula will further validate the program's capacity to reduce variability in skill acquisition and improve learning outcomes. Through these efforts, we aim to establish robust evidence of the curriculum's effectiveness and secure its development into a validated, impactful tool for nephrology training. Future studies should focus on implementing this curriculum in a variety of nephrology fellowship programs across different institutions to assess its effectiveness and adaptability in diverse settings. ³⁵ Comparative studies could evaluate the outcomes of fellows trained with this AI-assisted curriculum versus those trained with traditional curricula. Further research could also explore the potential of AI to facilitate ongoing curriculum updates, ensuring that the educational content remains current with the latest advancements in POCUS technology and practice. The implications of this study extend beyond nephrology. The AI-assisted curriculum development model could be adapted for other medical specialties, offering a scalable solution for creating standardized, high-quality educational programs across various disciplines.

The AI-generated curriculum was developed at the Mayo Clinic, a well-resourced institution, and its current form reflects the availability of advanced technology and faculty expertise. However, the curriculum was intentionally designed with a modular structure that facilitates adaptation across different resource settings. For example, the core components—such as fundamental imaging techniques, basic ultrasound physics, and essential clinical applications—are applicable using standard ultrasound equipment and do not require advanced digital infrastructure. In settings with limited resources, these core modules can be prioritized, while advanced modules that depend on specialized software (e.g., Qpath) or enhanced telemedicine capabilities can be modified or substituted with locally available alternatives. A tiered implementation approach may also be employed to accommodate varying levels of faculty expertise. In institutions where faculty resources are constrained, online training modules, tele-mentoring, and remote expert consultations can supplement in-person instruction, ensuring that trainees receive the necessary guidance and support. This flexible framework allows for incremental adoption of the curriculum, enabling institutions to build capacity gradually while still benefiting from the structured, AI-enhanced educational model. Further research is necessary to evaluate the curriculum's effectiveness across diverse settings. Future studies will include pilot implementations in institutions with varying resource levels to assess the adaptability, scalability, and overall impact of the curriculum. Such comparative analyses will inform necessary modifications to ensure that the educational benefits of the AI-assisted curriculum are accessible and effective regardless of institutional constraints.

Conclusions

In summary, the AI-assisted, expert-reviewed curriculum development process successfully produced a comprehensive Nephrology POCUS curriculum that is both innovative and practically applicable. The integration of AI in the design and refinement stages, including iterative feedback from advanced AI models and validation by expert nephrologists, ensured that the final curriculum addressed the critical needs of nephrology fellows while remaining adaptable to emerging technologies and evolving clinical practices. This hybrid approach not only standardized educational content but also demonstrated the potential of AI to enhance the quality and consistency of medical training programs.

The study's outcomes suggest that this approach holds significant promise for improving nephrology education and serving as a model for curriculum innovation across other medical specialties. While initial implementation results are promising, comprehensive longitudinal studies are needed to validate the curriculum's effectiveness in improving trainee competence and patient outcomes. Future research should also focus on adapting this curriculum to diverse institutional settings, particularly resource-limited environments, to assess its scalability and broader applicability.

Supplemental Material

sj-pdf-1-dhj-10.1177_20552076251328807 - Supplemental material for Digital transformation of nephrology POCUS education—Integrating a multiagent, artificial intelligence, and human collaboration-enhanced curriculum with expert feedback

Supplemental material, sj-pdf-1-dhj-10.1177_20552076251328807 for Digital transformation of nephrology POCUS education—Integrating a multiagent, artificial intelligence, and human collaboration-enhanced curriculum with expert feedback by Mohammad S Sheikh, Kianoush B Kashani, James R Gregoire, Charat Thongprayoon, Jing Miao, Iasmina M Craici and Wisit Cheungpasitporn, Fawad M Qureshi in DIGITAL HEALTH