Abstract
Introduction:
The United States is experiencing maternity care shortages. Family physicians can play a role in addressing these shortages. Family medicine obstetrics fellowships train family physicians in obstetrics care. Fellowship websites are important for promoting programs and attracting applicants. However, whether websites provide sufficient program information is unknown. This study aimed to assess completeness and utility of family medicine obstetrics fellowship websites across the United States.
Method:
The study analyzed 46 family medicine obstetrics fellowship websites. The component analysis evaluated the presence of 17 components related to orientation, curriculum, program, personnel, and additional content. The qualitative analysis included ratings for navigation and application, information quality, and esthetics. Analysis included percentages for websites and components and average qualitative ratings.
Results:
Common components included overviews, training requirements, and contact information. Description of the patient population was the least common component. Usability ratings varied across programs, with higher ratings observed for navigation and application, and information quality. Esthetics and visual appeal received lower ratings. Regional analysis indicated that websites from fellowships in the West and Southwest tended to include more components compared to those in the Southeast.
Discussion:
Family medicine obstetrics fellowship websites serve as valuable sources of program information for prospective applicants. However, not all websites include essential program details. Some information is rarely provided. Given the shortage of maternity care providers, it is crucial to develop informative and functional websites to attract applicants. Improving website content and design could prove to be a cost-effective strategy to increase the number of applicants.
Keywords
Introduction
Family Medicine (FM) obstetrics (OB) fellowships play a vital role in preparing physicians to provide maternity care and OB services. 1 Across the United States (U.S.), an estimated one million women live in counties without an obstetrician/gynecology (OB/GYN) physician, and about 50% of rural counties are without hospital OB services. 2 The maldistribution of care contributes to maternal mortality and morbidity in resource-poor areas. 3 FM physicians with OB qualifications are a cornerstone in rural health care 4 and can play a vital role in filling the OB care gap. Currently, more than 50 fellowship programs in the U.S. provide FM physicians with training necessary to independently manage most complications typically experienced in maternity care in rural and resource-limited areas. FM OB Fellowships are not accredited, and no standard curriculum has been established; therefore, training on specific procedures, types of care, and conditions varies somewhat across fellowships. 5
FM OB Fellowship websites are increasingly crucial for recruitment. In-person recruitment was limited during the COVID 19 pandemic 6 making programs’ online presence—including websites—essential for potential applicants. 7 Given cost savings associated with virtual recruiting,8,9 some programs may not return to in-person interviews. Without the benefit of site visits, informative, well-designed websites are more important than ever for fellowships programs. 7
Our interest in fellowship websites and improving recruitment is not meant to suggest a shortage of applicants to FM OB fellowships. There are many reasons to improve recruitment other than a dearth of applicants. For example, fellowships have an interest in getting high numbers of highly qualified applicants. Fellowships may also be interested in attracting diverse applicants and increasing their programs reputation and visibility. Finally, in light of the current maternity care shortages, fellowships have an interest in expanding capacity to train additional physicians, and improved recruitment can help with these efforts.
Despite the importance of websites for recruitment, little is known regarding the quantity or quality of information provided in FM OB fellowships in the U.S. Studies have assessed program websites of various medical training programs including radiology, orthopedic surgery, plastic surgery, neurosurgery, otolaryngology, and OBGYN residencies, as well as pediatric radiology and OBGYN fellowships.10 -16 The majority of these studies concluded medical education program websites are not being used to their full potential. On average, across multiple types of programs, websites included about 50% or less of website components assessed.10,12-15 Many websites included important information about programs and curricula, such as descriptions of didactic training, clinical experiences, and hospitals and training sites. However, information about current or previous residents and fellows, away rotations, and call responsibilities was rarely included in websites. Commonly, program-peripheral information such as parking, meal allowance, social life, and housing was missing from the majority of websites.10,12 -16
In our assessment of website content and usability, we borrowed from an existing framework for website evaluation. The framework was designed to evaluate all kinds of websites 17 and not all of the constructs were applicable to FM OB fellowship websites. For example, a large part of the framework focuses on website services, which are not relevant in this context. Our study focused on usefulness, which in the framework consists of functionality and usability. 17 The specific aspect of functionality assessed was the content of the website. The specific aspects of usability assessed included navigability, including ease of application, esthetics, and visual appeal.
To our knowledge this is the first study of its kind to focus on FM OB fellowships in the U.S. The aim of the study was to evaluate the completeness and usability of websites for FM OB fellowships in the U.S. with the intent to identify strategies to improve website content and design.
Method
This research was conducted as part of a larger project funded by the Health Research Services Administration (HRSA). The project includes enhancing OB training for FM residents and FM OB fellows at the University of Utah. As part of enhancements, this FM OB fellowship expanded enrollment from 2 to 4 fellows annually at the University of Utah and made efforts to improve recruitment materials including the fellowship website. This study was conducted to support and inform FM OB fellowship website improvements. This research was reviewed by the University of Utah Institutional Review Board (IRB). The IRB determined the research did not constitute human subjects research (IRB_00168904).
Fellowship Website Selection
The American Association of Family Physicians Fellowship Directory for Surgical OB provided our initial sampling pool of 52 programs. We cross-referenced programs with the American Board of Physician Specialties, which also listed 52 FM OB Fellowship programs. Given the recruitment focus of the analysis, 6 fellowship programs that do not open recruitment to outside FM residencies were excluded (ie, fellowships that add an additional year to a residency or only recruit from the residency they are attached to). The final sample consisted of 46 FM OB fellowships (Supplemental Table 1). Most of these fellowship programs (more than 75%) are located in the Southeast, Pacific, or Midwest regions of the U.S. The remaining programs are located in the Northeast, West, or Southwest (Figure 1).

Family medicine obstetric fellowship program counts for U.S. regions.
Data Collection
After review of studies assessing other graduate medical training program websites and reviewing the content of several FM OB fellowship websites, researchers including directors of the FM residency and FMOB fellowship, FMOB physicians, fellowship alumni, a prospective fellow, and experienced health service researchers developed a list of key website components. Components included were similar to the information components included in other studies of graduate medical training program websites.10 -16 The final list of 17 components (Table 2) was organized into categories including 3 orientation components, 3 curriculum components, 5 program components, 3 people components, and 3 extra components. In September of 2022, 2 researchers (A.G. and R.O.) reviewed fellowship websites for the presence of the 17 components. Prior to reviewing all websites, the researchers independently reviewed 10 fellowship websites and checked for the presence or absence of the components. They discussed discrepancies until consensus was reached and used discussions to develop and refine component descriptions and criteria. In addition to the component analysis, we conducted qualitative ratings for 3 usability elements of the fellowship websites, (1) ease of navigation and application, (2) quality of information provided, and (3) esthetics and visual appeal (Table 2). Four individuals (A.G., R.O., M.M., and A.L.) rated websites. Each rater provided a score from 1 to 5 for each of the 3 elements for each fellowship website. Usability ratings were completed in October 2022.
Analyses
Univariate analyses consisted of calculating percentages and averages in relation to various website characteristics. For the component analysis, we considered and report both the prevalence of components across websites and the completeness of each website. For each component, we calculated the percentage of websites including the component. For each website, we calculated the percent of components present. A similar approach was used for the usability analysis, we calculated the average ratings for the 3 elements across all websites, and the average ratings on the 3 elements for each website. Given the subjective nature of the usability analyses, we calculated interrater agreement using Krippendorff’s alpha. We calculated Krippendorff’s alpha with a tolerance of 0, 1, and 2 for navigation and application, information quality, and esthetics and visual appeal. We report Krippendorff’s alphas as percentages. Finally, we calculated regional results for both the component and the qualitative analyses. We calculated the average percentage of components included in websites and the average ratings for each qualitative element for each region.
Results
Component Analysis
In terms of component prevalence, most websites included important information about fellowship programs (Table 1). For example, all but 3 websites (93%) included a program overview. In addition, 37 websites (80%) included a list of training requirements. Similarly, 37 websites included contact information for the program. The least common component was a description of the patient population, which was present in only 3 websites (7%).
Family Medicine Obstetrics Fellowship Website Components included in Review.
More than a list of training locations or rotation sites.
Either stated or inferred based on information on current or past fellows.
For example, refugee populations, urban underserved, rural, etc.
None of the websites included all 17 components. One website included 14 of the 17 (82%) and was only missing social media links, housing or living information, and a welcome or introduction. Three fellowship websites (7%) included 13 or more of the components. Twenty of the fellowship websites (43%) included 8 or more components. The website with the fewest components included only 2, namely program overview and application link or instructions (Supplemental Table 2).
Usability Ratings
Overall average usability ratings differed between websites in terms of ease of navigation and application, information quality, and esthetics and visual appeal. Table 2 includes rating criteria, overall mean, standard deviation, and range for quality ratings. Scores ranged from 1.25 to 5 for ease of navigation and application; 1.25 to 5 for information quality; and 1.5 to 4.75 for esthetics and visual appeal. As shown in Tables 2 and 3, ratings were on average higher and more varied for ease of navigation and application (M = 3.22, SD = 1.06) and information quality (M = 3.39, SD = 1.04). Ratings were on average lower and less varied for esthetics and visual appeal (M = 2.69, SD = 0.80)
Summary of Usability Ratings of Family Medicine Obstetrics Fellowship Websites.
Family Medicine Obstetrics Fellowship Website Average Usability Ratings.
The
We used Kippendorff’s alpha to assess agreement between raters on elements of usability. Agreement was poor across all 3 elements with a tolerance of 0. With a tolerance of 1 (ie, ratings that were one point apart counted as agreement), agreement was fair for esthetics and visual appeal (
Krippendorff’s Alpha Interrater Reliability for Usability Measures.
Usability ratings took place in October of 2022.
Results by Region
Analysis of website components by region revealed that on average, fellowship websites from the West and Southwest included more components, 56% and 49% respectively. Websites of fellowships in the Midwest, Pacific, and Northeast averaged 41% to 46%. The Southeast average of 33% (about 6 of 17 components) was lowest (Table 5).
Regional Analyses of Family Medicine Obstetrics Fellowship Websites.
Figure 1 is a map of United States Regions. Component analyses took place in September and usability ratings in October of 2022.
In terms of average quality ratings for the regions, there was some variation, but all regions followed the same pattern as the overall finding that ratings tended to be higher for ease of navigation and application and information quality but lower for esthetics and visual appeal. In all 6 regions the average ratings were roughly between 3 and 4 for navigation and application and for information quality (except for the Pacific region which averaged 2.93 for navigation and application). Esthetics and visual appeal averages were lower, between 2.25 and 2.85. Table 4 provides average quality ratings for each region for the 3 elements.
Discussion
The purpose of the study was to assess the inclusion of information in FM OB fellowship websites and the quality of websites. Results indicate that in general, most websites include some important program information for potential applicants. More than half of the fellowship websites included an overview, contact information, training requirements, application information, the number of fellows, an application deadline, and benefits and compensation information. Although most websites included valuable program information, some relevant program information was missing from many of the websites. Over half of the websites included fewer than half of the components assessed. Fewer than a quarter of websites included curriculum information such as clinical descriptions and elective options, and only 7% provided a list or description of the patient populations served. Websites often did not include information components such as faculty, current and past fellows, pictures, information about housing and living in the area, links to social media, and a welcome or introduction message.
Study findings are aligned with findings of similar studies focused on websites for various medical residency and fellowship programs. Our results mirrored the pattern of other studies in which many or most websites contain some important program or curriculum information but are missing other pertinent program and curriculum information. Finally, our results were similar to results of other studies in that few websites included program-peripheral information.
Usability ratings suggest many program websites are well designed. For each of the 3 elements assessed, several programs had average ratings of 4 or higher. A few websites received a rating of 5 from each rater for navigation and application or information quality. Ratings also indicated potential areas for improvement, particularly in terms of esthetics and visual appeal, which had the lowest average ratings overall and for each region. Esthetics and visual appeal considerations included design features such as inclusion of pictures, color schemes, and layout. Design is an important consideration because design features influence accessibility and credibility of website content. 17
To the best of our knowledge, qualitative usability ratings have not been a part of website analyses for medical fellowships or residencies, and not considering website quality has been recognized as a limitation of previous research on medical program websites.10,13,14 Specifically, ease of use11,14 and esthetics 14 have been suggested as important qualitative considerations. One strength of the current study is the inclusion of qualitative usability ratings. Additional research is needed to determine the importance of website quality to potential applicants.
One limitation of this study is the potential subjectivity in identifying components assessed. However, the components reviewed in this study are similar to components included in website analyses of other types of medical programs.10,12,15,16 Another potential limitation is regarding the dynamic nature of websites. The reviews took place in September and October of 2022. Some websites may have been updated since reviews, making any inferences, conclusions, or generalizations based on these findings tenuous. Finally, the importance of the components and the usability elements included in this study for applicants’ decision making has not been established in the literature. Nonetheless, this study provides useful information regarding what information is presented and the completeness, usability, and design of fellowship websites.
Given maternity workforce shortages and the importance of OB fellowships in further preparing FM physicians to provide maternity care, informative, well-designed websites are essential. Given changes to recruitment procedures resulting from the COVID-19 pandemic, websites provide an essential source of information for residents considering fellowship training. The AAFP lists 20 types of FM fellowship programs, 18 reflecting the diverse interests of Family Medicine residents. As such FM OB fellowships are not only competing with one another for fellow recruitment but additionally with a variety of other FM fellowship opportunities. As OB fellowships compete with other fellowships for applicants, improvements to websites could represent a simple, low-cost opportunity with potentially high payoff in increasing the number of program applicants.
This research project was supported by the HRSA of the U.S. Department of Health and Human Services (HHS) as part of an award totaling $2,797,464.00. The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement, by HRSA, HHS, or the U.S. Government.
Supplemental Material
sj-docx-1-jpc-10.1177_21501319231225365 – Supplemental material for Assessment of Family Medicine Obstetrics Fellowship Websites in the United States: Content and Usability
Supplemental material, sj-docx-1-jpc-10.1177_21501319231225365 for Assessment of Family Medicine Obstetrics Fellowship Websites in the United States: Content and Usability by Robert Owens, T. Carl Whittaker, Annie Galt, Saskia Spiess, Matthew Mervis, Alex Lockrey, Elena Gardner, Kirsten Stoesser and Dominik Ose in Journal of Primary Care & Community Health
Supplemental Material
sj-docx-2-jpc-10.1177_21501319231225365 – Supplemental material for Assessment of Family Medicine Obstetrics Fellowship Websites in the United States: Content and Usability
Supplemental material, sj-docx-2-jpc-10.1177_21501319231225365 for Assessment of Family Medicine Obstetrics Fellowship Websites in the United States: Content and Usability by Robert Owens, T. Carl Whittaker, Annie Galt, Saskia Spiess, Matthew Mervis, Alex Lockrey, Elena Gardner, Kirsten Stoesser and Dominik Ose in Journal of Primary Care & Community Health
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research project was supported by a grant from the Health Resources and Services Administration of the U.S. Department of Health and Human Services (HHS).
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
