Abstract
Research capacity building is a process by which individuals and organizations develop skills and infrastructure resulting in greater ability to conduct useful research. Frequently, research quality and productivity measures focus on outcomes, such as peer-reviewed publications or grants awarded. Such distal measures are not informative when measuring initial stages of research, a common situation in low- and middle-income countries. This study used Cooke’s six-principle framework to assess the progress of an autism research collaborative in northern Mexico. We established 64 criteria across the six principles and rated them as Not Started, In Progress, or Achieved, based on narratives and process documentation collected for this purpose. Ratings were established for the years 2018, 2020, and 2022, which correspond to Years 5, 7, and 9 since the beginning of the collaborative, respectively. As of 2022, Principles 2 and 3 (“Close to practice” and “Linkages, collaborations, and partnerships,” respectively) had all begun; no criteria were rated as Not started. The principles with the highest Achieved rates were “Linkages, collaborations, and partnerships” (90%), and “Infrastructure” (Principle 6, 70%). Cooke’s research capacity building framework is a useful way to monitor progress of research capacity building and contribute to harmonious development of relevant principles at different levels.
Lay abstract
To inform improvement of care and public policy, quality research is required. Conducting research projects requires skills and infrastructure. Research capacity building is the process by which individuals and organizations develop greater ability to conduct useful research. However, in the scientific community, research quality and productivity measures are often focused on long-term products, such as publications or grants awarded. Those measures are not helpful when measuring initial stages of research, a common situation in low- and middle-income countries. We used a six-principle framework designed by J. Cooke to assess the progress of an autism research collaborative in northern Mexico. We established 64 criteria across the six principles, and rated them as Not Started, In Progress, or Achieved, based on narratives and process documentation collected for this purpose. Ratings were established for the Years 2018, 2020, and 2022. The principles with the highest Achieved rates were “Linkages, collaborations, and partnerships” (90%), and “Infrastructure” (Principle 6, 70%). Cooke’s research capacity building framework is a useful way to monitor progress of research capacity building and contribute to harmonious development of relevant principles at different levels.
Background
Despite the great progress in the field of research on autism spectrum disorder (ASD), there is still great regional disparity in the production of high-quality, rigorous research (Elsabbagh et al., 2012). Most of the knowledge on ASD and developmental disorders comes from research conducted in high-income countries (HICs), such as the United States, the United Kingdom, and other European countries. However, 90% of people who live with ASD live in low- and middle-income (LMICs) countries, such as in Africa, Southern and Western Asia, and South America (De Vries, 2016). This disparity in the way autism research is produced is not always conducive to generalization of findings or their much-needed implementation in low-resource settings.
In addition, and arguably more importantly, autism research conducted via community partnerships is linked to raising awareness and strengthening service capacity within the environment where it occurs. This service capacity is generated from within the community involved in the development process to use resources in an effective, efficient, relevant, and sustainable way (Lavergne & Saxby, 2001; Miranda et al., 2018). For example, in the process of conducting research on early detection of autism, strong clinicians must be trained to conduct reliable diagnostic assessments. Those clinicians, who have learned by doing and have willfully acquired skills over time, will increase capacity to provide services where research was conducted even after the study is complete, and even further, may share their expertise with colleagues. Other health practitioners will interact with families who benefited from early detection, becoming aware of the implications for the lives of their patients or clients. In time, best clinical practices resulting from these high-quality, comprehensive assessments may also inform local policies.
Due to these cumulative and important benefits, it is most rational to consider the structural transformation caused by research processes as yet another facet of the societal impact of research, beyond the relevance of research findings to health care delivery and policymaking (Health Sciences Subcommittee of the Medical Committee of Royal Netherlands Academy of Arts and Sciences, 2001; Smith, 2001). Therefore, the fact that autism research happens almost exclusively in wealthy countries perpetuates the inequality in access to services and supports for individuals and families living in low-resource settings.
Research capacity building (henceforth RCB, but also addressed elsewhere as Research Capacity Development or Research Capacity Strengthening) has been defined as “a process of individual and institutional development which leads to higher levels of skills and greater ability to perform useful research” (Trostle, 1992, p. 1321), or a process of developing sustainable abilities and skills enabling individuals and organizations to perform high-quality research (Holden et al., 2012). As such, RCB is an integral component of the pipeline for the production of sound evidence to inform clinical practice and policy decision-making. However, research capacity (RC) is often measured with longer-term outcomes of high-quality research, such as peer-reviewed publications, conference presentations, or successful grant applications (see Khisa et al., 2019, for an ample analysis). While this conceptualization of RC may offer a practical, panoramic view to assess research in HICs, it does not address relevant inducements and constraints playing a role in this progress, particularly throughout the initial stages of RCB or when it occurs in LMICs. RCB has been pointed out as an essential process by which LMICs can develop the capacity to perform effective and locally led health research addressing the major health problems affecting their own populations (Davey, 2004). Historically, health RCB involving international collaboration has been delivered through four main approaches: vertical research projects, centers of excellence, North–South collaborations, and networks (Franzen et al., 2017). Although each of the main approaches has strengths and weaknesses, there is still a lack of empirical evidence on monitoring and evaluating the implementation of these approaches.
Because of the constraints imposed by current standards in autism research regarding proprietary diagnostic tools and evidence-based interventions, transfer of knowledge between HICs and LMICs results in under-representation of LMICs in the literature and remains a consistent need that usually requires international collaboration (Durkin et al., 2015). North–South scientific cooperation (between a HIC and a LMIC) is a common approach that, despite its benefits, has been heavily criticized for originating from, and perpetuating, a power imbalance between the northern and southern parties (Kok et al., 2017). Traditionally, in these collaborations, the northern party leads and makes decisions, while the southern party frequently takes on a passive role and almost completely becomes the subject of research. To counter this power imbalance, some authors have proposed a distinctive argument that describes more equitable collaborations as “partnerships” and emphasizes that RC should not be foreign-led or foreign-owned (Costello & Zumla, 2000; ESSENCE on Health Research, 2011). Furthermore, a variation of the North–South partnerships, the South–North partnership, has been conceptualized (Marcellesi & Palacios, 2008) stemming from the former to emphasize the leadership role of the southern partner. This leading role entails an equitable participation in setting research goals, playing a strong role on data analysis and acting not only as a data collections site but also as an equal research partner.
To plan change and measure progress in relevant areas of RCB across four levels of operation, a six-principle framework was proposed by Cooke (2005). This bidimensional framework allows for the evaluation of both process and outcome measures. The six principles, as outlined by Cooke are “(1) Research capacity is built by developing appropriate skills, and confidence, through training and creating opportunities to apply skills” (Building skills and confidence); “(2) Research capacity building should support research ‘lose to practice’ in order for it to be useful” (Close to practice); “(3) Linkages, partnerships, and collaborations enhance research capacity building” (Linkages, collaborations, and partnerships), “(4) Research capacity building should ensure appropriate dissemination to maximize impact” (Appropriate dissemination and impact); “(5) Research capacity building should include elements of continuity and sustainability” (Continuity and sustainability); and “(6) Appropriate infrastructures enhance research capacity building” (Infrastructure). In this model, these six principles are examined across four structural levels of operation: (1) individual, (2) team, (3) organizational, and (4) network or supra-organizational support. The framework also acknowledges that RCB occurs within a policy context. This framework has been successfully used to evaluate initiatives to increase research capabilities in primary care settings (Cooke et al., 2008; Moore et al., 2012), allied health professions (Wenke et al., 2017), and local government (Homer et al., 2022).
In Mexico, the estimated prevalence of autism is of 1 in 115, as calculated by the only prevalence study up-to-date (Fombonne et al., 2016). While there have been efforts to strengthen the country’s capacity to increase autism awareness and address clinical needs, autism research occurs primarily in the country’s capital, Mexico City, and a handful of tertiary referral centers (Lizardi & Salazar, 2018/2020). In their analysis to describe the bibliometric profile of the global scientific research on ASD, Sweileh and colleagues (2016) found that the whole region of Latin America produced only 12% of the published research on ASD between 2005 and 2014, whereas 65% came from three countries (the United States, the United Kingdom, and Canada). Mexico’s budget for research and development has shrunk steadily since 2016, with the latest report from the Organization for Economic Co-operation and Development (OECD) calculating the research budget at 0.3% of the national Gross Domestic Product (GDP) in 2020, an all-time low since the year 2000, when the GDP spending in Research and Development (R&D) was 0.306% (OECD, 2022). Such important constraints are just one of the challenges that researchers must overcome, and one of the reasons for which research capacity in Mexico has been hindered to rise and meet the country’s many health-related needs. In light of this scenario, research often happens under the sponsorship of private foundations or non-profit entities. Such conditions are not predictable and depend highly on a group’s ability to advocate and appeal to potential donors.
Autismo ABP is a city-level non-profit organization in Monterrey, Mexico, which has served approximately 200 adolescents and young adults on the autism spectrum with high-support needs through a day center program since 2010. Since its inception, Autismo ABP set out to build capacity to serve the needs of high-support youth and young adults on the autism spectrum who had never been included in organized programs or had received very limited or no services. With the experience gained after several years of running the day center, Autismo ABP manualized its operations, developed a social franchise model, and secured the collaboration of the local Catholic archdiocese to provide parochial spaces around the city that could be used by neighborhood groups interested in replicating the initiative. Its presence in the community has been steady and holds a reputation as a trustworthy community resource that has an important advocacy voice in the Monterrey metro area. Autismo ABP, along with similar organizations that were providing support to a critically underserved population, identified as urgent the need for early detection and intervention in ASD. To address this need, the Autism Global Panel (AGP) was created in 2013. AGP is a transdisciplinary, trans-sectorial network of publicly and privately funded organizations in the state of Nuevo Leon, in northern Mexico. AGP’s goal is to advance clinical and research capacity in autism, foster awareness and acceptance, and engage in advocacy in northern Mexico. Such unprecedented collaboration stemmed from the need to unify efforts from the (1) education sector (i.e. state-level education secretariat—Secretaría de Educación—, and several universities); (2) health sector (i.e. Instituto Mexicano del Seguro Social (Social Security administration), Academia Mexicana de Pediatría Sección Noreste (regional chapter of the Mexican Academy of Pediatrics), and Colegio de Pediatría de Nuevo León (state-level college of pediatricians), including state-level public health services (i.e. Servicios de Salud de Nuevo León) and local private health organizations (e.g. Clínica Nova); (3) private business sector (e.g. Nemak, Soriana, Alestra, and Grupo Promax); and (4) social development sector (e.g. Secretaría de Desarrollo Social, Desarrollo Integral para la Familia, and multiple local non-profits, including Autismo ABP). The AGP representatives envisioned a working plan that would not only benefit the region’s population directly but also generate knowledge about their own population through research and facilitate their participation in the global conversation. For this purpose, in 2017, the AGP reached out to the A.J. Drexel Autism Institute (AJDAI), in Philadelphia, PA, USA, in what became a South–North research partnership with the dual goal of (a) building autism research capacity in northern Mexico and (b) carrying out specific research projects regarding autism early detection and intervention. Importantly, and in support of this southern-led model, the AJDAI entered the research partnership as a consultant. Such arrangement underlies and protects the provision that the southern partner establishes the research priorities, owns the research data, and holds an equitable share of any scientific products from this collaboration. The AJDAI’s focus is autism research from a public health perspective and the study of the health determinants triad in autism: (1) primary determinants, or factors contributing to the origins of an outcome (e.g. a diagnosis of autism or its underlying features); (2) secondary determinants, including factors influencing timely detection and early intervention, services, and supports; and (3) tertiary determinants comprising factors impacting daily functioning and quality of life over the life course (Schendel et al., 2021).
The AGP’s first collaborative research project with the AJDAI, “Único en 115,” launched in December 2018, on a 5-year timeline. The aim of this study is to validate the illustrated Spanish version of the Modified Checklist for Autism in Toddlers, Revised with Follow-Up (M-CHAT-R/F; Perez Liz, 2022). Recruitment of toddlers attending the well-child visit and their caregivers was initially carried out in primary care clinics and then moved online when the COVID-19 pandemic hit. Screening data were collected through an electronic platform, and participants were contacted telephonically to collect follow-up answers and schedule in-person developmental and diagnostic evaluation when appropriate. The AGP has since added two other research collaborations that launched in 2021: (1) the country’s adaptation of the Caregiver Skills Training Program (CST, World Health Organization/Autism Speaks) and (2) a randomized-clinical trial (RCT) stemming from animal model studies (Sauer et al., 2021) to address the developmental outcomes of children born from mothers receiving zinc-amino-acid supplementation during pregnancy. At the time of the third time point of the current RCB evaluation, the CST had not started recruiting participants and the RCT had just started recruiting participants (pregnant women on their first trimester) at a maternal and child hospital in Monterrey, Mexico.
Given these collaborative projects, the aim of this study was to evaluate the ongoing process of autism RCB set by the AGP in northern Mexico. To accomplish this, we used Cooke’s framework, applying it to three time points: 2018, 2020, and 2022. This evaluation will allow for identification of strengths and needs, and will inform future directions of the RCB effort.
Methods
Measure
Based on capacity development strategies designed to address specific barriers to health research as outlined in Franzen et al. (2017), leaders of the organizations in the Monterrey metro area comprising the AGP created a list of priorities to be addressed by the autism RCB effort. The research team drew from this narrative to create criteria for each of the six principles of RCB according to Cooke’s framework, guided by the examples of suggested criteria provided by the author. These criteria were then categorized into the four levels of capacity development (individuals, teams, care-giving organizations, and networks and support units): (1) Skills and confidence building (14 criteria); (2) Support of research “close to practice” (11 criteria); (3) Linkages, Partnerships, and Collaborations (10 criteria); (4) Appropriate dissemination (9 criteria); (5) Continuity and sustainability (10 criteria); and (6) Appropriate infrastructures (10 criteria). The rating of the criteria was based on the contents of oral narratives collected from two founding members of AGP, project coordination and administrative records, written communications with the local team, and input from the local research team. See Figure 1 for the complete RCB matrix with ratings at each time point. Table 1 shows two examples of how the progress ratings were granted based on available evidence.
Matrix of criteria for the evaluation of RCB in Monterrey, Mexico, for the years 2018 (T1, 2020; T2; and 2022 (T3), across the six principles and four dimensions outlined by Cooke (2005).
Examples of evidence used to grant a rating.
Procedures
The matrix of criteria for each principle × level was rated according to the development stage as Not started, In Progress, or Achieved at three time points: November 2018, December 2020, and March 2022. The first 5 years since the creation of the AGP (2013–2018) were devoted to raising awareness, building relationships, creating research partnerships, and exploring and securing funding sources. Once the first research study was ready to launch, we decided to establish the baseline for the RCB evaluation (Time 1 (T1)) to monitor the study’s progress. Originally, the RCB evaluation was set to occur every 3 years. However, due to the COVID-19 pandemic, all participant recruitment and data collection activities for the first active study, “Único en 115,” were temporarily suspended, and the team decided to re-evaluate RCB a year earlier (T2) to make adjustments to the processes if necessary. T3 ratings were conducted in 2022 for consistency and also to establish another measure as the burden from the pandemic began to decrease and recruitment and data collection activities returned to the expected pace.
Two co-authors (G.P.L. and J.R.M.C.) independently rated the criteria based on all information gathered. Inter-rater agreement was evaluated using intra-class correlation. Agreement rates were 98% for T1, 96% for T2, and 98% for T3. A third author G.V. (founder of AGP) resolved any discrepancies between raters.
Community involvement
Involvement of families with autistic members of all ages was part of the development of the research capacity initiative and the formulation of a plan to evaluate it. The authoring team comprised clinicians, scholars, researchers, and community leaders. Some of them also have personal experience as parents and caregivers of autistic children, or close friends of families with autistic children. The interest and preferences of families with autistic children informed this analysis and are reflected in the future directions of our work.
Results
At T1 (2018), 31 of 64 criteria (48.4%) were rated as Achieved or In Progress. The remaining criteria (51.5%) were rated as Not Started. As shown in Table 2, at T1, all six principles had 66% or less of their criteria rated as Achieved or In Progress (summarized as “Engaged” on Table 2). Only Principle 3 had more than one third of its criteria rated as Achieved, while the other five principles had less than 33% of their criteria in this classification. Overall, Principles 1 and 2 (Building skills and confidence, and Close to practice) were rated as having 50% and 54.5% of criteria within the Not started status, respectively, whereas Principle 3 (Linkages, collaborations, and partnerships) was rated as having all of its criteria within the In Progress or Achieved status. Principles 4, 5, and 6 (Appropriate dissemination and impact, Continuity and sustainability, and Infrastructure) obtained a status rating of Not Started for 77.7%, 70%, and 63.4%, respectively (see Figure 1).
Classification of principles at the three time points according to the percentage of their criteria rated.
In Progress + Achieved.
At T2 (2020), 51 of the 64 criteria (79.6%) were rated as Achieved or In Progress. The remaining criteria (20.4%) were rated as Not Started (Figure 1). Principles 2 and 3 (Close to practice, and Linkages, collaborations, and partnerships) were rated as not having any of the criteria within the Not started status, whereas Principles 4 and 5 (Appropriate dissemination and impact, and Continuity and sustainability) were rated as having none of its criteria within the Achieved status yet. Principles 1 and 6 (Building skills and confidence, and Infrastructure) decreased the status of Not Started from 50% to 21.4%, and 60% to 10%, respectively. As shown in Table 2, at T2, all but Principle 4 had their criteria rated more than 66% within the category of “Started” (either In Progress or Achieved). More than two thirds of the criteria from Principle 3 (Linkages, collaborations, and partnerships) were rated as Achieved, while Principles 2, 4, and 5 (Close to practice, Continuity and sustainability, and Infrastructure, respectively) had less than one third of their criteria rated as Achieved by T2.
Finally, at T3 (2022), 59 of the 64 criteria (92.2%) were rated as either In Progress or Achieved. As shown in Table 2, all six principles had more than two thirds of their criteria rated as In Progress or Achieved (summarized as “Started”). Principles 3 and 6 (Linkages, collaborations, and partnerships, and Infrastructure, respectively) had 66% or more of their criteria rated as Achieved, while Principle 4 (Appropriate dissemination and impact) had less than one third of its criteria with this rating. The five criteria rated as Not started at T3 were as follows: Training in advocacy skills for board (from Principle 1, Building skills and confidence), Seminar programs related to research undertaken (from Principle 4, Appropriate dissemination and impact), Successful grant/fellowship applications, and Fellowship support (from Principle 5, Continuity and sustainability), and co-organize/co-host scientific events (from Principle 6, Infrastructure).
Discussion
The progress documented in this initial evaluation of the RBC process in northern Mexico is promising and will contribute to inform the efforts prioritized as new funding are pursued and upcoming projects launched.
Performing an evaluation of the ongoing projects every 2 years has allowed a panoramic view of the development of several areas relevant to RCB. Moreover, it has highlighted the strengths of the group collaboration and its projects, which will allow for AGP to address challenges and barriers and to advance the RCB efforts and set specific research goals. The principle identified as the main strength throughout AGP’s process of autism RCB was Linkages, collaborations, and partnerships (Principle 3), which at T1 had all of its criteria engaged and remained as the principle with highest percentage of criteria rated as Achieved, with 50%, 70%, and 90% during T1, T2, and T3, respectively. Likewise, this framework allowed for the identification of challenging areas that require further attention for optimal development. Such was the case of five criteria, all of which were rated as Not started at all three time points: Training in advocacy skills for board, from Principle 1 (Building skills and confidence); Seminar programs related to research undertaken, from Principle 4 (Appropriate dissemination and impact); Successful grant/fellowship applications, and Fellowship support, from Principle 5 (Continuity and sustainability); and Co-organize/co-host scientific events, from Principle 6 (Infrastructure). Four of these five criteria imply an established scholarly practice, which is in progress as existing resources continue to develop.
This analysis allowed for a closer look at how we evaluate progress within processes even within the same principle, and long before the research outcomes materialize. For example, some criteria are “finite,” meaning that after being achieved, they do not require further advancement, such as Establishing a primary chair committee for research within AGP, from Principle 1. Others, however, are either “continuous” processes that remain active throughout time, such as Professional and community trainings for intervention strategies in early childhood, or “cyclic,” such as Training for volunteers and research assistants, both from Principle 1 as well. These differences are not necessarily reflected in the ratings of the criteria but remain important elements to consider when evaluating the process of RCB. In the context of our RCB initiative, this framework was helpful to delineate relevant criteria that would require monitoring and re-evaluation throughout time. It also allows flexibility to remove criteria that are permanently achieved, lost relevance to the project, or have radically changed to follow the needs of the population where RCB is carried out.
While it is out of the scope of this analysis to make a determination regarding the effectiveness or appropriateness of our RCB approach, tracking progress across time on several principles and levels supports the equity of roles in this South–North research partnership. It also highlights the commitment that northern partners must make to contribute to the strengthening of the southern partner’s research capacity and the symmetry of the collaboration.
We identified some limitations through the use of this framework. First, in some categories, the descriptor In progress was more complex, or took more intermediate steps, than in others. Some criteria require more intermediate steps to advance from Not started to Achieved, and because of this, a more quantitative approach of the evaluation would be more appropriate to measure progress more sensitively. Similarly, this framework does not weight the importance of specific criteria within a level as RCB is built. For example, on a specific project, some of the criteria under the Individual level might be as critical for Principle 1 (Building skills and confidence) as the criteria under the Organizational level is for Principle 6 (Infrastructure).
Cooke’s framework is useful in that it allows for flexibility to establish the measurement of intermediate products of the process of RCB. However, building capacity at any structural level depends on funding and support opportunities, which are heavily influenced by policy and funding sources, making comparisons between research initiatives difficult. While frameworks to evaluate RCB help “unpack” the contents of the black box of processes, more research is still needed to elucidate the link between intermediate achievements and terminal outcomes.
Throughout the process of research partnership and its present analysis, we have observed that social impact starts happening well before research outcomes are palpable and academic scientific impact are noticeable.
Conclusion
This study used Cooke’s framework to conduct sequential preliminary evaluations of the process of RCB in northern Mexico, as set by the AGP, at three time points (November 2018, December 2020, and March 2022). This framework allowed monitoring the progress on intermediate (proximal) products in the process of RCB, in addition to the terminal (distal) outcomes, more traditionally established as markers of successful research (scientific publications, grants). We were able to document change and progress in the criteria that were set at the beginning of the trans-sectorial, transdisciplinary autism research partnership, highlighting strengths and weaknesses of our work group throughout the process. This type of analysis can inform prioritization of efforts in the following years and hopefully serve as a guide for other nascent autism research initiatives in LMICs.
