Content validity of an item bank to assess intervention characteristic determinants of implementing evidence-based practices

Abstract

Background

Barriers and facilitators, collectively called determinants, of evidence-based practice implementation are key to identifying the best strategies for promoting implementation. Assessing determinants before implementation can help tailor strategies to those that would be most effective. Current measures of determinants are not comparable across implementation settings and implementation scientists and practitioners often have to create their own measures. This study was the first step in creating determinants item banks that are usable across settings and focused on intervention characteristics. We aimed to establish the content validity of the item bank.

Method

This study used a concurrent mixed methods approach. Items for assessing intervention characteristic determinants were first identified through systematic reviews. Implementation scientists then completed a survey where they provided both quantitative and qualitative feedback on the items. Finally, three experts with both clinical and implementation experience provided feedback on redundancy and representativeness.

Results

The systematic reviews identified over 1,959 items so subsequent steps were limited to focus on intervention characteristic determinants (271 items) such as adaptability of the practice. Based on feedback from thirty implementation scientists, the items were reduced to 92 but an additional 53 were added, most due to qualitative feedback. Items were also rewritten based on qualitative results. Three experts reviewed the remaining 145 items. Based on their feedback, the number of items was reduced to 109.

Conclusions

Creating a determinants item bank was feasible and the final items had content validity. The next steps include testing reliability and validity in a larger sample of clinicians implementing evidence-based practices.

Plain Language Summary

Barriers prevent or impede an organization from using a new practice or innovation. Facilitators help promote the use of a new practice or innovation within an organization. Assessing barriers and facilitators before starting a new practice can help target barriers and increase the chances of successfully using the practice. This study created new measures of barriers and facilitators of using a new practice or innovation. Previous measures were identified through literature reviews. Implementation scientists provided feedback on the measures through an online survey. Three experts with clinical and implementation experience provided additional feedback. Measures were revised based on the survey and expert feedback. The next steps are to test the measures with clinicians implementing a new practice.

Keywords

determinants barriers strategies uptake

Introduction

One of the first steps to implementing an evidence-based intervention is identifying the barriers and facilitators to implementation. Barriers and facilitators are called determinants because they influence whether the evidence-based intervention gets implemented (Koppelaar et al., 2009). However, determinants are not always identified before deploying implementation strategies (Baker et al., 2010). Current methods for identifying determinants are very labor intensive (Lawrence et al., 2016) or too general to then identify appropriate implementation strategies (Geerligs et al., 2018; Kajermo et al., 2010). Other measures are specific to an intervention or setting and therefore not usable across studies, or do not map to implementation science frameworks (Martinez et al., 2014). Flexible yet comparable and accessible methods of assessing determinants are needed.

The purpose of the current study was to start addressing this critical barrier to assessing determinants of implementing evidence-based interventions. This study is the first in a series that aims to develop flexible and valid measures of determinants that can be easily used to assess determinants before and during implementation. Due to the potential scope of determinants, the project was initially limited to determinants around intervention characteristics that include relative advantage, complexity, trialability, adaptability, evidence strength and quality, cost, compatibility, and observability as defined by the Consolidated Framework for Implementation Research (CFIR) and the Diffusion of Innovation theory (Damschroder et al., 2009; Rogers, 2003). CFIR labels and defines determinants at various levels (provider, intervention, organization, and community levels). We use the CFIR definition of intervention characteristics, specifically that these are the attributes of a practice that can influence implementation. We chose intervention characteristics to start with because most implementation scientists and practitioners will have some control over adaptations to the intervention that could be guided by determinants. The aim was to create an item bank for each intervention characteristic, pick a term for evidence-based intervention and decide whether questions should apply to before implementation, during implementation, or both. Item banks are often developed using item response theory (Reise & Haviland, 2005), a modern statistical approach that allows items to be used in any configuration. Item response theory scores the items in such a way that different studies or projects can select different items based on what is most important to them but scores are still comparable across studies and to norms or cutpoints. Here, we report the initial content validity studies for these determinant item banks, following the procedures outlined by Streiner and Norman (2015) using a literature review, expert opinion, and a survey to create a quantitative content validity assessment.

Method

We first compiled items of determinants used in previous studies into an item bank. We started with items from one general review (Miake-Lye et al., 2020) and then included measures identified from two other reviews of determinant measures, specifically because these reviews identified which measures focused on intervention characteristics (Chaudoir et al., 2013; Lewis et al., 2021). Reviews not focused on intervention characteristics were included to capture measures with relevant items but that were not specifically intervention characteristic measures. Items were added to a spreadsheet and coded by which the CFIR determinant was represented. Over the course of coding the items, several were coded in a general “other” category. After reviewing the content of these items, the investigative team decided to add three categories. The first category was observability from the Diffusion of Innovation (DOI) framework (Rogers, 2003). The last two categories were risk and burden. Table 1 summarizes the definitions of each determinant including the ultimate definition derived from the item bank and the CFIR and DOI frameworks. We then obtained copyright approval for items and eliminated items for which copyright approval was not available (N = 100).

Table 1

Definitions of Intervention Characteristic Determinants

Domain	Damschroder et al. (2009) definition	Rogers’ definitions	Definition from an item bank, incorporating previous definitions
Evidence Strength and Quality	Stakeholders’ perceptions of the quality and validity of evidence supporting the belief that the intervention will have desired outcomes.		Perceived quality, flaws, and applicability of research or guidelines on the practice to the adopter's setting or patient population.
Relative Advantage	Stakeholders’ perception of the advantage of implementing the intervention versus an alternative solution.	The degree to which the intervention is believed to be better than the practice it replaces.	Perception of the advantages and disadvantages of the practice compared to current practice, previous practice, and other potential practices.
Adaptability	The degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needs.		The degree to which the practice can be adapted to the setting or patients and the clarity about how the practice should be adapted and what should not be adapted.
Trialability	The ability to test the intervention on a small scale in the organization, and to be able to reverse course (undo implementation) if warranted.	How testable an intervention is before committing to adopting it.	Ease of testing a practice before adopting it with all patients, locations (such as clinics), or more globally.
Complexity	Perceived difficulty of implementation, reflected by duration, scope, radicalness, disruptiveness, centrality, and intricacy and a number of steps required to implement.	The level of difficulty when using or trying to understand the intervention.	How easy or difficult a practice is to implement or use.
Design Quality and Packaging	Perceived excellence in how the intervention is bundled, presented, and assembled.		Quality or ease of using written materials, information, or terminology related to the practice.
Perceived Costs	Costs of the intervention and costs associated with implementing the intervention including investment, supply, and opportunity costs.		Adopters’ perceptions about the costs to implement or use the practice and perceptions of available funding to implement or use the practice.
Compatibility	The degree of tangible fit between meaning and values attached to the intervention by involved individuals, how those align with individuals’ own norms, values, and perceived risks and needs, and how the intervention fits with existing workflows and system	The degree to which the intervention is consistent with an adopter's values, experiences, and beliefs.	Alignment of the practice with the values, mission, beliefs, and other practices of the individual adopter, their team, and their larger organization.
Observability		The intervention or practice produces benefits to patients or others that can clearly be seen, are tangible, or clearly due to the intervention or practice.	The perceived ability of the practice to produce clear, meaningful benefits to patients.
Risk	Partially included in compatibility		The practice can increase or decrease the risk of lawsuits, harming patients, bringing up negative information or other bad outcomes.
Burden			The practice is too time-consuming, labor-intensive, or otherwise consumes too many resources other than costs.

Content Validity Survey

Next, we assessed the content validity of the item bank using a concurrent mixed methods approach. The institutional review board reviewed procedures and determined the content validity survey was exempt. We recruited 30 implementation scientists to complete an online survey and provide feedback on the 271 items in the bank. Participants were recruited through the professional networks of the investigative team and were sent email invitations explaining the study and providing a link to the survey. Inclusion criteria were self-identifying as an implementation scientist and the ability to complete the survey in English. Participants reviewed an informed consent statement before completing the survey. The quantitative portion of the survey used the content validity index (CVI; Polit et al., 2007) and had participants rate each item's relevance to the determinant domain using a 4-point scale (not relevant, somewhat relevant, quite relevant, highly relevant). Items were retained if 78% of participants rated the item relevance as quite or highly relevant to that determinant (Polit et al., 2007). Incentives were not provided for completing the survey.

Qualitative feedback was solicited in two ways. First, questions at the beginning of the survey asked for feedback on using two sets of items, one for a planned implementation effort and another for an implementation effort underway (Supplemental Materials). Another open-ended question asked for feedback on terminology for referring to the evidence-based intervention and the larger organizations in which people work. Each question prompt had an open text field in which participants could type responses. Data was coded by two members of the investigative team using a primarily inductive, framework approach (Elo & Kyngas, 2008). The first coder read through the responses and created a preliminary codebook with 19 codes. The second coder read the codebook and responses. The two coders then independently coded each response. Discrepancies in coding were identified and discussed by the two coders. All discrepancies were resolved through discussion and revision of the codebook. The definitions for three codes were revised and one code was split into two codes. The second method of soliciting qualitative feedback was an open text field after each determinant that asked for feedback on missing items or concepts for that domain. Each domain had a question on missing or misplaced items for that domain. The principal investigator drafted new items based on the comments. The investigative team reviewed new items for wording and representativeness of the determinant. The quantitative data informed the inclusion of specific items and the qualitative data informed the general structure of the item bank.

Expert Panel Review

Following the content validity survey, additional feedback on the item bank was sought from experts. Experts had to be actively practicing in some form of clinical care (medicine, psychology, social work, physical therapy, and nursing) and be either an implementation scientist or have experience implementing new practices in a clinical setting. The investigative team identified experts through their professional networks by contacting colleagues in social work, nursing, and physical therapy. Experts (n = 3) then reviewed a spreadsheet of the remaining items and provided feedback on redundancy, representativeness, and clarity for each item. Experts received a stipend for completing the review. The principal investigator then reviewed the feedback and eliminated or reworded items based on the expert feedback. Another member of the investigative team then reviewed the principal investigator's decisions and any disagreements were resolved through discussion.

Results

After compiling items from previous measures and obtaining copyright approval, 271 items were identified as potential candidates for the item bank. Through the process of categorizing the items into the CFIR categories, the need for additional intervention characteristic categories became clear as many items did not fall within the CFIR intervention characteristics based on item content. The DOI category of observability was added as were categories for risk and burden from the evidence-based practice (Table 1). The risk and burden categories were added based on item content and because these items did not clearly fit within either the DOI or CFIR definitions for intervention characteristic domains. The principal investigator reviewed and categorized each item that did not fit in a CFIR domain based on item content and another member of the investigative team provided feedback when categorization was unclear. Items were then rewritten to have a consistent format and so items could apply to multiple settings and implementation phases. For multiple settings, the items each used a generic term (“practice”) and asked respondents to think of the practice being implemented when they saw the term “practice.” For implementation phases, verbs, and verb tenses were used that could apply to a future implementation project or a current implementation project.

Content Validity Survey: Quantitative Results

Thirty implementation scientists rated the relevance of the 271 items to the corresponding domain (see Table 2 for characteristics). Based on the 23 participants likely completing the survey in one sitting, we estimate the survey took 45 min to complete. Based on the CVI, 179 items were eliminated resulting in 92 items (Supplemental Materials). However, based on suggestions from the participants and analysis of the text responses, an additional 46 items were drafted and added to the item bank. An additional 7 items were added as copyright approval was received after the content validity survey was fielded. Retained items were also revised based on feedback from participants. After the content validity survey, 145 items remained in the item bank across 11 intervention characteristic domains.

Table 2

Sample Description

Characteristic	Mean (SD) or N (%)
Age, years	47.2 (12.5)
Gender
Male	6 (20.0%)
Female	21 (70.0%)
Other, declined to answer	3 (10.0%)
Race/Ethnicity
American Indian/Alaska Native	0 (0.0%)
Asian American	0 (0.0%)
Black or African American	1 (3.3%)
Hispanic or Latina/American	0 (0.0%)
Middle Eastern or North African	0 (0.0%)
Native Hawaiian or Pacific Islander	0 (0.0%)
White or Caucasian	25 (83.3%)
Other	2 (6.7%)
Declined to answer	3 (10.0%)
Research Area
Cancer	15 (50.0%)
Mental Health	9 (30.0%)
Substance Use	5 (16.7%)
Primary care	10 (33.3%)
Other	8 (26.7%)

Content Validity Survey: Qualitative Results

The results of the qualitative analysis on having one set of questions or two sets of questions for practices implemented and not yet implemented indicated a split between preferring one set of questions or two sets (Table 3). Support for both options was demonstrated. A single set of questions would be simpler and easier to use and could track determinants over time. Tracking determinants over time could inform whether implementation strategies worked. Two sets of questions were indicated because there would be differences between the two phases. Participants stated that planning the implementation and implementing the practice were two separate processes and determinants might differ between the two phases. Some participants reasoned that one set of questions would lose the specificity and relevance for assessing determinants, thereby supporting two sets of questions. Preventing mistakes was a potential advantage for both options. One set or two sets of questions was not clearly marked as the better option.

Table 3

Qualitative Analysis Results for One Versus Two Sets of Questions

Category	Definition	Illustrative Quote for Those Preferring Standardized
One set of questions (Option 1)	Participant prefers to have the same set of questions that assesses barriers for people implementing a practice and people planning to implement a practice OR sees value in having one set of questions.	n = 14“Phrase questions so they apply to both situations"
Two sets of questions (Option 2)	Participant prefers two different sets of questions, one for places already implementing a practice and a separate set for those planning an implementation OR sees value in having two sets of questions.	n = 16“Separately would be my preference”
Simpler	Preference is because one option would be simpler, more pragmatic, easier, or more parsimonious.	n = 6“It seems like it would be difficult to create generic questions”
Different	Planning an implementation is too different from the process of actually implementing a practice. Some barriers and facilitators may not apply before vs. during an implementation. Before vs. during implementation are fundamentally different.	n = 12“Potentially two different challenges”
Track over timer	One option allows barriers and facilitators to be tracked or assessed over time, possibly to see if an implementation strategy worked	n = 2“When it comes to analyses you can potentially compare”
Prevent mistakes	One option increases the chances of the questionnaires being used correctly, not being altered or not being misused.	n = 2“This option seems more likely to avoid misuse”
Both	The participant reports the benefits of both options or suggests an alternative that uses parts of both.	n = 7“One set of questions with the option of tailoring the referent”
Does not matter	The issue of wording is not crucial to measuring barriers and facilitators.	n = 1“I'm not sure that this is necessarily a critical question”

Participants were asked whether they preferred one set of questions for both pre-implementation and during implementation or two sets of questions, one for each phase.

Results on using the terms “practice” and “organization” suggested support for this terminology (Table 4). Most participants supported the use of the term “practice,” citing it was a more generalized term that was not only associated with clinical practices. Alternative terms included “intervention” instead of “practice” and “setting” instead of “organization.” However, others reported “intervention” implied experimental intervention and was not as inclusive as “practice”. The majority supported using “organization” because it was broad and could be clearly defined to expand beyond medical practices. Participants also stated the need to define these terms clearly, consider non-health settings, and make the questions apply beyond healthcare. For example, the term “practice” is often associated with clinical practice and not necessarily public health. Ultimately, support for using the terms “practice” and “organization” with definitions was found.

Table 4

Qualitative Analysis Results for Terminology

Category	Definition	Illustrative Quote for Those Preferring Standardized
Practice	The participant clearly implies or explicitly states that using “practice” makes sense to them.	n = 18“I like the term ‘this practice’ since it is inclusive."
Organization	The participant clearly implies or explicitly states that using “organization” makes sense to them.	n = 19“My preference is organization”
Do not use “practice”	The participant does not like the term “practice” or the term could be confusing such as medical practice.	n = 8“I do not like ‘this practice’”
Use intervention	Participants suggested using “intervention” in place of “practice.”	n = 4“Maybe use this intervention or innovation”
Do not use intervention	The participant does not like the term “intervention.”	n = 3“Prefer to avoid using ‘intervention’”
Use Setting	Participants suggested using “setting” instead of organization.	n = 3“Something generic like setting”
Need to define	The terms should be defined clearly on the final questionnaire including level for organization and setting and to what practice refers.	n = 5“Practice is probably fine if clearly defined”
Non-health setting	The questions need to account for nonhealthcare settings such as public health or community organizations or do not currently apply to nonhealthcare settings.	n = 4“So much work is in setting not related to healthcare so would want to capture that”
Do not use evidence-based practice	The participant does not like the term “evidence-based practice.”	n = 1“There is a ton of confusing around what evidence-based practice actually is”
Expansive	The participant suggested making the questions flexible so they can be tailored to different settings or that the questions should not be restrictive to just one type of organization like healthcare.	n = 4“I would try not to be restrictive”
Do not use organization	The term organization does not work or is too broad.	n = 4“From my experience organization is too broad”
Alternatives	Other alternative terms such as “program” not captured by the other codes.	n = 5“You might consider program”

Expert Review

Three clinical and implementation experts, representing social work and physical therapy, reviewed the 145 items and provided feedback about whether items were redundant with other items or representative of the domain and on item wording. As the qualitative feedback did not clearly indicate one or two sets of items was preferred, we decided to create one set of questions that could apply to both the planning and implementing phases. One set of questions allows tracking over time and is easier for scientists and practitioners to use. The terms “practice” and “organization” were retained as qualitative results supported these. Hence, experts were asked to review only one set of items that could apply to both phases and that used “practice” terminology. The principal investigator synthesized the expert feedback and eliminated 36 items. Items marked “redundant” or “not representative” across all three experts were eliminated. Items so marked by two experts were reviewed by the principal investigator for deletion. Another member of the investigative team reviewed the inclusion/exclusion decisions and disagreements were resolved through discussion. The remaining 109 items were rewritten based on expert feedback (see Supplemental Material for final item bank).

Discussion

The current study reported on the development and content validity of an item bank to assess intervention characteristic determinants of implementing an evidence-based practice. Using previous literature reviews and a mixed methods approach, this study established the content validity of the item bank across 11 determinants and informed the structure and wording of the questionnaire. Developing an item bank to assess intervention characteristic determinants was feasible, supporting the development of other item banks for additional determinants.

Our results suggest several potential drawbacks for current determinants measures that could be addressed by the item banks. Few items substantially overlapped and implementation experts suggested many items in addition to those already created. This implies that current measures are not adequately capturing intervention characteristic determinants. Nearly two-thirds of the items from previous measures were not rated as relevant by implementation experts and practically all the items needed to be rewritten. This could have been due to previous measures being specific to the setting or intervention. The determinant item banks will help address these drawbacks, by expanding what is measured, eliminating potentially redundant items, and creating comparable measures for use across settings.

Our next steps for developing the item bank include surveying a large sample of clinicians and other healthcare personnel involved in implementing an evidence-based practice. We plan to use item response theory (IRT; Reise & Revicki, 2015 ) to further establish the reliability of the item bank. Using IRT also means the item bank will be flexible and implementation scientists will be able to build customized assessments of determinants that are tailored to their settings but still comparable across studies. We will also use IRT to create a series of short forms for each determinant that are tailored to specific uses such as generally screening for barriers to target with implementation strategies and monitoring barriers over time.

Using the intervention characteristics determinants item bank as a model, we also plan to develop determinant item banks for other domains. For example, item banks to assess provider characteristics (Damschroder et al., 2009) could help target training and support to provider needs. Item banks to assess patient characteristics could inform intervention adaptations based on patient needs. Our plan is to use IRT for future determinants item banks.

The results should be considered within the strengths and limitations of the study. A strength was the use of multiple mixed methods to develop the item banks. While we developed these measures for use in healthcare, most items were phrased so they could be used in other implementation settings such as education or public health. However, additional content validity work may be needed before these items are used outside healthcare. Another limitation is that we were unable to determine response rates as recruitment methods did not provide a denominator or sampling frame. The diversity of the samples and the use of existing relationships for recruitment were limitations. The study showed that it is feasible to create item banks assessing determinants of implementing evidence-based practices. Future studies should assess the psychometric properties of the item banks and develop measures of additional determinants.

Supplemental Material

sj-docx-1-irp-10.1177_26334895231175527 - Supplemental material for Content validity of an item bank to assess intervention characteristic determinants of implementing evidence-based practices

Supplemental material, sj-docx-1-irp-10.1177_26334895231175527 for Content validity of an item bank to assess intervention characteristic determinants of implementing evidence-based practices by Salene M.W. Jones, Aditya Shrey and Bryan J. Weiner in Implementation Research and Practice

Supplemental Material

sj-docx-2-irp-10.1177_26334895231175527 - Supplemental material for Content validity of an item bank to assess intervention characteristic determinants of implementing evidence-based practices

Supplemental material, sj-docx-2-irp-10.1177_26334895231175527 for Content validity of an item bank to assess intervention characteristic determinants of implementing evidence-based practices by Salene M.W. Jones, Aditya Shrey and Bryan J. Weiner in Implementation Research and Practice

Supplemental Material

sj-docx-3-irp-10.1177_26334895231175527 - Supplemental material for Content validity of an item bank to assess intervention characteristic determinants of implementing evidence-based practices

Supplemental material, sj-docx-3-irp-10.1177_26334895231175527 for Content validity of an item bank to assess intervention characteristic determinants of implementing evidence-based practices by Salene M.W. Jones, Aditya Shrey and Bryan J. Weiner in Implementation Research and Practice

Footnotes

Acknowledgments

This work was funded by a supplement to grant P50CA244432 from the National Cancer Institute.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Salene M.W. Jones

Supplemental Material

Supplemental material for this article is available online.

References

Baker

Camosso-Stefinovic

Gillies

Shaw

E. J.

Cheater

Flottorp

Robertson

(2010). Tailored interventions to overcome identified barriers to change: Effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 3, CD005470. https://doi.org/10.1002/14651858.CD005470.pub2

Chaudoir

S. R.

Dugan

A. G.

Barr

C. H.

(2013). Measuring factors affecting implementation of health innovations: A systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation Science, 8(1), 22. https://doi.org/10.1186/1748-5908-8-22

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 50. https://doi.org/10.1186/1748-5908-4-50

Elo

Kyngas

(2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107–115. https://doi.org/10.1111/j.1365-2648.2007.04569.x

Geerligs

Rankin

N. M.

Shepherd

H. L.

Butow

(2018). Hospital-based interventions: A systematic review of staff-reported barriers and facilitators to implementation processes. Implementation Science, 13(1), 36. https://doi.org/10.1186/s13012-018-0726-9

Kajermo

K. N.

Bostrom

A. M.

Thompson

D. S.

Hutchinson

A. M.

Estabrooks

C. A.

Wallin

(2010). The BARRIERS scale – the barriers to research utilization scale: A systematic review. Implementation Science, 5(1), 32. https://doi.org/10.1186/1748-5908-5-32

Koppelaar

Knibbe

J. J.

Miedema

H. S.

Burdorf

(2009). Determinants of implementation of primary preventive interventions on patient handling in healthcare: A systematic review. Occupational and Environmental Medicine, 66(6), 353–360. https://doi.org/10.1136/oem.2008.042481

Lawrence

Ramsey

Baumann

Proctor

(2016). Dissemination & implementation barriers & facilitators. In Eight toolkits related to dissemination and implementation. Washington University. https://sites.wustl.edu/wudandi

Lewis

C. C.

Mettert

Lyon

A. R.

(2021). Determining the influence of intervention characteristics on implementation success requires reliable and valid measures: Results from a systematic review. Implementation Research and Practice, 2, 2633489521994197. https://doi.org/10.1177/2633489521994197

10.

Martinez

R. G.

Lewis

C. C.

Weiner

B. J.

(2014). Instrumentation issues in implementation science. Implementation Science, 9(1), 118. https://doi.org/10.1186/s13012-014-0118-8

11.

Miake-Lye

I. M.

Delevan

D. M.

Ganz

D. A.

Mittman

B. S.

Finley

E. P.

(2020). Unpacking organizational readiness for change: An updated systematic review and content analysis of assessments. BMC Health Services Research, 20(1), 106. https://doi.org/10.1186/s12913-020-4926-z

12.

Polit

D. F.

Beck

C. T.

Owen

S. V.

(2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199

13.

Reise

S. P.

Haviland

M. G.

(2005). Item response theory and the measurement of clinical change. Journal of Personality Assessment, 84(3), 228–238. https://doi.org/10.1207/s15327752jpa8403_02

14.

Reise

S. P.

Revicki

(Eds.). (2015). Handbook of item response theory modeling. Routledge.

15.

Rogers

E. M.

(2003). Diffusion of innovations (5th ed.). Free Press.

16.

Streiner

D. L.

Norman

G. R.

(2015). Health measurement scales: A practical guide to their development and use (5th ed.). Oxford University Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.02 MB

0.01 MB