Sage Journals: Discover world-class research

Abstract

Objectives:

Uptake of new knowledge by diverse and diffuse stakeholders of health-care technology innovations has been a persistent challenge, as has been measurement of this uptake. This article describes the development of the Level of Knowledge Use Survey instrument, a web-based measure of self-reported knowledge use.

Methods:

The Level of Knowledge Use Survey instrument was developed in the context of assessing effectiveness of knowledge communication strategies in rehabilitation technology. It was validated on samples representing five stakeholder types: researchers, manufacturers, clinician–practitioners, knowledge brokers, and consumers. Its structure is broadly based on Rogers’ stages of innovation adoption. Its item generation was initially guided by Hall et al’s Levels of Use framework. Item selection was based on content validity indices computed from expert ratings (n₁ = 4; n₂ = 3). Five representative stakeholders established usability of the web version. The version included 47 items (content validity index for individual items >0.78; content validity index for a scale or set of items >0.90) in self-reporting format. Psychometrics were then established for the version.

Results:

Analyses of data from small (n = 69) and large (n = 215) samples using the Level of Knowledge Use Survey instrument suggested a conceptual model of four levels of knowledge use—Non-awareness, Awareness, Interest, and Use. The levels covered eight dimensions and six user action categories. The sequential nature of levels was inconclusive due to low cell frequencies. The Level of Knowledge Use Survey instrument showed adequate content validity (≈ 0.88; n = 3) and excellent test–retest reliability (1.0; n = 69). It also demonstrated good construct validity (n = 215) for differentiating among new knowledge outputs (p < 0.001) and among stakeholder types (0.001 < p ≤ 0.013). It showed strong responsiveness to change between baseline and follow-up testing (0.001 < p ≤ 0.002; n = 215).

Conclusion:

The Level of Knowledge Use Survey instrument is valid and reliable for measuring uptake of innovations across diffuse stakeholders of rehabilitation technologies and therefore also for tracking changes in knowledge use.

Keywords

Knowledge translation knowledge use research evaluation rehabilitation technology health and function

Introduction

Scholars and government agencies operating in applied fields, such as rehabilitation and health care in general, are increasingly expected to demonstrate that new knowledge generated through sponsored research and development (R&D) projects has value outside the academic system, in compliance with the Government Performance and Results Act.^1–3 They are tasked with delivering evidence of new knowledge implementation by various stakeholder types such as practitioners/clinicians, designers/manufacturers, lay consumers, knowledge brokers and policy makers.^4,5 Knowledge brokers play a unique role. They facilitate knowledge use by other stakeholders by showing how applying the knowledge benefits consumers. Collectively, this evidence would demonstrate a link between scholarly generation of new knowledge and beneficial stakeholder outcomes and socio-economic impacts from R&D projects as justification for continued public investment.^6,7

The traditional communication strategies of passive diffusion or active dissemination work well for a closed network of peer scholars. They are less effective for other stakeholders who lack access to the materials and who may lack the specialized training to interpret the findings. Knowledge translation (KT) offers strategies to help scholars communicate new knowledge outputs from R&D projects to nontraditional stakeholders. Stakeholders’ use of new knowledge is increasingly recognized as a short-term indicator of long-term social benefits, both in the health-care and social services; in these arenas, KT is seen as a mechanism for increasing beneficiary impacts.^6,8–13

The strategies underlying KT include targeting recipients, tailoring material to their values and contexts, and employing multiple media. These KT approaches may indeed enhance communication and increase application. However, a persistent challenge to R&D investigators is measuring and tracking the extent to which these diverse and diffuse stakeholders engage with new knowledge outputs from their projects. The academic citation tracking system works for scholars who reference the original author in subsequent publications; it is irrelevant to stakeholders who implement new knowledge through practices and policies instead of publications.

To date, parallel systems designed to neatly track knowledge use by nonscholars do not exist. For example, many instruments that offer face validity for this task tend to favor data collection through observation or self-report on knowledge adoption and application within defined organizations and where implementation was pre-determined by management.^14–16 These instruments offer limited utility for capturing data across the various levels of knowledge use under conditions where individuals freely encounter new knowledge and independently consider adoption and application.

Current studies on measuring the so-called research impact offer limited guidance for constructing valid metrics of knowledge use by stakeholders in naturalistic settings for three reasons. First, they offer widely divergent frameworks and metrics for defining and assessing what they consider research impact.^17–20 They neglect to focus on short-term indicators of long-term impact.²¹ Notably, these studies neglect to consider stakeholder use of new knowledge as an indicator of impact, and they fail to recognize that instrumental, conceptual, and strategic/symbolic applications are distinct forms of knowledge use.^22,23

Second, the prevalent bibliometric indicators on publications and patents really represent interim assessments of knowledge output quality by journal reviews and patent exams, not by stakeholder audiences.

Third, these studies do not recognize the progression of knowledge as intellectual property, which involves three steps: (1) KT, which entails communication of conceptual knowledge; (2) technology transfer, which refers to the conveyance of ownership over prototype inventions; and (3) commercial transactions, including the exchange of goods and services between manufacturers and consumers (commercial transactions).²⁴

This progression is particularly important to technology-oriented applied R&D activities. Existing metrics focus on stakeholder use of “conceptual discovery” outputs from scientific research methods. However, a more comprehensive tool would also address new knowledge in the states of prototype invention outputs from engineering development and finished goods from industrial production (pp. 3–4).²⁵ Instruments capable of measuring stakeholder engagement with technology-based knowledge are particularly important when KT strategies are applied in support of these downstream technology transfer and commercialization outcomes.

The purpose of this article is to describe the development of an instrument that can (1) identify the extent to which stakeholders of technology-based new knowledge are aware of the knowledge, are making use of the knowledge in practice, or are interested in doing so and (2) inform KT and technology transfer activities undertaken by the R&D projects that generate technology-based knowledge. The opportunity to develop the instrument came from a randomized controlled intervention study at the Center on Knowledge Translation for Technology Transfer (KT4TT).²⁶ This center was engaged in comparing different approaches to effectively communicate new knowledge in rehabilitation technology to members of various stakeholder types. In order to achieve this objective, the study needed to measure stakeholder use of knowledge as evidence of intervention strategy effectiveness. The intervention study defined new knowledge as findings published in peer-reviewed journal publications, as this is the customary channel through which scholars communicate new knowledge. The design and procedures of the intervention study are detailed elsewhere.^26,27 For purposes of this article, they are summarized in Appendix 1.

The intervention study determined that the instrument should be capable of measuring knowledge use by study participants who (1) may or may not ever engage the new knowledge or (2) may engage it but neither express interest nor initiate application. A further requirement was that the instrument be capable of addressing published findings from R&D in any rehabilitation technology field. Finding no such ready instrument, the study team designed, constructed, and tested the Level of Knowledge Use Survey (LOKUS) instrument, which is the focus of this article. We clarify that the role of the instrument was to measure knowledge use by stakeholders, whereas the purpose of the intervention study was to evaluate knowledge communication strategies based on the measure. The remainder of this article presents the conceptual basis for the LOKUS instrument and describes its development including design, construction, validation, and establishment of psychometrics.

Method overview

Instrument conceptualization

As the development of the LOKUS instrument was propelled by the aforementioned intervention study, a literature review was conducted to inform both the intervention design and instrument development. The search for appropriate measures of knowledge use was part of a broader review that addressed the design of the knowledge communication strategies to be evaluated for effectiveness. Both classic database searches (Medline and PsycINFO) and simple, basic searches were conducted using the University at Buffalo’s library system. The search covered journal articles, books, conference papers, and other scholarly work including website postings. Search terms included varied combinations of knowledge, innovation, research, and evidence with impact, utilization, communication, dissemination, and diffusion. Examples are research utilization, research impact, and knowledge utilization and diffusion. Cross-cutting terms measure, tool, and instrument were used to narrow the search. Work older than the past six decades was excluded, considering the recency of the emergent KT concept. Targeted content areas included both health care (biomedical literature) and social sciences (education, communication). These were areas thought to be most relevant to the field of rehabilitation addressed by the intervention study. Also, they house the most frequently cited work related to KT and likely bear footprints of the KT process evolution. The search was further cross-checked against additional sources through professional networks, leading to useful organizational and conference websites among them the Canadian Institutes for Health Research (CIHR) and the National Center for the Dissemination of Disability Research (NCDDR).^9,12

Among the key results of the search was a scholarly literature review of KT models, methods, and metrics by Sudsawad.¹³ As characterized by the author,

This literature review, although not intended to be an in-depth or systematic review of any one aspect of knowledge translation, is designed to bring together several aspects of it from selected literature … The body of work included in this review was selected from frequently cited and thought-provoking literature and represents a variety of thoughts and approaches that are applicable to knowledge translation. (p. 1)

Sudsawad’s paper offers a comprehensive treatment of KT, which includes definitions, models, strategies, and their effectiveness based on literature in health-care fields including rehabilitation.

Sudsawad’s review offered a rich repository for initiating an in-depth search of relevant measures of knowledge use. The researchers used a snowball approach, by identifying relevant content and citations in the review and then branching out repeatedly to locate additional relevant papers. Promising measures or conceptual frameworks were next identified and obtained either through published sources or by direct contact with publisher. The criteria for the selection of the measure (or measurement framework) included its (1) merit and (2) worth, the two important standards recommended for any evaluative work.^28,29 Merit in this context referred to the measure’s quality or its psychometric soundness, including its validity and reliability. Worth or relevance implied that the measure should demonstrate practical utility to sponsors and investigators of R&D projects that generate and publish new knowledge in technology-related fields. From this standpoint, measures defined and linked closely to the process of innovation production and communication (KT) would be most meaningful to R&D investigators than isolated studies of knowledge use in a general sense. In this sense, Rogers’³⁰ five-stage model of innovation decision articulated with the innovation diffusion process was found particularly relevant to our context. The search then prioritized in-depth reviews of studies based on this model.

Two measurement models were found to be especially applicable. They were (1) Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework^14,15 and (2) the Levels of Use (LoU) scale.^16,31,32

The RE-AIM model is valuable for its inclusion of the Reach stage. That stage measures the percent and representativeness of individuals who are willing to use the knowledge. Thus, the model recognizes a key challenge that proponents of KT face: To reach stakeholders who are widely dispersed across sectors and organizations and to raise their awareness and interest before effectively engaging them in implementation and adoption.

Once the knowledge reaches the stakeholders, the next challenge is to capture knowledge use by stakeholders. Hall’s LoU framework seemed to offer an opportune way to do so. This framework proposes seven levels of use: Non-use, Orientation, Preparation, Mechanical use, Routine Use and Refinement, Integration, and Renewal. Each level is defined by expanding it into seven categories of specific behaviors. Potentially, the levels offer a basis by which to profile the stakeholder populations that are reached. The categories in each level can further inform investigators of the specific activity status of the stakeholders.

However, the operational model of the LoU instrument was a limiting factor in its applicability to our context. It required extensive qualitative interviewing of individuals in order to place them on the levels scale by probing for activities within the proposed categories. Despite its fit in purpose, the LoU model seemed unfeasible for soliciting responses from stakeholders scattered across sectors and organizations. We therefore opted to use the LoU framework of levels and categories as a foundation for a new instrument that we then systematically modified through empirical validation in the context of new knowledge communication (published findings) in rehabilitation technology. Further details are reported in the full report.^27,33

The new instrument was envisioned to help R&D sponsors, and investigators address their accountability requirements in two steps. First, they could survey stakeholders for self-reported levels of use of published findings to construct an overview of the reach and uptake status among diverse stakeholders. They could then identify specific stakeholders to interview in-depth or to continue documenting changes in use over time. To meet these requirements, the new instrument was designed as a survey questionnaire formatted to elicit self-reported responses, and to be administered online to broadly reach diverse stakeholder audiences.

Procedures

The LOKUS instrument was developed in two major phases. The first phase developed and validated the instrument and the second phase established its psychometrics. Both phases occurred in the context of the earlier mentioned intervention study in rehabilitation technology. As the first area addressed by the study was Augmentative and Alternative Communication (AAC) technology, the instrument content referred to published findings selected in the AAC technology area. The study participants consisted of stakeholders who had a potential interest in these published findings or had expertise in AAC technology.

Figure 1 outlines the flow of steps involved in developing the LOKUS instrument with the two phases occurring in sequence. We describe below methods and results corresponding to the two phases.

Figure 1.

Description of the development of the LOKUS instrument in the form of a flowchart, where the sequential activities summarize the creation, testing, revision, and organization of items in two parallel versions as well as activities that established the instrument’s psychometrics and the conceptual model.

Phase 1: instrument development and validation

This phase addressed instrument design, construction, and validation, which involved (1) item generation and validation; (2) questionnaire organization and improvement; and (3) pilot testing for context validation and model configuration.

Method

This section first describes the samples used in Phase 1 and then the corresponding data collection procedures.

Participants

The item development process included two successive rounds of item validity testing. Four researchers with expertise in KT (n = 4) participated in the first round, and rated 57 items for relevance and uniqueness. Items were included or excluded based on relevance. Uniqueness was used to revise items by rewording or merging with other items. A second set of three (n = 3) scholars with expertise in transferring rehabilitation technology rated the revised set of items. The resulting questionnaire contained 47 items. Details are presented in the next section.

Next, field testing data for evaluating and improving the questionnaire came from participants as below. Participants were recruited after obtaining approval by the University at Buffalo’s Institutional Review Board on meeting ethical guidelines for human subject participation. Written prior consent was duly obtained by each participant.

One-on-one testing was used to establish the instrument’s usability. Five participants (n = 5) who represented five stakeholder types relevant to the published findings pilot tested the instrument for content comprehension, meaningfulness, and accessibility. The five stakeholder types were (1) AAC clinical therapist, (2) knowledge broker related to disability service, (3) clinical researcher, (4) product manufacturer, and (5) lay consumer of AAC technology.

For pilot testing the instrument, participant responses in the earlier mentioned intervention study were used as the data source. This was the first application of the instrument in a large sample (n = 215) representing the five stakeholder types. Participants were recruited from a national pool through professional organizations in the following numbers: AAC clinical therapists (n = 45), knowledge brokers in disability services (n = 65), clinical researchers (n = 29), product manufacturers (n = 26), and lay consumers with communication disabilities (n = 50). All participants gave written consent before starting the study.

Participants answered the LOKUS instrument three times—once at baseline, a second time after a 4-month follow-up period, and then again after another 4-month follow-up period. The study lasted 8 months (see Appendix 1, Table 6). Participants were presented with findings excerpted from peer-reviewed journal articles published in the AAC technology area. As described in the next section, the respondents’ data served for preliminary configuration of the conceptual model for the LOKUS instrument. It also helped establish its psychometric properties as described under Phase 2. Only data at baseline and at the first follow-up were used for the pilot testing analyses. Analysis of all other data including follow-up 2 was more pertinent to the intervention study and the results are reported elsewhere.²⁶

Data collection and analysis procedures

This section summarizes the methods used for item generation and content validity testing, followed by the organization, improvement, field testing, and validation of the instrument. A complete report of methods is provided elsewhere.²⁷

Items to be included in the LOKUS instrument were queries centered on the published findings in the AAC technology area. For item creation, the LoU framework was examined for transferability to the context in rehabilitation technology for measuring knowledge use. The 7 × 7 matrix, which refers to 7 levels of use with 7 categories under each, was modified to a new matrix: 9 levels and 7 categories. We populated the cells of the new matrix with relevant behaviors, selecting cells as applicable. In all, 57 items were generated to comprise a web-based survey questionnaire. The items were formatted for self-reporting through multiple-choice or check-off responses.

Next, four experts in KT rated each item’s content validity for relevance and uniqueness. They did so by examining the corresponding item descriptions against the levels and categories, and rating each item on a five-point Likert scale (from strongly agree to strongly disagree). They also provided explanatory comments. These ratings were dichotomized during analysis, by assigning a score of 1 for strongly disagree/disagree, 2 for neither agree nor disagree, and 3 for agree/strongly agree. Thus, a rating score of 3 meant approval of the item on relevance (or uniqueness as the case may be), whereas a rating score of 1 meant no approval of the item on relevance (or uniqueness).

Data were analyzed by computing three types of indices: overall ratings, content validity index for individual items (I-CVI), and content validity index for a scale or set of items (S-CVI/Ave), separately on relevance and on uniqueness.^34,35 The corresponding Kappa coefficients were also computed. I-CVI is an established agreement method for content validity of individual items, computed as the number of experts giving the highest rating divided by the total number of experts. The S-CVI is the proportion of items given the highest rating by the raters.³⁴ The second round of three experts evaluated the revised list of items for final approval.

Relevance ratings determined item inclusion, using the recommended standards of I-CVI values of 0.78 or higher for items and S-CVI/Ave values of 0.90 or higher for scales.^35–37 Items with I-CVI values lower than 0.78 on uniqueness were revised using rater comments. The process resulted in a refined 10 × 6 framework and 37 items distributed under them, in addition to 10 items that measured levels as shown in Table 1. They were then organized into a draft instrument suitable for online administration to respondents. A logical branching sequence was used for administration of items, where each respondent was first required to report his or her status regarding a specific level of new knowledge use. Based on the response chosen, the respondents would automatically be routed to the corresponding page of queries on categories (i.e. behaviors subordinate to that level). Three open-ended questions invited participant’s comments and were placed at the end of the items pertaining to each published finding presented. Two more general questions asked about preferred knowledge communication strategies and were placed at the very end of the questionnaire.

Table 1.

Initial structure of the LOKUS instrument: item distribution over a 10 x 6 layout.

Levels	Related Q #	CATEGORIES
		Being Aware	Getting Information	Sharing	Assessing	Planning	Implementing
Non-awareness	Description: User had not heard of the new knowledge from the study until present time (Vs. User had heard of the new knowledge from the study before.)
	Q2^a	N/A^b	N/A	N/A	N/A	N/A	N/A
Awareness	Description: User had heard of the new knowledge from the study but has not tried to get more information about it.
	Q3	N/A	N/A	N/A	N/A	N/A	N/A
Orientation	Description: User is seeking details on whether the new knowledge from the study will be useful; however has not yet decided to use it.
	Q4	Q5a	Q5b	Q5c	Q5d	N/A	Q5e
Preparation	Description: User is preparing to use the new knowledge from the study; but has not used it yet.
	Q6	Q7a	Q7b	Q7c	N/A	Q7d	Q7e
Initial use	Description: User has just begun to use the new knowledge from the study, but has not yet mastered how to use it.
	Q8	N/A	Q9a	N/A	Q9b	N/A	Q9c
Routine use	Description: User is using the new knowledge from the study regularly and does so with ease. However, User has not tried using it in ways other than originally intended.
	Q10	Q11a	N/A	N/A	Q11b	Q11c	Q11d
Expansion	Description: Based on own evaluations, User is using the new knowledge from the study in ways different from originally intended.
	Q12	Q13a	Q13b	Q13c	Q13d	Q13e	Q13f
Collaboration	Description: User is either considering collaborating with others, or has started to do so, on the use of the new knowledge from the study.
	Q14	Q14a	Q14b	Q14c	Q14d	Q14e	Q14f
Integration	Description: User’s collaboration with others has led to a different way in which they use the new knowledge from the study.
	Q16	Q17a	Q17b	N/A	Q17c	N/A	Q17d
Modification	Description: User has made modifications to the new knowledge from the study, individually or jointly with others.
	Q18	Q18a	Q18b	N/A	Q18c	N/A	Q18d

LOKUS: Level of Knowledge Use Survey; Q: Question; N/A: Not applicable.

Q2 to Q18d refer to item numbers.

N/A indicates an irrelevant category hence no item.

Two parallel versions of the LOKUS instrument—one for stakeholders who were lay consumers and the other for stakeholders identified as professionals—were uploaded to a web-based platform within the Vovici software.³⁸ Both versions of the instrument were tested for usability with five individuals, representing each of the five stakeholder types described earlier. Revisions were made to the questionnaire and its Vovici interface, bringing it to a ready-to-use format for use in the intervention study as intended. The foregoing process of initial development of the LOKUS instrument took 18 months. The first administration of the LOKUS instrument to a large sample of representative stakeholders in the intervention study (n = 215) served as its pilot test and its validation in that context. This pilot version introduced the three published findings (Publications A, B, and C) drawn from the AAC technology area. Only Publication A was the subject of intervention (i.e. communicated through chosen strategies), while the other two were presented as distracters. They served to prevent respondents from recognizing that Publication A was the focus and therefore giving responses biased in its favor (i.e. interested in its findings or has used its findings). The three publications were introduced consecutively within the LOKUS instrument presenting the same questions for each publication in turn.

Analyses of data from the pilot test suggested a new instrument structure as shown in Table 2. It consisted of four levels—Non-awareness, Awareness, Interest, and Use; eight dimensions—Preparation, Orientation, Initial Use, Routine Use, Expansion, Collaboration, Integration, and Modification; and 37 user activities.

Table 2.

Revised structure of the LOKUS instrument: item distribution under levels, dimensions, and activities.

Levels	Dimensions	Related Q #	CATEGORIES / ACTIVITIES
			Being Aware	Getting Information	Sharing	Assessing	Planning	Implementing
1 Non-awareness	_	Q2^a	Description: User had not heard of the new knowledge from the study until present time (vs User had heard of the new knowledge from the study before).
1 Non-awareness			N/A^b	N/A	N/A	N/A	N/A	N/A
2 Awareness	_	Q3	Description : User had heard of the new knowledge from the study but has not tried to get more information about it.
2 Awareness			N/A	N/A	N/A	N/A	N/A	N/A
3 Interest	Orientation	Q4	Description: User is seeking details on whether the new knowledge from the study will be useful; however has not yet decided to use it.
			Q5a	Q5b	Q5c	Q5d	N/A	Q5e
	Preparation	Q6	Description: User is preparing to use the new knowledge from the study; but has not used it yet.
			Q7a	Q7b	Q7c	N/A	Q7d	Q7e
4a.Intended Use	Initial use	Q8	Description: User has just begun to use the new knowledge from the study, but has not yet mastered how to use it.
			N/A	Q9a	N/A	Q9b	N/A	Q9c
	Routine use	Q10	Description: User is using the new knowledge from the study regularly and does so with ease. However, User has not tried using it in ways other than originally intended.
			Q11a	N/A	N/A	Q11b	Q11c	Q11d
4b. Modified Use	Expansion	Q12	Description: Based on own evaluations, User is using the new knowledge from the study in ways different from originally intended.
			Q13a	Q13b	Q13c	Q13d	Q13e	Q13f
	Collaboration	Q14	Description: User is either considering collaborating with others, or has started to do so, on the use of the new knowledge from the study.
			Q15a	Q15b	Q15c	Q15d	Q15e	Q15f
	Integration	Q16	Description: User’s collaboration with others has led to a different way in which they use the new knowledge from the study.
			Q17a	Q17b	Q17c	Q17d	Q17e	Q17f
	Modification	Q18	Description: User has made modifications to the new knowledge from the study, individually or jointly with others.
			Q19a	Q19b	N/A	Q19c	N/A	Q19d

LOKUS: Level of Knowledge Use Survey; Q: Question; N/A: Not applicable.

Q2 to Q18d refer to item numbers.

N/A indicates an irrelevant category hence no item.

Results

The output from Phase 1 was the LOKUS instrument in its web form, resulting from the development and validation methods described above. This instrument, in its automated web version, is available on the KT4TT Center’s website. Visitors to the website can also find a linear listing of items along with an illustration of item branching used for the web version.³⁹ Summarized next are results related to the instrument’s content validity, usability, and conceptual model. Details are available in the full report.²⁷

Content validity

Testing was conducted for all items in the LOKUS instrument, where 10 items measured the initial 10 levels and the other items measured behaviors across the 6 categories of activities spread under the levels. All level items and 37 of the activity items met the relevance standard (I-CVI > 0.78; S-CVI/Ave > 0.90). They were all included in the final instrument under a new structure of four levels, eight dimensions, and 37 activity items as mentioned earlier.

Usability

All five pilot testers of the online questionnaire found both versions usable and efficient. The average time for completion was recorded at 35 min. This might overestimate the respondents’ time, as it included the additional time they required to evaluate each item in order to provide feedback for any needed improvements.

Reliability

A parallel yet independent analysis involving a convenience sample (n = 69) of therapists and researchers concluded that the instrument was usable and efficient and reported excellent test–retest reliability (1.0).⁴⁰

Conceptual model

The underlying model for the instrument was investigated using stakeholder data (n = 215) from the intervention study (see Table 6 in Appendix 1.) We sought to verify whether the original 10 levels are developmental (sequential), an assumption necessary for confidently placing individuals on a scale ranging from low to high. In his more generic model, Rogers³⁰ pointed out that an individual’s decision to use an innovation does not always flow from the stages of knowledge and persuasion in that order. In the LoU model, Hall et al.³¹ qualify the sequential nature of their LoUs with no guarantee “that an individual will move through all levels in a lock-step developmental fashion” (p. 11). Instead, Hall’s team viewed each LoU as independent. This position was corroborated by three of the four experts (75%) who validated the LOKUS items and agreed on the overall developmental nature of the 10 levels. The subsequent testing by the convenience sample (n = 69) also suggested an overall sequence of the levels, but the conclusion was limited to the first four levels—Non-awareness, Awareness, Orientation, and Preparation—due to low response frequencies within the other levels.⁴⁰

The foregoing reduces the question of sequential nature of levels to the last six levels (i.e. Initial Use, Routine Use, Expansion, Collaboration, Integration, and Modification). In this regard, an analysis of the response pattern of the intervention study participants (n = 215) was revealing. A total of 645 responses came from all participants (i.e. two experimental groups and a control group) who answered the LOKUS instrument at baseline and at two follow-up periods.

Response data between baseline and the first follow-up were examined to see how knowledge use changed among participants over the intervening 4-month interval. Under the hypothesis that levels are developmental, a steady upward movement was expected only of individuals exposed to the experimental approaches in the intervention study but not of those in the control group. However, bi-directional movements between levels were found, with diminishing frequencies at follow-up, suggesting regressive movements through levels. Table 3 shows such regressions. This table tracks the movement of individuals (n = 151) exposed to the two experimental approaches, level by level, between baseline and the first follow-up. As shown by the frequencies to the left of the boxed diagonal cells, about 10 individuals moved backward. The response pattern seemed counterintuitive to the hypothesized developmental nature of levels.

Table 3.

Movement of treatment groups through levels of knowledge use between baseline and follow-up: Publication A.

		Follow-up
		Level
		n		0	1	2	3	4	5	6	7	8	9
Baseline	Level 0(Non-awareness)	131	→	93	9	6	6			1	12	1	3
	Level 1(Awareness)	9	→	3	1		1		4
	Level 2(Orientation)	1	→								1
	Level 3(Preparation)	2	→	1							1
	Level 4(Initial Use)	1	→						1
	Level 5(Routine Use)	0	→
	Level 6(Expansion)	2	→		1								1
	Level 7 (Collaboration)	1	→						1
	Level 8(Integration)	1	→		1
	Level 9(Modification)	3	→	1			2						5
	Total	151	→	98	12	6	9	0	6	1	14	1	4

Responses were then examined by grouping the levels into hypothetical sequences. Grouping decisions were informed by several data sources. First, content validation experts had pointed out that “use” of technology-related knowledge involved transformations in the state of knowledge. They argued that knowledge users might not all take the same path after Preparation; rather, they may opt between using the knowledge as intended (i.e. going through Initial Use and Routine Use) and modifying it before using (i.e. Expanding, Collaborating, Integrating, Modifying). For example, a lay consumer will likely apply new knowledge as it was intended by the creator, being drawn to knowledge in its current state. Meanwhile, a manufacturer might have the confidence or insight to modify the new knowledge prior to implementation. Thus, it is logical to include knowledge use both as intended by the creator and as modified by the recipient, treating these as variations within the same implementation level called “use.”

Data in Table 3 lend some support to the foregoing argument. Of the 131 people in the first row who started out as Non-aware, 93 remained Non-aware 4 months later. Nine reported being in Awareness, six in Orientation, and six in Preparation. No one reported being in Initial Use or Routine Use, but 17 reported being engaged in Modified Use (i.e. Expansion, Collaboration, and Modification). It is plausible that all 17 individuals skipped the Initial and Routine Use levels, assuming that they could not have completed these steps in an interval as short as 4 months. Furthermore, the frequency pattern within Modified Use in this row suggests that Expansion, Collaboration, Integration, and Modification do not form a “lock-step” sequence. The foregoing suggested a new logical sequence of Non-awareness, Awareness, Orientation, and Preparation followed by the remaining levels collapsed into Intended Use or Modified Use, as parallel occurrences.

This provisional sequence was further examined by overlaying the Levels of Knowledge Use on Rogers’ stages of innovation-decision, as well as Hall and colleagues’ LoUs, as shown in Table 4. Note that Rogers points out that in the Implementation stage (see column 3), the innovation may not always be invariant: users may not copy exactly or imitate how it has been used in prior settings. It can be modified by the user. This suggests that use may be either as intended by the knowledge creator or as modified by user. Furthermore, Rogers also points out that in the Confirmation stage (Table 4, column 3), an individual may reverse an already made decision if exposed to conflicting messages about the innovation implementation. Rogers’ position is corroborated by the movement pattern found in Table 3.

Table 4.

Levels of knowledge use corresponding to LoUs of the innovation^a and stages of the innovation-decision process^b.

LoUs of the innovationa	Levels of knowledge use	Stages of the innovation-decision processb
Level 0 Non-Use – State in which the user has little or no knowledge of the innovation, has no involvement with the innovation, and is doing nothing toward becoming involved.	Non-awareness
	Awareness	Knowledge: occurs when an individual (or a decision-making unit) is exposed to the innovation’s existence and gains some understanding of how it functions. Rogers points out that awareness knowledge may imply a passive stage, whereas active knowledge seeking may occur if a need precedes it.
Level I Orientation: State in which the user has acquired or is acquiring information about the innovation and/or has explored or is exploring its value orientation and its demands upon the user and the user system.	Orientation	Persuasion: occurs when an individual (or a decision-making unit) forms a favorable or unfavorable attitude toward the innovation. The individual becomes more psychologically involved with the innovation, and actively seeks information about it.
Level II Preparation: State in which the user is preparing for first use of the innovation.	Preparation	Decision: occurs when an individual (or a decision-making unit) engages in activities that lead to the choice to adopt or reject the innovation. This decision may depend on a prior trial by the individual or trial by others. Also, rejection can occur not only at this stage but potentially at any stage. Active rejection consists of considering the adoption of the innovation (including even its trial) but then deciding not to adopt it. Passive decision (also called non-adoption) consists of never really considering use of the innovation. Thus, the three stages of knowledge, persuasion and decision cannot be assumed to always occur as a linear sequence in that order.
Level III Mechanical use: State in which the user focuses most effort on the short-term, day-to-day use of the innovation with little time for reflection. Changes in use are made more to meet user needs than client needs. The user is primarily engaged in a stepwise attempt to master the tasks required to use the innovation, often resulting in disjointed and superficial use.	Initial use	Implementation: occurs when an individual (or a decision-making unit) puts an innovation into use. A certain degree of uncertainty about the expected consequences of the innovation still exists for the individual at this stage; so active information seeking usually takes place.
Level IVA Routine: State in which the user of the innovation is stabilized. Few if any changes are being made in ongoing use. Little preparation or thought is being given to improving innovation use or its consequences.	Routine use	Implementation (continued): Rotinization or institutionalization marks the end of this stage when the new idea becomes a regularized part of the adopter’s ongoing operations.
Level IVB Refinement: State in which the user varies the use of the innovation to increase impact on clients within immediate sphere of influence. Variations are based on knowledge of both short- and long-term consequences for clients.	Expansion Collaboration Integration Modification	Implementation (continued): Rogers points out that the innovation may not always be invariant, that is, copied exactly or imitated how it has been used in prior settings. Re-Invention, according to Rogers, is the degree to which an innovation is changed or modified the user in the process of its adoption and implementation, which is distinct from reinterpretation where the adopter uses the innovation in a different way and/or for different purposes.
Level V Integration: State in which the user is combining own efforts to use the innovation with the related activities of colleagues to achieve a collective impact on clients within their common sphere of influence.
Level VI Renewal: State in which the user re-evaluates the quality of use of the innovation, seeks major modifications or alternatives to the present innovation to achieve increased impact on clients, examines new developments in the field, and explores new goals for self and the system.		Confirmation: occurs when an individual (or a decision-making unit) seeks reinforcement for the innovation-decision already made, but may reverse this decision if exposed to conflicting messages about the in- novation. A replacement discontinuance may occur in order to adopt a better idea; a disenchantment discontinuance may occur as a result of dissatisfaction with the innovation’s performance.

LoUs: Levels of Use.

Rogers.³⁰ The table contains our summary interpretation

Across the three columns of Table 4, note that Rogers’ stages of innovation-decision correspond to levels in the LOKUS instrument and in the LoU framework. Rogers’ Persuasion stage seems to correspond to the Orientation level in the other two models. Notably, Decision is a distinct stage in Rogers’ model, but it appears between levels in the LoU framework. In the case of the LOKUS instrument, one could argue that Decision may be ongoing at the Preparation level before the user selects alternate paths of use. This was supported by discussions with the experts in technological adoption who had participated in content validation. Therefore, the levels Preparation and Orientation in the LOKUS instrument were combined into a broader level called Interest, a stage where the user has yet to decide about actual use.

As a result, the 10 levels in the original framework of the LOKUS instrument were collapsed into 4 new levels, with some of the other original levels absorbed as dimensions.

The revised model configuration for the LOKUS instrument resulting from the foregoing considerations is shown in Figure 2. The model consists of.

Figure 2.

Graphic representation of the conceptual model of the LOKUS instrument that shows inter-relationships between its components: levels, dimensions, and user actions.

Four levels of knowledge use:

L1: Non-awareness, L2: Awareness, L3: Interest, and L4: Use, with two sub-levels: L4a: Intended Use and L4b: Modified Use.

Eight of the original levels, absorbed as “dimensions” under the levels Interest and Use:

D1: Orientation and D2: Preparation (located within L3: Interest), D3: Initial Use and D4: Routine Use (located within L4a: Intended Use), D5: Expansion, D6: Collaboration, D7: Integration, and D8: Modification (within L4b: Modified Use).

Likewise, categories under the original levels recognized as user actions/activities under the above eight dimensions, and labeled as below.

B: Being Aware, G: Getting Information, S: Sharing, A: Assessing, P: Planning, and I: Implementing. The activities only appear under the appropriate Levels or Dimensions.

Phase 2: establishing psychometrics for the LOKUS instrument

Method

Phase 2 established the psychometrics for the new instrument resulting from the earlier phase. We reiterate that it consisted of 47 items distributed under four levels: (1) Non-awareness, (2) Awareness, (3) Interest, and (4) Use. Two items measured Non-awareness and Awareness; eight items measured the eight dimensions under Interest and Use. The dimensions themselves were further measured by the 37 activity items. This section describes the samples used and the corresponding data collection procedures including its content validity, construct validity, responsiveness to change, and its conceptual model.

Participants

Content validity was established through a new round of judgment by three experts in AAC technology. Two subsequent participant samples helped establish the instrument’s test–retest reliability, its construct validity in differentiating the published findings and stakeholder types, and its responsiveness to change. The first one was a convenience sample (n = 69) representing clinicians and researchers with relevant expertise who participated in an independent investigation of the instrument’s psychometrics.⁴⁰ The second one was the larger sample of participants in the intervention study (n = 215), composed of the five stakeholder types as described earlier.

Data collection and analysis

Described below are the procedures to investigate the psychometric properties of the new instrument.

Content validity

This was established by considering items under the eight dimensions and the four levels separately. Table 2 presents the definitions of the eight dimensions. As in Phase 1, content validity was established separately for relevance and uniqueness. Overall rating, I-CVI and S-CVI were computed based on ratings from three experts in AAC technology, who used a three-point Likert scale.^34,35 An I-CVI of 0.80 or higher was considered highly relevant and 1.00 as excellent for three raters.^34,41 The S-CVI for the eight dimensions taken together was computed as the proportion of items given a rating of quite/very relevant by two raters and an index of 0.80 or higher was considered acceptable.^35–37,42 These indices were averaged for raters one versus two, raters one versus three, and raters two versus three for all dimensions.

The items under levels Non-awareness and Awareness were assessed independently, using the same scheme used for the dimensions. For Interest and Use levels, the respective dimensions under each level were considered, and their content validity indices calculated as above were averaged to reflect the content validity of that level.

Construct validity and responsiveness to change

These two were established by testing the following hypotheses, respectively:

Hypothesis 1: The LOKUS instrument differentiates baseline levels of knowledge use among the given examples of published findings within a chosen technology area, and among different stakeholder types. Under this hypothesis, the instrument was expected to detect differences in stakeholder engagement with new knowledge because its relevance varied in the following ways: (1) between the published findings, because of varying participant interest (individual circumstances); (2) across stakeholder types because of varying personal or professional roles; and (3) between individuals within each stakeholder type due to varying levels of knowledge use at the study’s outset.

Hypothesis 2: The LOKUS instrument has good responsiveness to change in level of knowledge use over a time frame of 4 months. Under this hypothesis, the instrument was expected to detect changes in the knowledge use levels of respondents in the two groups that were exposed to an intervention.

Both hypotheses were tested by analyzing responses to the LOKUS instrument by all participants in the intervention study (n = 215). As mentioned earlier, these analyses only used participant responses at baseline and at follow-up 1 (see Appendix 1, Table 6) for purposes of evaluating the LOKUS instrument. Results from full analyses including follow-up 2 data from the intervention study are reported in Stone et al.²⁶ For construct validity, baseline levels of knowledge use were compared across all three examples of published findings, as well as among the five stakeholder types, using Chi-square test for analysis. Responsiveness to change was established by verifying whether participants who received an intervention (i.e. exposed to the two experimental approaches) during the 4-month period advanced from Non-awareness level to a higher level of knowledge use, beyond any change occurring due to testing effect (detectable by changes in the control group). Analyses were limited to Wilcoxon Signed-Rank Tests for lack of statistics that compare two related variables with more than two levels for a contingency table.

The developmental nature of levels

The instrument was revisited to verify the inherent logic between and within the four levels of knowledge use in the revised conceptual model, through continued analyses of responses from the study participants exposed to the two experimental approaches.

Results

This section summarizes results from analyses that refer to the instrument’s content validity, construct validity, and responsiveness to change.

Content validity

One of the 47 items in the instrument measured Non-awareness. One item measured Awareness. Eight items measured the eight dimensions within the Interest and Use levels. Analyses on content validity of the eight dimensions found the items good on relevance and uniqueness, with the three content validity indices ranging from 0.92 to 0.97. On relevance, the mean overall rating, the I-CVI, and the S-CVI were respectively 0.97, 0.92, and 0.75. On uniqueness, these indices were 0.92, 0.94, and 0.92, respectively. Item descriptions for Preparation, Collaboration, and Modification were examined and modified using experts’ comments. In sum, content validity was good except for one index (0.75), which was low due to one rater’s disagreement with the other two.

Analyses of content validity of the level items showed the overall rating for relevance (0.96) was very good, and I-CVI and S-CVI were adequate (0.83) to good (0.89). On uniqueness, they scored lower. The overall rating and I-CVI were good (0.90) or adequate (0.82), but S-CVI was not acceptable, especially for Non-awareness (0.70).

Overall, individual items for levels and dimensions of LOKUS instrument had adequate to very good content validity (0.73–0.97).

Construct validity

The data came from the 215 participants in the intervention study as mentioned earlier. They had been randomly assigned to the three groups. Groups T1 and T2 were exposed to experimental communication approaches. The Control group was not exposed to any intervention (see Table 6 in Appendix 1). Participants were equivalent on all included variables: age (F = 1.344; p = 0.263), gender (χ² = 2.645; p = 0.266), race (χ² = 14.063; p = 0.297), education (χ² = 6.317; p = 0.788), work status (χ² = 1.647; p = 0.949), and years of experience (F = 0.867; p = 0.422) in AAC technology.

The LOKUS instrument queried all three groups on three published findings A, B, and C (see Appendix 1). Participants differed significantly in their baseline levels of knowledge use among the three published findings presented to them (χ² = 27.177, p < 0.001). The majority of participants were at the Non-awareness level despite the fact that all three findings had previously been published in peer-reviewed journals within AAC technology. Publication A was least known (86.5% non-aware), followed by Publication B (79.1% non-aware). Publication C was most known (67.9% non-aware). Publication A was least in Use (7.0%). Publication C was most in Use (22.3%).

Baseline levels of knowledge use also differed significantly among stakeholder types across the three published findings:

Publication A (χ² = 53.237, p < 0.001): Researchers (59.1%) differed from knowledge brokers (98.5%) on Non-awareness. Knowledge brokers (0%) differed from manufacturers (18.5%) on Use.

Publication B (χ² = 25.420, p = 0.013): Researchers (54.5%) differed from knowledge brokers (83.8%) on Non-awareness; manufacturers (3.7%) differed from researchers (36.4%) on Use.

Publication C (χ² = 75.768, p < 0.001): Researchers (31.8%) differed from knowledge brokers (94.1%) on Non-awareness; knowledge brokers (1.5%) differed from manufacturers (51.9%) on Use.

Strong construct validity was therefore established for the LOKUS instrument in two ways. It discerned levels of use among three different cases of published new knowledge within a single technology area. It also discerned differences in levels of use among different stakeholder types.

Responsiveness

Responses of the above 215 participants about findings in Publication A (which was the object of intervention through the communication approaches) were then compared with their responses about findings in Publications B and C (which were not objects of intervention). Changes in level of knowledge use over 4 months between baseline and first follow-up were compared.

Reported advances in level were expected only for Publication A and only for participants in the two treatment groups, T₁ and T₂. Results indicated that the T₁ group registered largest decrease in Non-awareness (25.3%) as well as the largest increase (12.0%) in Use for Publication A. These changes were significant (Z = 4.056, p < 0.001). For distracters (Publications B and C), these changes were not significant (Z = 1.433, p = 0.152; Z = 0.266, p = 0.790, respectively). Similarly, the T₂ group registered largest decrease in Non-awareness (18.4%) and the largest increase in Use (11.8%) for Publication A (statistical significance: Z = 3.029, p = 0.002). For Publication B, frequency at the Use level did not change, while it actually decreased for Publication C. These changes were not significant (Z = 0.408, p = 0.683; Z = −1.471, p = 0.141, respectively).

As expected for participants in the Control group, no changes in reported levels of use were found for Publications A, B, or C (Z = 0.992, p = 0.321; Z = 0.064, p = 0.949; Z = 1.632, p = 0.103, respectively).

The results indicate the LOKUS instrument’s excellent responsiveness to changes in level of knowledge use over 4 months of initial intervention period.

Developmental nature of the levels

Table 5 shows how the 151 participants in the two treatment groups T₁ and T₂ moved from one level of knowledge use to the next during the 4 months between baseline and the first follow-up. In T₁ (n = 75), 7 participants advanced one level, 10 advanced two levels, and 8 advanced three levels. But four T₁ participants dropped to lower levels. In T₂ (n = 76), five participants advanced one level, six advanced two levels, and nine advanced three levels. Five T₂ participants dropped to lower levels. The pattern of bi-directional movement, viewed in light of the low frequencies in the cells of Table 5, permitted no conclusion about the developmental nature of the four levels of knowledge use. This will have to await the results of replication studies.

Table 5.

Frequency changes in knowledge use levels of treatment groups between baseline and follow-up: Publication A. (n=151).

			Follow-up
			Frequency and Percentage
Baseline	Group	Levels	Non-awareness (n = 47)	Awareness (n = 6)	Interest (n = 10)	Use (n = 12)
	T₁ (n = 75)	Non-awareness (n = 66)	45 (68.2%)	5 (9.1%)	8 (12.1%)	8 (12.1%)
		Awareness(n = 5)	2 (40.0%)	0	1 (20.0%)	2 (40.0%)
		Interest(n = 1)	0	0	0	1 (100%)
		Use(n = 3)	0	1 (33.3%)	1 (33.3%)	1 (33.3%)
		Levels	Non-awareness (n = 51)	Awareness (n = 6)	Interest (n = 5)	Use (n = 14)
	T₂ (n = 76)	Non-awareness (n = 65)	48 (73.8%)	4 (6.2%)	4 (6.2%)	9 (13.8%)
		Awareness(n = 4)	1 (25.0%)	1 (25.0%)	0	2 (50.0%)
		Interest(n = 2)	1 (20.0%)	0	0	1 (40.0%)
		Use(n = 5)	1 (20.0%)	1 (20.0%)	1 (20.0%)	2 (40.0%)

T₁ = group exposed to Tailor & Target communication strategy.

T₂ = group exposed to Target-only communication strategy.

Discussion

The LOKUS instrument was created as a web-based survey instrument for measuring the extent to which various types of stakeholders to technology innovations engage with new knowledge. Specifically, it measures use of the new knowledge in the form of published findings excerpted from peer-reviewed journal publications. As explained in this article, the LOKUS instrument was systematically developed in a context, which reflected respondent engagement with technology-related knowledge. In using the KT4TT intervention study for its context, the instrument derived its terms and operational definitions from the study. It drew upon partial data from the study’s participants (corresponding to the first 4 months of the study) for establishing important psychometric attributes. In turn, it fulfilled its role in the KT4TT study by identifying levels of knowledge use by five types of stakeholders. It thus enabled the investigators to compare and judge knowledge communication strategies for effectiveness. The objective of the LOKUS instrument was to measure knowledge use and it performed successfully in that role in the intervention study. It has since demonstrated its feasibility and utility also in the two replication studies that followed.²⁶

The LOKUS instrument is based conceptually on Rogers’ stages of innovation diffusion and was guided initially by the LoUs framework.^30,31 As presented earlier, both the LOKUS instrument and the LoU conceptual models overlap with each other and with the generic stages of innovation decision in Rogers’ model. However, the LOKUS instrument differs from the LoU scale both in final structure and operational model despite some parallels between the two models.

The 10 stages classified under the four levels in the LOKUS instrument bear a parallel to the sequence of the seven LoUs. Both instruments view Use as a sequence of stages ranging from Non-use/Non-awareness to an ultimate adaptation and adoption, or Renewal/Modification, by the user. However, each instrument classifies and defines the sequence differently. The LOKUS instrument covers the 10 steps under Levels one and two (Non-awareness and Awareness), Level three (Orientation and Preparation), and Level four (Initial Use, Routine Use, Expansion, Collaboration, Integration, and Modification). The last four dimensions from expansion to modification are exclusive to technology-related knowledge use by autonomous individuals and mark the clearest point of departure of the LOKUS instrument from the LoU scale. Also, the LOKUS instrument views these last four dimensions as modified use (knowledge modified by the user) as distinct from the dimensions of Initial and Routine Use, which denote intended use (as originally intended by knowledge producer).

Approach to assessment in the LOKUS instrument differs from the LoU operational model. The LoU scale applies systematic qualitative interviewing to place respondents along the LoUs based on what users are doing; the LOKUS instrument elicits the users’ self-reported levels of new knowledge use through their responses to items in multiple-choice formats. The inherent nature of closed responses in the LOKUS instrument likely compromises the instrument’s ability to differentiate minute variations in individual levels. However, the instrument’s added capability of reaching multiple and disperse stakeholders is the tradeoff between a potential loss of accuracy (merit) for a gain in relevance (worth).^28,29

The potential users of the LOKUS instrument are sponsors and investigators of technology-based R&D projects. Demonstrating evidence of knowledge use by nontraditional stakeholders is of great value to these projects, but it is also a challenge these projects face. The challenge is twofold. First, reaching a dispersed stakeholder population and raising awareness of the knowledge can be especially difficult. Typically, academic research centers tend to communicate innovations through peer-reviewed publications and scholarly conferences. This mode of communication relies on the diffusion process to reach potential users through a sequence of naturally occurring events rather than through deliberate intervention. An academic mindset of relying heavily and exclusively on such a lengthy process can pose a formidable barrier for reaching multiple and geographically dispersed stakeholder audiences. Research to overcome this barrier through alternative dissemination approaches is urgent. More active and targeted dissemination practices commonly used in business and industry circles might offer alternative models. On a basic level, the language and format in which findings are presented in peer-reviewed publications might be a barrier to access by stakeholders with diverse professional backgrounds. Diversifying dissemination channels might improve accessibility and reduce the barrier. For example, manufacturers interested in transforming the new knowledge into concrete, usable prototypes might prefer trade journals for information access. Clinical therapists and knowledge brokers interested in applying evidence-based innovations might prefer the channels of communication that their professional organizations use with their membership. On yet another level, Graham et al. ^10,11 have proposed a knowledge-to-action model that recommends tailoring the new knowledge to the context of specific stakeholders. The intervention study discussed in this article evaluated two communication strategies compared to a control group. One tailored the knowledge to the context of stakeholders to bring out the relevance, and the other delivered the knowledge to targeted stakeholders with no tailoring. Using the LOKUS instrument, the study investigators were able to conclude that both strategies effectively raised stakeholder levels of knowledge awareness and use.²⁶ Future investigations on alternative KT strategies addressed in these and other models are needed for effectively removing barriers of access to specific stakeholders and increasing knowledge use.

A second challenge to R&D investigators is measuring the extent of knowledge use by their stakeholders, either as a basis for planning new KT strategies or demonstrating the effectiveness of existing strategies. A current barrier to investigator efforts is the lack of a reliable tool to measure knowledge use. The LOKUS instrument provides a means for investigators to measure knowledge uptake and overcome that barrier. It combines the advantages of both the RE-AIM and the LoU models, by broadening their application while maintaining their sensitivity to detecting knowledge use by stakeholders distributed along a sequence of levels. For example, data reports from the instrument in the form of frequencies at Awareness and Interest levels permit computation of the reach indices (the percent and representativeness of individuals willing to use the knowledge) recommended by the RE-AIM model. All the same, it is important to recognize that increasing access to knowledge is a role played by KT (i.e. effective communication strategies) rather than a role of the instrument. While the LOKUS can verify if and when the population was reached (or not), the burden of performing KT still rests with the investigator.

Strengths and limitations

Recognizing potential threats to measurement accuracy due to the instrument’s design constraints, we took due care to ensure its merit (intrinsic psychometric quality) during the development procedures, alongside its worth (external value or relevance to users).^28,29 The instrument underwent iterative validation and improvement as well as follow-up psychometric investigation, in the context of technology-related knowledge use by stakeholder respondents. The results for content relevance, test–retest reliability, content and construct validity, and responsiveness to change all demonstrated the LOKUS instrument to be feasible, valid, and reliable for capturing and detecting temporal changes in knowledge use by individuals representing multiple nontraditional stakeholder types.

Additionally, one-on-one testing of the instrument on a web-based platform showed both versions met the interface and logistic criteria of simplicity, feasibility, and usability, indicating an ability to reach diverse and dispersed audiences.²⁷

Initial analyses that triangulated the results from the independent analysis (n = 69) with the results from the larger intervention study (n = 215) (Table 2) strongly suggested the four-level sequence in the conceptual model for the LOKUS instrument. However, this was not conclusively substantiated in the final Phase 2 analysis. One difficulty was posed by data constraints. The low number of participants reporting being in either the Interest or Use levels at baseline limited the ability to accurately interpret changes in reported levels over the 4 months. This limitation was further compounded by reliance on respondents’ ability to recall over time by selecting responses associated with key word prompts. The self-reporting nature of responses called for cautious interpretation of results.

The fact that very few respondents were indeed using the published findings presented to them made it difficult to interpret their bi-directional movement patterns between levels. Regressions from higher levels to lower levels (e.g. dropping from Use to Interest or Awareness) are plausible over time. Capturing evidence of active use may be as dependent on the time people are surveyed as on the utility of the new knowledge itself. Investigators need to consider how long after an innovation it is reasonable to administer the LOKUS. The interval should be long enough to allow stakeholders to reach awareness and decide to use so that they can receive assistance for implementation. Yet if the interval is too long, the investigator might miss identifying any stakeholder that did actually use the knowledge but regressed for reasons of technical or logistical difficulty, or because of losing interest.

The above results might also point to available resources as a factor in the stakeholder’s decision to implement. The 4-month interval between two measurements in the intervention study might have been too short for securing resources. Notwithstanding, the results raise a more fundamental issue: to what extent does the stakeholder value the new knowledge? Relevance (utility) of the new knowledge is a factor in one’s decision to begin using it (moving from Awareness level to Use level). It can also affect one’s decision to continue using it (remaining at the Use level). Everett Rogers³⁰ addresses the issue in his persuasion stage by distinguishing needs from wants. People do not always know what they need. They may also want things they do not need. According to Rogers, a pre-existing need can drive a user to seek newly generated knowledge (such as a new pesticide desperately needed by a farmer). On the other hand, sometimes needs must be created for existing knowledge (such as for new fashions in the clothes industry). In the context of KT for technological innovations, the two scenarios are often likened to demand pull and supply push approaches. Both are valid. However, it is important to recognize that validating stakeholder needs (i.e. ensuring relevance) before the creation of new knowledge increases the likelihood of user acceptance.²⁴ Marketing effort to push new knowledge to stakeholders after its creation is harder and not always successful. It is also important to recognize that stakeholders include intermediaries such as clinicians and manufacturers who must first apply the new knowledge for eventual consumer use. Evidently the challenge to KT involves ensuring knowledge relevance both to consumer needs and to needs of intermediary stakeholders. Ensuring it beforehand is less burdensome on KT than creating or identifying a need for the knowledge afterwards. In the KT4TT intervention study discussed in this article, the two KT strategies were found effective in raising awareness and use by all five stakeholder types.²⁶ However, fewer numbers decided to use the new knowledge, although larger numbers were reached (i.e. became aware of the knowledge).

In light of the foregoing, the conclusion is that the LOKUS instrument can at least differentiate a four-level sequence (see Figure 2). Use may occur as intended by the knowledge creator or as modified by the knowledge user. Also, dimensions within modified use may not follow a lock-step developmental sequence.

Low cell frequencies posed challenges to analyses. They included difficulty in establishing comprehensive validity tests beyond content validity, limited use of parametric statistics, and difficulty in addressing psychometrics for the activity items. Further studies on dimensions and activities are therefore necessary.

Notwithstanding the restrictions discussed, the LOKUS instrument may be useful to investigators for broadly inferring user status regarding the four levels of use and, to some extent, their eight dimensions. However, the activity items listed under each dimension are too generic to offer investigators the in-depth information they need about different stakeholders for increasing knowledge uptake and use through collaboration or technical assistance.

Future work on expanding and tailoring the instrument to each stakeholder type will be beneficial. Currently, the instrument places a respondent along one of four levels of knowledge use and indicates the corresponding dimension. When administered across multiple stakeholders for a given case of published new knowledge, it can distribute the surveyed stakeholders across all four levels. Periodic re-assessment can further document changes in knowledge use behaviors over time. To the extent that documenting the use of new knowledge is important to demonstrate project efficacy, and to justify budgetary allocations for sponsored programs, the LOKUS instrument offers value to investigators and sponsors alike.

Finally, as a general precaution, the timing of administration of the LOKUS instrument is an important consideration. In this article, it was found to be sensitive to change induced by Publication A (the object of intervention) over 4 months. However, for Publications B and C (for which no intervention was provided), some respondents regressed across levels, raising questions about whether 4 months was too long a time frame to accurately recall and report levels of use. Therefore, it is important to consider how soon to administer the LOKUS instrument following the initial communication of new knowledge whether via publication, presentation, or electronic media.

Results reported in this article refer directly to pre-selected published findings presented to respondents in the AAC technology area. However, items in the instrument are designed for response by anyone considered to be a potential user of new knowledge in any technology area within rehabilitation and through technology-related R&D projects in general. The KT4TT center has since replicated the intervention study discussed in this article in two additional areas within rehabilitation technology. The results substantiate the instrument’s feasibility and utility as generalizable across these areas of application.²⁶ Future research is needed for generalizing utility across other areas.

Conclusion

Developed as an online measure of self-reported knowledge use by multiple stakeholders assumed to have an interest in new knowledge within a selected technology-related area, the LOKUS instrument has demonstrated strong (1) content validity; (2) test–retest reliability; (3) construct validity for distinguishing levels of knowledge use, both across published findings and across stakeholder types; and (4) responsiveness to changes in level of knowledge use over a 4-month time frame. At any given time point, sponsors and investigators involved in R&D projects can use the instrument to document the level of knowledge use within and across a range of stakeholder types, across four levels of (1) Non-awareness, (2) Awareness, (3) Interest, and (4) Use that enclose eight dimensions and 37 user activities. Future work is needed to further substantiate the sequential nature of the levels and interrelations among dimensions. Additional work is needed to enable LOKUS instrument to differentiate among user activities within a specific stakeholder type. Until such time, investigators attempting to intervene and improve uptake of their outputs might need to do further follow-up with interested stakeholders.

As noted in the “Discussion” section, the application of the LOKUS instrument in three technology areas validated its feasibility and utility.²⁶ As technology-based knowledge production provided the context for the development of the instrument, generalizing its use beyond this context would require additional research.

The ultimate utility of the LOKUS instrument lies in its value to R&D sponsors and investigators. Can they document evidence of stakeholder reach and use of their knowledge outputs? Can they track them to demonstrate eventual impact for their intended target beneficiaries? The value offered by the instrument is closely related to how effectively R&D projects can overcome the twofold challenge mentioned earlier. The KT challenge is to contrive effective strategies to improve stakeholder awareness and use of the new knowledge. The evaluation challenge is to gather data on the use of the knowledge as evidence of effectiveness of these strategies. As pointed out in the “Discussion” section, the LOKUS instrument effectively addresses the evaluation challenge. The KT challenge to reach, educate, and motivate users to apply new knowledge still rests with investigators and sponsors. Support from future research is urgent on investigation of strategies that increase knowledge use. It calls for illuminating dissemination alternatives that effectively remove barriers and facilitate the use of knowledge by stakeholders engaged in varied contexts. Equally important is research on alternative ways to generate relevant knowledge. These should take into consideration the issue of knowledge value to intended stakeholder audiences.

Footnotes

Appendix 1 Acknowledgements

The authors gratefully acknowledge critical comments made in the early stages of the project by Dr Gene Hall, professor at the University of Nevada, Las Vegas and author of the LoU scale that inspired our work. The authors also thank the national Rehabilitation Engineering Research Centers (RERCs) on Augmentative and Alternative Communication for their expert collaboration. Thanks are also due to Shreya Telang for making the LOKUS instrument operational on the Vovici software and for her initial contributions toward the psychometric evaluation.

Declaration of conflicting interests

The authors declare that they have no competing interests.

Funding

This research was funded by the National Institute on Disability and Rehabilitation Research (NIDRR) of the US Department of Education, under grant H133A080050.

References

United States General Accounting Office. Results-oriented government report to congressional requesters, http://www.gao.gov/new.items/d0438.pdf (2004, accessed 15 January 2014).

United States Government Printing Office. GPRA Modernization act of 2010, http://www.gpo.gov/fdsys/pkg/PLAW-111publ352/html/PLAW-111publ352.htm (2010, accessed 15 January 2014).

U.S. Office of Management and Budget. Government Performance Results Act of 1993, http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m (1993, accessed 15 January 2014).

Bauer

Flagg

. Technology transfer and technology transfer intermediaries. Assist Technol Outcome Benefit 2010; 6(1): 129–150.

Lane

. Understanding technology transfer. Assist Technol 1999; 11: 5–19.

Ottoson

Penelope

. Knowledge utilization, diffusion, implementation, transfer, and translation: implications for evaluation. New Dir Eval 2009; 124: 3–100.

Weiss

. The many meanings of research utilization. Public Admin Rev 1979; 39(5): 426–431.

Barwick

. Evaluation use and knowledge translation: an exchange for the future. In: Expert lecture delivered at the annual meeting of the American Evaluation Association, Anaheim, CA, 3 November 2011.

Canadian Institutes for Health Research (CIHR). About knowledge translation, http://www.cihr-irsc.gc.ca/e/29418.html (2013, accessed 15 January 2014).

10.

Graham

Logan

Harrison

. Lost in translation: time for a map? J Contin Educ Health Prof 2006; 26(1): 13–24.

11.

Graham

Tetroe

. Whither knowledge translation: an international agenda. Nurs Res 2007; 56(4): S86–S88.

12.

National Center for the Dissemination of Disability Research (NCDDR), http://www.ncddr.org/du/products/dddreview/index.html (1999, accessed 15 January 2014).

13.

Sudsawad

. Knowledge translation: introduction to models, strategies, and measures. Austin, TX: Southwest Educational Development Laboratory, National Center for the Dissemination of Disability Research, 2007, vol. 4, pp. 21–22.

14.

Glasgow

Klesges

Dzewaltowski

. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision-making involving complex issues. Health Educ Res 2006; 21(5): 688–694.

15.

Glasgow

Vogt

Boles

. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999; 89: 1322–1327.

16.

Hall

. The concerns-based adoption model: a developmental conceptualization of the adoption process within educational institutions. In: Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL, 18 April 1974.

17.

Coryn

CLS

Scriven

(eds). Reforming the evaluation of research. New directions for evaluation, issue 118. New York: John Wiley & Sons, 2008.

18.

Donovan

. State of the art in assessing research impact: introduction to a special issue. Res Evaluat 2011; 20(3): 175–179.

19.

Donovan

Hanney

. The “Payback Framework” explained. Res Evaluat 2011; 20(3): 181–183.

20.

Spaapen

van Drooge

. Introducing “productive interactions” in social impact assessment. Res Evaluat 2011; 20(3): 211–218.

21.

Brewer

. The impact of impact. Res Evaluat 2011; 20(3): 255–256.

22.

Beyer

. Research utilization: bridging the gap between communities. J Manage Inquiry 1997; 6: 17–22.

23.

Estabrooks

. The conceptual structure of research utilization. Res Nurs Health 1999; 22: 203–216.

24.

Flagg

Lane

Lockett

. Need to knowledge (NtK) model: an evidence-based framework for generating technological innovations with socio-economic impacts. Implement Sci 2013; 8: 21.

25.

Lane

Flagg

. Translating three states of knowledge: discovery, invention & innovation. Implement Sci 2010; 5: 9.

26.

Stone

Lane

Tomita

. Effectively communicating knowledge to assistive technology stakeholders: three randomized controlled case studies. Assist Technol Outcome Benefit, Submitted.

27.

Stone

Usiak

Lockett

. Creation and validation of the Level of Knowledge Use Survey (LOKUS) instrument, http://kt4tt.buffalo.edu/publications/WorkingPapers/Updated%20Working%20Paper%20II%20for%20Website%2010%2010%2013.pdf (2012, accessed 15 January 2014).

28.

Joint Committee on Standards for Educational Evaluation. The program evaluation standards. 2nd ed. Thousand Oaks, CA: SAGE, 1994.

29.

Yarborough

Shulha

Hopson

. The program evaluation standards: a guide for evaluators and evaluation users. 3rd ed. Thousand Oaks, CA: SAGE, 2011.

30.

Rogers

. Diffusion of innovations. 3rd ed. New York: Free Press, 1983, pp. 163–209.

31.

Hall

Dirksen

George

. Measuring implementation in schools: levels of use. Austin, TX: Southwest Educational Development Laboratory (SEDL), 2006.

32.

Hall

Loucks

Rutherford

. Levels of use of the innovation: a framework for analyzing innovation adoption. J Teach Educ 1975; 26(1): 52–56.

33.

Stone

Lane

. Measuring reach and uptake of new knowledge from technology innovations: need, context and concept, http://kt4tt.buffalo.edu/publications/WorkingPapers/Working%20paper%20I.pdf (2011, accessed 15 January 2014).

34.

Polit

Beck

. The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health 2006; 29: 489–497.

35.

Polit

Beck

Owen

. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health 2007; 30: 459–467.

36.

Lynn

. Determination and quantification of content validity. Nurs Res 1986; 35: 382–385.

37.

Waltz

Strickland

Lenz

. Measurement in nursing and health research. 3rd ed. New York: Springer, 2005.

38.

Vovici Corporation. Vovici 6. Herndon, VA, www.vovici.com (2011, accessed 15 January 2014).

39.

Center on Knowledge Translation for Technology Transfer. The LOKUS Instrument, http://kt4tt.buffalo.edu/publications/ResourceMaterials/index.php (2013, accessed 15 January 2014).

40.

Telang

. Establishing psychometric properties of the level of knowledge survey questionnaire for knowledge translation for technology transfer. MA Thesis, University at Buffalo, Department of Rehabilitation Science, Buffalo, NY, 2011.

41.

Penfield

Miller

. Improving content validation studies using asymmetric confidence interval for the mean of expert ratings. Appl Meas Educ 2004; 17: 359–370.

42.

Grant

Davis

. Selection and use of content experts for instrument development. Res Nurs Health 1997; 20: 269–274.

43.

Bryen

. Vocabulary to support socially-valued adult’s roles. Augment Altern Commun 2008; 24: 294–301.

44.

Miller

Spilker

. Readiness to change and brief educational interventions: successful strategies to reduce stroke risk. J Neurosci Nurs 2003; 35: 215–222.

45.

McNaughton

Symons

Light

. My dream was to pay taxes: the self-employment experiences of individuals who use augmentative and alternative communication. J Vocat Rehabil 2006; 25: 1–196.

46.

McKelvey

Dietz

Hux

. Performance of a person with chronic aphasia using personal and contextual pictures in a visual scene display prototype. J Med Speech Lang Pathol 2007; 15: 305–317.

Development of a measure of knowledge use by stakeholders in rehabilitation technology

Abstract

Objectives:

Methods:

Results:

Conclusion:

Keywords

Introduction

Method overview

Instrument conceptualization

Procedures

Phase 1: instrument development and validation

Method

Participants

Data collection and analysis procedures

Results

Content validity

Usability

Reliability

Conceptual model

Phase 2: establishing psychometrics for the LOKUS instrument

Method

Participants

Data collection and analysis

Content validity

Construct validity and responsiveness to change

The developmental nature of levels

Results

Content validity

Construct validity

Responsiveness

Developmental nature of the levels

Discussion

Strengths and limitations

Conclusion

Footnotes

Appendix 1

Acknowledgements

Declaration of conflicting interests

Funding

References