Abstract
Background:
A variety of instruments exist to measure human factors for lower limb amputation and prosthesis research yet, there is no valid or reliable tool available that focuses on technical potentials.
Objective:
This study aimed to validate and revise the Questionnaire to Explore Human Factors and their Technical Potential.
Study Design:
Cross-sectional study
Methods:
A total of 150 persons with lower limb amputation from Germany participated in the study. Statistical properties, including Cronbach’s alpha, item difficulty, item-total correlation, and distribution of missing values were calculated. Thresholds for acceptable psychometric properties were defined, unsuitable items were removed, and problematic items were reviewed regarding formulation.
Results:
The Cronbach’s alpha for subscales within the Questionnaire to Explore Human Factors and their Technical Potential were between 0.72 and 0.89. A total of 56 items showed acceptability, and 14 items had problematic item property values. Four of those items were reformulated, five were excluded, six were left in the scale, and an additional one was added to the scale.
Conclusion:
Evaluation of the Questionnaire to Explore Human Factors and their Technical Potential indicates it exhibits good internal consistency and acceptable psychometric properties. The scale was revised and is recommended to explore aspects of technical prosthesis development.
Clinical relevance
Our results show that the revised Questionnaire to Explore Human Factors and their Technical Potential may serve as a reliable and valid means, when designing prostheses, both during development and clinical evaluations and fittings, to assess the technical potential of lower limb prostheses directly according to the needs of users with lower-limb amputations.
Keywords
Background
The loss of a limb is a decisive event in a person’s life and can drastically reduce motor capabilities and quality of life. 1 Between 2005 and 2015, 589,240 leg amputations were reported in Germany. 2 Prosthetic components, including knees and sockets, have been developed and enhanced continually over time to restore function3,4 and thereby increase the patients’ quality of life. 5 Not only is prosthetic technology crucial to improving a user’s quality of life, it is also important to consider how the device interacts with the user.6,7 For example, poor prosthesis fit has been found to be associated with user depression, prior arterial reconstruction, diabetes, and pain in the residual limb. 8
There are a number of questionnaires that aim to measure patients’ well-being, only a few have been tested for reliability and validity.8,9 Due to this, the validity of patient satisfaction surveys are subject to debate.9,10 And despite the existence of validated tools to survey the well-being of people with lower limb amputation, there remain differences among their underlying human factors.
The Prosthesis Evaluation Questionnaire (PEQ), Trinity Amputation and Prosthesis Experience Scale (TAPES), and Orthotics and Prosthetics Users’ Survey (OPUS) are among the most established measures for prosthesis users. Prosthesis usefulness, residual limb health, appearance sounds, ambulation, transfers, perceived responses, frustration, social burden, and well-being constitute subscales of the PEQ.5,11 The TAPES includes three subscales regarding psychological adjustment, one about activity restriction, and two scales about satisfaction with the prosthesis. All subscales aim to measure psychological adjustment to wearing a prosthesis and quality of life of people with lower limb amputation.1,12 The OPUS includes four subscales, including functional status, health-related quality of life, and satisfaction with device and satisfaction with services. 13
Beyond these three multi-dimensional instruments, several valid and reliable measures have been designed to assess only one construct each. The Prosthetic Limb Users Survey of Mobility (PLUS-M), for example, measures mobility.14,15 The Amputee Body Image Scale (ABIS) was designed to assess the body image of people with lower-limb amputations,16,17 and the Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST) measures assistive device users’ satisfaction.18,19
All of the surveys mentioned above were developed from a clinical perspective and are used to assess the user’s experience with existing prosthetic devices,5,12–14,16,18 but do not focus on the technical potential of the prosthesis. The term technical potential describes the ability of a prosthesis and its technical components to enhance the user’s well-being. 7 Therefore, the Questionnaire to Explore Human Factors and their Technical potential (QEHFT)20–22 was developed, which serves as a basis for the human-machine-centered prosthetic design method.7,22 This approach aims to facilitate development of user-oriented devices and components by considering constructs (i.e., human factors) that may be directly influenced by the technical design of the prosthesis, including: satisfaction (SAT), feeling of security (FOS), mobility (MOB), body schema integration (BSI), support (SUP), socket (SOC), and outer appearance (OUT). 22 To evaluate the corresponding technical potentials, interrelations between human factors and design of the prosthesis need to be identified and assessed. 22 A definition for each included human factor is given in Table 1. The QEHFT items were initially developed and categorized by users and experts 6 and are formulated in a way that targets the technical potential of an assistive device. 22 This study was intended to assess the reliability and validity the QEHFT by examining the internal structure and psychometric quality of the included subscales and items. Statistical item properties were calculated following the classical test theory approach, and problematic items from a previous version of the questionnaire were identified, and revised or removed as appropriate. Both, the original German as well as the revised English version (with different item numbering), are included in the supplementary material.
Definitions and reliability (internal consistency) values for the seven scales in the Questionnaire to Explore Human Factors and their Technical potential (QEHFT).
Valid cases listed as an average due to pairwise deletion. Individual item correlations for each subscale may be found in the supplementary material.
Methods
Participants
A total of 189 individuals with lower limb amputation were recruited to participate in this study. Criteria for exclusion were indicating “no” when asked whether they were prosthesis users, not specifying demographic data, or when the level of limb loss was at or below the ankle. A description of the final sample (n=150) is provided in Table 2.
Demographic details of the study sample (n=150).
SD: standard deviation; Active prostheses: prostheses with active components like a mechanical motor that provides power to the human-machine system; Microprocessor-controlled: prostheses that adjust dynamic characteristics of the prosthesis due to changes in the environment or different moving styles of the user; Mechanical: prostheses that have fixed mechanical movement patterns.
Material
To measure the seven human factors mentioned above, the QEHFT includes 70 items that are subdivided into seven subscales. Each item is evaluated using a 5-point Likert-type scale including, the possibility of not providing an answer. The scale, as used in this study, is included in the online Supplement.
Procedure
Data were collected between March 2014 and March 2017 using both an online and a pen and paper version of the QEHFT. The link leading to the online version of the questionnaire was distributed by topic-related Facebook-groups, Internet forums, and online self-help groups. Information flyers and pen and paper versions of the questionnaire were also distributed to selected specialized clinics and health care supply stores. Among the 150 participants, only four completed the questionnaire in pen and paper form. Based on textual similarity between the pen and paper and online version and findings in related studies we do not expect any bias due to mode of administration. 23 Informed consent was obtained from all participants prior to completing the survey.
Statistical analysis
Item and reliability analyses following the classical test theory approach24,25 were carried out using the Statistical Package for Social Sciences (SPSS, Version 23.0, Chicago, IL, USA) and the open-source edition of RStudio (RStudio Inc, Boston, MA USA). Descriptive analyses were performed to determine the demographic properties of the sample, as well as to summarize questionnaire responses.
To determine internal consistency, a reliability analysis calculating Cronbach’s Alpha was conducted for each human factor subscale. To identify problematic items in the QEHFT item pool, four characteristic values were calculated for each item: Difficulty (P), part-whole corrected item-total correlation (selectivity), the percentage of missing values, and changes in Cronbach’s alpha value when removing particular items.
Item difficulty describes the proportion of participants answering the item in the symptomatic direction of the underlying trait on a scale from 0 to 100. In the case of positive subscales, high P-values were considered to signify items with high rates of trait-symptomatic answers. In the case of negative subscales, a reversed interpretation was applied. 26 A P-value of 50 marked the highest differentiation potential between participants with high and low characteristics. Items with extreme P-values can be added if the test differentiates between extreme characteristics. 27
Part-whole corrected item-total-correlation is the Pearson’s product-moment correlation coefficient of an item and its related human factor. It describes the relationship among the differentiation between participants based on the specific item and the differentiation between participants based on the associated human factor and can, therefore, be used to analyze subscale homogeneity. Part-whole correction implies that the specific item is excluded from the calculation of the subscale-value used in the correlation analysis leading to its item-total correlation value. 28 The identification of problematic items was tied to the threshold values of the four-item indices calculated. Item-difficulty values were defined to be problematic if they were located outside 20 and 80, as this indicates items with extreme difficulty manifestations. Item-total correlation values were defined as problematic if they surpassed a lower limit of 0.3 following published recommendations.28,29 Furthermore, cumulative occurrences of items showing item-total-correlation above 0.8 were interpreted as problematic. 29 Changes in subscale-reliability (Cronbach’s alpha) in the case of removing particular items were interpreted as problematic when an item deletion resulted in an increase of the Cronbach’s alpha value. Additionally, to identify items with low completion rates, items with more than 20% missing values were defined as problematic.
The general interpretation of Cronbach’s alpha values and the often arbitrarily specified thresholds are highly debatable but nevertheless popular in statistical research. 30 Interpretation rules given by Everitt and Skrondal 29 were applied to interpret the internal consistency of the QEHFT’s scales.
Both reliability and item analyses were performed with pairwise deletion to include the highest amount of data possible. This means that statistical analyses only include pairs of cases without missing values which results in different sets of averagely valid cases for each Cronbach’s alpha coefficient. 31 To preclude the possibility of distortion as a consequence of systematically occurring missing data, Little’s missing completely at random (MCAR) test was conducted. Originating from statistical analysis, problematic items were analyzed regarding terminology and later on discussed in terms of possible issues regarding the relation to the construct of each respective underlying human factor.
Results
Little’s MCAR test resulted in accepting the null hypothesis (p = .482). This result legitimizes using pairwise deletion as no dependencies that could bias results of the statistical analysis were found in missing data.31,32
Cronbach’s alpha values calculated for each subscale as well as the individual values of the corresponding items belonging to each factor are shown in Table 1. Additionally, the average number of valid cases in each particular reliability analysis and the total cases used are also presented in Table 1. Very good internal consistency was found for satisfaction (alpha = .89), feeling of security (alpha = .89), and body schema integration (alpha = .88) subscales, which all consist of 11 items. Consisting of 14 items, socket subscale (alpha = .84) also showed very good internal consistency. The support subscale (alpha = 72) consisting of five items, mobility subscale (alpha = .79) consisting of 14 items, and outer appearance subscale (alpha = .72) consisting of four items, all showed respectable internal consistency.
In ten cases, problematic items showed an increase in the Cronbach’s alpha value of their subscale if the item was deleted. In seven cases (items 12, 20, 34, 35, 37, 55, 67), the increase amounts to 0.01. Three cases show higher results: removal of item 54 resulted in an increase of 0.1, removal of item 22 resulted in an increase of 0.02, and removal of item 29 resulted in an increase of 0.04. In four cases, the Cronbach’s alpha value of the problematic item’s subscale decreased if the item was deleted. The removal of item 18 resulted in a decrease of 0.1 while the removal of both items 58 and 59 resulted in a decrease of 0.2. The highest decrease (0.3) followed removal of item 60.
Both items 67 and 20 show extreme difficulty and rather low item-total correlation values. Regarding item 20, it is notable that this statement is similar to the statements of items 19, 18, and 17, all of which are part of the socket subscale. Both items 20 and 17 contain the term “technical aid” and also have the lowest item-total correlation values among those items. Items 67 and 18 show no special terminology features and are solely located in the extreme range of the value item difficulty. While items 12, 22, 54, and 55 are found in different human factors, they all ask for the presence of particular prosthesis features and show exceptionally low item-total correlation. Item 12 additionally shows a high rate of missing values, the only case where the rate surpasses the threshold of 20 %. The items 34, 35, and 37 from the subscale obility show very low item-total correlation values as well. All of these items contain different statements on the user’s experience in changing their walking speed. A majority of items (61 out of 70) show inconspicuous property values, based on the threshold values mentioned. The 14 items interpreted as problematic are presented in Table 3.
Problematic items and their properties.
MOB: Mobility; SOC: Socket; OUT: Outer appearance; SUP: Support; SAT: Satisfaction; BSI: Body schema integration. Item properties for all items in the QEHFT are included in the supplementary material.
Discussion
As stated above, the main aim of this study was to assess the structure and psychometric quality of the QEHFT subscales, revising or removing individual items as appropriate. Thereby, the survey is prepared for further application in identifying technical potentials of human factors worth considering in technical design of a prosthesis.
The subscales of the QEHFT show very good (SAT: 0.89, FOS: 0.89, BSI: 0.88, SOC: 0.84) and respectable (SUP: 0.72, MOB: 0.79, OAP: 0.72) reliability values, respectively. Regarding the item properties difficulty, item-total correlation, percentage of missing values, and Cronbach’s Alpha in case of item exclusion, 56 items showed good or acceptable values, whereas 14 items were identified to be problematic, namely, items 12, 18, 20, 22, 29, 34, 35, 37, 54, 55, 58, 59, 60, and 67. Upon checking construct validity, we found that in most cases, the item formulation aimed at relevant facets of the factor construct. Some items, however, especially those with identified problems do not ask respondents about aspects relevant to the underlying construct. The overlap between statistically problematic items with formulation issues is noticeable. Therefore, the problematic item property values are interpreted as a result of inaccurate item terminology. Thus, based on weighting between heterogeneity and construct-validity of the subscales, we drew the following conclusions:
While we recommend excluding item 12, battery life is a crucial factor for every electrically supported prosthesis. 33 Instead, we suggest including an item to measure maximum walking distance with the prosthesis, which might be easier for participants to comprehend and is included in other similar questionnaires. 34
Items 19 and 20 measure almost the same concepts as items 17 and 18, yet slightly more specific. In order to avoid unnecessary redundancies, while still assessing technical potential, we, therefore, suggest omitting items 19 and 20, and leaving items 17 and 18 in the scale. The critically seen item difficulty value of item 18 is expected to change due to restructuring of the subscale.
Items 22, 54, and 55 ask if the prosthesis contains special features but omits the user’s satisfaction with those features. Hence, a reformulation to focus on the subjective perspective of technical factors is necessary. Specifically, for item 22, we suggest the formulation “I am satisfied with the independent adjustment of my socket to the shape and volume of my stump.” We recommend changing item 54 to “I am satisfied with the way my prosthesis offers mechanical or electronic support while rolling over from heel to toes.” Finally, item 55 should be reformulated to “I am satisfied with the way I can adjust my prosthesis adequately regarding different heel levels of my shoes.” We expect these changes to improve assessment of satisfaction with the technical features of the prosthesis. Asking users about their satisfaction with features of the prosthetic device aligns with associations between the human factors support, socket and mobility found by Beckerle et al. 22
Item 29 is part of the subscale Outer Appearance. Its critical item properties imply missing linkage to the other items in the subscale. We suggest adding another item to measure appearance not only in public but also at home in order to improve heterogeneity of the subscale. Thereby, we expect a decline in scale internal consistency as well as a rise of item-correlation throughout the subscale’s items. Measuring outer appearance is important for prosthesis design, Was it impacts people with lower limb amputation’s acceptance and use of the prosthesis35,36 and is linked to user satisfaction.12,20,22
Walking speed and gait patterns are important aspects when assessing mobility, as they are also associated with user satisfaction. 7 Therefore, several items about these aspects are included in the mobility subscale. However, items 34, 35, and 37 are all formulated in a way that measures attitudes of prosthesis users toward walking speed, disregarding experience with the prosthesis. Therefore, we suggest omitting these items and exchanging them with one item that focuses on user experience rather than on general attitudes or wishes in order to better align the item formulation with the underlying construct. Items 58, 59, and 60 are part of the subscale Body Schema Integration subscale, which was designed based on the work of Longo and Haggard 37 as well as Longo et al. 38 The mentioned items show very high values on item-total correlation. We suggest leaving these items in the subscale to guarantee overall heterogeneity. Item 67 is part of the same subscale and showed high item difficulty. We recommend omitting item 67 from the subscale, as it will improve heterogeneity of the subscale and assesses an important aspect of the subscale’s construct. 39 At last, we suggest including the answer option “not applicable,” as well as the instruction to select said choice, in case the patient’s prosthesis does not include the technical component mentioned in the question.
After these revisions, all items aim to point out the technical potential of prostheses and their respective technical components, and are therefore supporting the underlying human-machine-centered design approach described by Beckerle 7 and Beckerle et al. 22 To promote further exploration of the QEHFT and its potential to contribute to prosthesis development, the original German, revised English, and final English versions (along with scoring information) are provided as supplementary material of this article.
The reliability and validity analysis was based on a rather large set of data from 150 participants. This allowed analysis of reliability and item properties. By connecting the analysis of statistical item properties with a textual discussion of problematic items, we made suggestions concerning item formulation, in case the subscale construct was missed. We aimed to increase validity by omitting or reformulating problematic items or adding questions if necessary. We also defined thresholds for problematic item properties, which are not fully supported by statistics literature, as only a few sources implied thresholds and proper handling of these values. To complement the findings beyond the limitations of classical test theory, a probabilistic reliability analysis conducted with a suitable sample size might be useful for future validation of QEHFT. 25 The same suggestion was made in a study verifying the structure of the QEHFT. 40 To gain deeper insight into the internal structure of the survey, correlation analyses to measure and illustrate interactions between the human factors could be another useful addition to the present study. Also, qualitative interviews with people with lower limb amputation, as well as clinicians, such as prosthetists, therapists, and rehabilitation physicians may help to improve the item formulation or to identify and include other important aspects of the constructs. Therefore, we recommend the questionnaire be used in prosthesis development studies.
Conclusion
The QEHFT scale demonstrated good to excellent item property values as well as valid item formulation. For those items, which did not support the subscale’s construct, new item formulations were suggested. While there remains room for improvement, our results suggest that the QEHFT is an overall valid and reliable measure that has the potential to contribute to technical developments in lower limb prosthetics. We consider the revised version of the QEHFT as a valid instrument for measuring human factors as a means for prosthetic design.
Supplemental Material
POI931991_Appendix – Supplemental material for Validation and revision of the questionnaire to explore human factors and their technical potential for lower limb prosthetics
Supplemental material, POI931991_Appendix for Validation and revision of the questionnaire to explore human factors and their technical potential for lower limb prosthetics by Jonathan Herbst, Lucas MT Polanski-Schräder, Joachim Vogt, Tim Schürmann and Philipp Beckerle in Prosthetics and Orthotics International
Footnotes
Acknowledgements
The authors thank all participants for their time and input as well as Katrin Neuheuser and Diana Adrian for their support in preparing this study.
Author contribution
All authors contributed equally in the preparation of this manuscript.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Ethics Review and Approval
Study procedures were conducted based on review and recommendations by the local ethics committee of the Institute for Psychology at Technische Universität Darmstadt, Germany.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
