Abstract
Objective
Electronic patient-reported outcome (ePRO) measures are increasingly used in respiratory care, yet evidence regarding their feasibility and measurement performance in real-world clinical settings remains limited. The S3-NIV questionnaire is a brief patient-reported outcome tool assessing respiratory symptoms, sleep quality, and side effects in patients receiving non-invasive ventilation (NIV). This study aimed to develop a Japanese electronic version of the S3-NIV questionnaire and to evaluate its feasibility and internal consistency.
Methods
The S3-NIV questionnaire was translated into Japanese according to ISPOR Translation and Cultural Adaptation guidelines. After linguistic validation, a web-based electronic version was developed. A cross-sectional observational feasibility study was conducted in patients with chronic respiratory failure receiving home NIV. The Japanese electronic version of the Severe Respiratory Insufficiency (SRI) questionnaire was used as a comparator. Internal consistency was assessed using Cronbach’s alpha coefficients.
Results
Seven patients successfully completed the electronic questionnaires, confirming technical feasibility. Internal consistency was acceptable to high for the S3-NIV total score and respiratory symptoms subscore (Cronbach’s alpha = 0.79 and 0.92, respectively), whereas the sleep and side-effects subscore showed low internal consistency (alpha = 0.31). Several subscales of the electronic SRI questionnaire also demonstrated low alpha coefficients.
Conclusion
Although the Japanese translation of the S3-NIV questionnaire was linguistically validated and electronically feasible, reduced internal consistency was observed in specific subscales. Similar findings in the electronic SRI questionnaire suggest that factors related to mode of administration may influence measurement properties; however, this hypothesis requires confirmation in adequately powered equivalence studies.
Keywords
Introduction
Electronic patient-reported outcome (ePRO) measures are increasingly incorporated into clinical research and routine healthcare,1–3 driven by advances in mobile health technologies and digital data collection. Electronic administration offers practical advantages, including automated scoring, real-time data capture, and reduced missing data. However, concerns remain regarding feasibility and measurement equivalence when paper-based questionnaires are digitized for real-world clinical use, particularly when instruments are applied outside controlled research settings. These issues are especially relevant in chronic respiratory care, where patient-reported outcomes (PROs) play a central role in evaluating symptom burden, treatment effects, and patient experience over time.
In patients with chronic respiratory failure (CRF) receiving long-term non-invasive ventilation (NIV),4–6 PRO measures are essential for capturing respiratory symptoms, sleep-related problems, and treatment-related side effects that may not be fully reflected by physiological assessments alone. Disease-specific instruments such as the Severe Respiratory Insufficiency (SRI) questionnaire and the Maugeri Respiratory Failure questionnaire provide comprehensive health-related quality-of-life assessments7–11; however, their length and complexity limit routine use in busy clinical settings. To address this gap, the S3-NIV questionnaire was developed as a concise PRO tool focusing on three clinically relevant domains: respiratory symptoms, sleep quality, and NIV-related side effects. 12 Owing to its brevity and simple structure, the S3-NIV questionnaire is well suited for repeated assessment and electronic administration, aligning with current trends in mobile health and digital patient monitoring. Nevertheless, while several paper-based language versions of the S3-NIV questionnaire have demonstrated acceptable measurement properties,13–15 evidence regarding the feasibility and measurement performance of its electronic implementation across different languages and healthcare systems remains limited.
Against this background, the primary objective of the present study was to develop a Japanese electronic version of the S3-NIV questionnaire and to examine its feasibility in routine clinical practice. In addition, we aimed to explore the internal consistency of the electronic S3-NIV questionnaire and to compare its measurement performance with that of an established electronic PRO instrument, the Severe Respiratory Insufficiency (SRI) questionnaire. 16 By focusing on practical implementation rather than formal psychometric validation, this study sought to identify potential measurement challenges associated with digitizing PRO tools in real-world respiratory care settings. Accordingly, all psychometric findings in this study should be regarded as exploratory and preliminary, not confirmatory.
Materials and methods
Study design
This study was designed as a cross-sectional feasibility study aimed at developing and implementing a Japanese ePRO version of the S3-NIV questionnaire and examining its practical feasibility in routine respiratory care settings. Analyses of internal consistency and correlations were included as exploratory assessments of measurement performance in a pilot context and were not intended as formal psychometric validation.
Translation and linguistic validation
The project was launched on December 6, 2018, with the permission of the original developer of the S3-NIV questionnaire (Dr. Dan Adler). Translation of the original English version of the S3-NIV questionnaire into Japanese, together with linguistic validation of the translated version, was conducted in accordance with the 2005 International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Translation and Cultural Adaptation (TCA) Task Force Report. 17 The standardized process comprised the following steps: (1) preparation, (2) forward translation, (3) reconciliation, (4) back translation, (5) back translation review, (6) harmonization, (7) cognitive debriefing, (8) review of cognitive debriefing results and finalization, (9) proofreading, and (10) preparation of the final report. 17 In addition, a developer review was incorporated during the harmonization stage to ensure conceptual equivalence between the Japanese and original versions. The translation process was initiated in December 2018 and coordinated by the project manager, a chest physician (KN).
Briefly, two independent translators produced forward translations of the original questionnaire, including all 11 items and response options, from English into Japanese. One translator was a chest physician practicing in Japan (TH), and the other was a former professional translator currently working as a medical investigator (SO). These two forward translations were reconciled into a single provisional version by an in-country consultant, also a chest physician (TO). This reconciled Japanese version was subsequently back translated into English by a professional translator (NS). The project manager reviewed the back-translated version and submitted a detailed report to the original questionnaire developer for review and feedback.
Cognitive debriefing interviews were conducted between February and October 2020 at the Respiratory and Allergy Clinic of Tosei General Hospital. Six patients with chronic respiratory failure receiving non-invasive ventilation (NIV) participated voluntarily. This phase of the study was approved by the Tosei General Hospital Medical Ethics Committee (Approval No. 828; December 26, 2019), and all participants provided written informed consent. Efforts were made to include patients with diverse characteristics in terms of sex, age, disease severity, and area of residence.
The Japanese paper-based version of the S3-NIV questionnaire was finalized and reported to the original developer, Dr. Dan Adler, on May 16, 2021. Ethical approval for the subsequent feasibility study was obtained on January 31, 2022. No revisions or updates were made to the questionnaire content during the interval between finalization of the translation process and initiation of the feasibility study.
Feasibility assessment of the electronic version
Following finalization of the Japanese paper-based version, a web-based ePRO system was developed to enable real-world administration of the S3-NIV questionnaire in routine clinical practice. The electronic version was designed for use on a handheld tablet computer (screen size ≥17 × 24 cm) or a standard personal computer (PC). Study staff encouraged the use of a larger screen whenever possible. The system website (https://koichi1.com/) consisted of three components: a welcome page, the questionnaire page, and a summary page displaying calculated scores.
The entire questionnaire was presented on a single page, with item wording and response options identical to those of the paper-based version. The system was programmed to prevent progression without completion of all items and to restrict responses to a single option per item. Upon completion, a summary screen automatically displayed the calculated S3-NIV scores. No additional formal specifications regarding font size standardization, adaptive layout, or device-specific rendering were predefined. The need for scrolling was not uniform and depended on the screen size and browser conditions.
The questionnaires used in this study were based on previously developed instruments. The original S3-NIV questionnaire has been reported previously, and in the present study its Japanese version underwent a structured translation and linguistic validation process, including cognitive debriefing, before electronic implementation. The SRI questionnaire is an established instrument for patients receiving NIV, and the validity of its Japanese paper-based version has also been reported previously.18–20 Accordingly, the objective of the present study was not to perform formal psychometric validation of these instruments, but rather to examine the feasibility of their Japanese electronic administration in routine clinical practice, with internal consistency and correlation analyses included only as exploratory assessments in a pilot context.
To examine measurement performance in this exploratory context, the electronic S3-NIV questionnaire was evaluated in comparison with the Severe Respiratory Insufficiency (SRI) questionnaire. An electronic version of the SRI questionnaire based on the original instrument has also been developed as a dedicated application. 16 When contacting the SRI developer, we were informed that a Japanese version of the application was available; however, its functionality could not be confirmed within our local environment. Consequently, permission was obtained from the developer to create a web-based Japanese electronic version of the SRI questionnaire. The resulting SRI website (https://koichi1.com/sri-top/) comprised a welcome page, a three-page questionnaire, and a final summary page, with item wording and response options identical to those of the original paper-based version.
Study population and procedures
A cross-sectional observational feasibility study was conducted between February 2022 and April 2024 in the departments of respiratory medicine at four medical facilities, including the National Center for Geriatrics and Gerontology (NCGG). The study was conducted in accordance with the Declaration of Helsinki and was approved by the ethics committees of all participating institutions, including the NCGG Ethics Committee (Approval No. 1573; January 2022). Written informed consent was obtained from all participants prior to study enrollment.
Eligible participants were adults aged 18 years or older who attended outpatient clinics and met the following criteria: (1) at least three months had elapsed since their first outpatient visit; (2) at least one month had elapsed since initiation of home NIV therapy; (3) no hospitalization within the preceding month; and (4) a clinically stable condition. Stability was defined by resting arterial blood gas analysis demonstrating a partial pressure of arterial carbon dioxide (PaCO2) > 50 torr while inhaling oxygen at the minimum concentration required to achieve a partial pressure of arterial oxygen (PaO2) > 55 torr. If symptoms suggestive of infection had occurred within the preceding month, study assessments were postponed.
Participants completed the Japanese electronic versions of the S3-NIV and SRI questionnaires using a PC or tablet device. Clinic supervisors opened the initial webpage and provided the device to participants. Supervisors offered assistance with device operation and navigation when necessary; however, all questionnaire responses on the web-based questionnaires were entered independently by the participants and exported automatically. After completion, the devices were returned to the supervisors. Participant demographic and clinical data, rather than the questionnaire responses themselves, were subsequently entered into the electronic data capture (EDC) system by study staff. No manual transcription of questionnaire responses into the EDC system was performed. Participant demographic and clinical data entered into the EDC system were checked against the source records by study staff. For this feasibility study, the target sample size was set at 50 participants, but the final sample consisted of seven participants.
Statistical methods
All data are presented as mean ± standard deviation (SD). A two-sided p-value < 0.05 was considered statistically significant. Internal consistency of the questionnaires was evaluated using Cronbach’s coefficient alpha, and associations between questionnaire scores were assessed using Spearman’s rank correlation coefficient. Because of the pilot nature of the study and the very small sample size, these analyses were performed as exploratory evaluations of measurement performance rather than confirmatory psychometric assessments.
Results
The results of this cross-sectional feasibility study summarize the development of the Japanese version of the S3-NIV questionnaire, including translation and linguistic validation, as well as the implementation, completion rates, and exploratory measurement performance of the Japanese electronic versions of the S3-NIV and SRI questionnaires in routine clinical practice.
Translation and linguistic validation
Detailed results of the translation and linguistic validation process, from initial forward translation through finalization, are provided in Supporting Information 1. Following review of the back-translated version by the project manager, the materials were submitted to the original questionnaire developer for further evaluation. As a result, although additional discussion and consensus were required for four items (items #3, #7, #10, and #11), the developer confirmed that there were no major concerns regarding the remaining seven items or the five response options. Feedback from the developer was subsequently incorporated into the cognitive debriefing process.
Six male patients aged 64–80 years participated in the pilot cognitive debriefing (Supporting Information 2). Interviews were conducted by four trained therapists using a standardized interview format. Detailed findings from these interviews, which assessed linguistic clarity and conceptual validity, are presented in Supporting Information 3. For six of the eleven items (items #1, #2, #3, #4, #5, and #7), all participants agreed with the Japanese translations finalized during the harmonization stage. For three items (items #6, #8, and #9), one or two participants expressed a preference for alternative wording; however, the originally proposed translations were retained, as they were considered more widely acceptable and conceptually appropriate. For item #10, participant opinions were divided; nevertheless, the translation was retained based on consistency with the developer’s recommendations. In contrast, the Japanese wording for item #11 was revised to reflect participant preferences expressed during cognitive debriefing.
Regarding the five response options, the initial wording for two options was modified and replaced with simpler expressions, as these were more readily understood and preferred by participants. The finalized Japanese version of the S3-NIV questionnaire is presented in Supporting Information 4.
Feasibility assessment of the electronic version
Patients’ demographic and clinical characteristics.
*The blood gas analysis was performed on the same day as the responses to the electronic questionnaire, with the NPPV switched off (oxygen was administered via nasal cannula at 0.5-2.5 L/min in five patients). BMI, body mass index; IPAP, inspiratory positive airway pressure; EPAP, expiratory positive airway pressure; COPD, chronic obstructive pulmonary disease.
The internal consistency and score distribution of the questionnaires.
SRI: Severe Respiratory Insufficiency Questionnaire.
Spearman’s rank correlation coefficients (Rs) of the S3-NIV questionnaire scores with other variables.
Not statistically significant except for *: p < 0.05; SRI: Severe Respiratory Insufficiency Questionnaire.
Discussion
The results of this study demonstrate that the Japanese version of the S3-NIV questionnaire was developed through a rigorous translation and linguistic validation process, as detailed in the Results, thereby providing a sound methodological basis for evaluating its electronic implementation in routine clinical practice. Building on this foundation, the present feasibility study contributes to the digital health literature by suggesting that successful linguistic translation of a PRO measure may not necessarily guarantee comparable measurement performance when the instrument is implemented electronically in real-world clinical settings. Although the Japanese S3-NIV questionnaire could be administered electronically without technical difficulty, reduced internal consistency was observed in specific subscales. Importantly, similar reductions in internal consistency were also identified in the electronic version of the SRI questionnaire, an established and previously validated instrument. This observation may indicate that the findings were not confined to a single questionnaire, although tool-specific effects cannot be excluded. Factors related to mode of administration may therefore deserve consideration as possible contributors to measurement variability, while acknowledging that alternative explanations remain plausible. By directly comparing two electronic PRO instruments in routine respiratory care, this study underscores the importance of feasibility testing and empirical evaluation of measurement equivalence before widespread adoption of ePRO tools.
Comparison of Cronbach’s α coefficients of the different versions of S3-NIV total score and its subscores.
An additional point requiring caution is the interpretation of Cronbach’s alpha for very short subscales. The sleep and side effects subscore of the S3-NIV questionnaire comprises only a limited number of items, and Cronbach’s alpha is well known to be influenced by scale length. Accordingly, a low alpha value in such a short subscale does not necessarily indicate poor internal consistency per se but may partly reflect the mathematical properties of the coefficient and the structural characteristics of the scale. Therefore, the particularly low alpha observed for this subscale should be interpreted cautiously and should not be overinterpreted as evidence of reduced measurement performance by itself.
Comparison of Cronbach’s α coefficients of the different versions of seven subscales and Summary Scale.
SRI, Severe Respiratory Insufficiency Questionnaire; SS, Summary Scale; RC, Respiratory Complaints; PF, Physical Functioning; AS, Attendant Symptoms and Sleep; SR, Social Relationships; AX, Anxiety; WB, Psychological Well-Being; SF, Social Functioning.
However, the most important limitation in interpreting these findings is the extremely small sample size used for the electronic feasibility and internal consistency analyses. Although this study was originally planned with a target sample size of 50 participants, recruitment proved more difficult than anticipated, and only seven participants were ultimately included in the electronic evaluation. Cronbach’s alpha coefficients derived from such a small sample are statistically unstable and highly sensitive to random variation, and even minor differences in response patterns may substantially influence the estimated coefficients. Therefore, the lower internal consistency observed in some subscales of the electronic S3-NIV and SRI questionnaires should be interpreted with considerable caution. These findings should not be regarded as firm evidence that digitization reduced internal consistency, but rather as preliminary and hypothesis-generating observations that require confirmation in larger studies.
Several mechanisms may explain why electronic administration of PRO instruments could influence internal consistency even when linguistic translation is successful. First, electronic administration may alter how patients perceive and interpret questionnaire items. Differences in screen layout, font size, and item presentation, such as displaying all items on a single screen or enforcing sequential navigation, may influence response behavior compared with paper-based formats. Second, electronic systems typically prevent item skipping and enforce single-response selection. While these features reduce missing data, they may also constrain respondents’ ability to express uncertainty, particularly for subjective domains such as sleep quality or treatment-related side effects. Third, in real-world clinical settings, electronic questionnaire completion may require assistance with device operation or navigation, potentially introducing subtle influences on item interpretation. Finally, contextual factors such as time constraints during outpatient visits or limited familiarity with digital devices among older patients may further increase response variability. Collectively, these factors may have contributed to the observed findings; however, the present study was not designed to determine their relative importance or to establish a causal effect of digitization. These observations should therefore be regarded as preliminary and hypothesis-generating, requiring confirmation in adequately powered studies designed to assess measurement equivalence between paper-based and electronic formats.
These interpretations are consistent with prior findings in digital health and ePRO literature. Juniper and colleagues have shown that patients may respond differently to paper and electronic versions of the same respiratory questionnaires, with only modest agreement despite identical item wording, indicating that mode of administration itself can influence responses. 24 Similar concerns have been emphasized in guidance from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), which cautions that measurement equivalence between paper-based and electronic PRO instruments should not be assumed but instead demonstrated empirically. 17 In chronic respiratory disease, studies comparing electronic and paper versions of respiratory symptom and health status questionnaires, including the COPD Assessment Test and the Evaluating Respiratory Symptoms in COPD (E-RS), have reported systematic differences in score distributions, with electronic formats sometimes yielding lower symptom scores. 25 The present findings extend this body of evidence by showing that such digitization-related effects may also occur in brief, disease-specific instruments and may disproportionately affect subscales assessing subjective or context-dependent experiences.
Several limitations should be acknowledged. First, the very small sample size and cross-sectional design precluded any formal psychometric validation, including robust assessment of internal consistency, test–retest reliability, responsiveness, and measurement equivalence between paper-based and electronic formats. Second, because the analyses of internal consistency and correlations were based on only seven participants, these results should be considered exploratory and underpowered. Third, Cronbach’s alpha should be interpreted cautiously for very short subscales, because the coefficient is strongly influenced by scale length. In particular, the low alpha observed for the sleep and side effects domain may partly reflect the limited number of items in that subscale rather than reduced internal consistency alone. Fourth, the use of a web-based platform and the need for staff assistance during questionnaire completion may limit generalizability to other digital environments. Despite these limitations, this study provides useful implementation-level information regarding the practical challenges of introducing ePRO instruments into routine respiratory care. The psychometric findings reported here should therefore be interpreted as preliminary observations generated within a feasibility study, rather than as confirmatory evidence of measurement validity. In addition, the correlation analyses were based on only seven participants and were therefore severely underpowered. For this reason, the absence of statistically significant correlations should not be interpreted as evidence of lack of convergent validity, but rather as an inconclusive finding in an exploratory pilot context.
Conclusion
In conclusion, this feasibility study demonstrates that successful linguistic translation of a PRO measure does not necessarily ensure comparable measurement performance after electronic implementation in routine clinical practice. The findings suggest that factors related to mode of administration may influence measurement properties; however, this possibility should be regarded as preliminary and requires confirmation in adequately powered studies designed to assess measurement equivalence.
Supplemental material
Supplemental material - Development and feasibility of a Japanese electronic S3-NIV questionnaire in routine respiratory care
Supplemental material for Development and feasibility of a Japanese electronic S3-NIV questionnaire in routine respiratory care by Koichi Nishimura, Masaaki Kusunose, Toru Oga, Shonosuke Tajima, Tomoya Ogawa, Fumiko Watanabe, Sawako Okamoto, and Tomoki Kimura in Digital Health.
Supplemental material
Supplemental material - Development and feasibility of a Japanese electronic S3-NIV questionnaire in routine respiratory care
Supplemental material for Development and feasibility of a Japanese electronic S3-NIV questionnaire in routine respiratory care by Koichi Nishimura, Masaaki Kusunose, Toru Oga, Shonosuke Tajima, Tomoya Ogawa, Fumiko Watanabe, Sawako Okamoto, and Tomoki Kimura in Digital Health.
Supplemental material
Supplemental material - Development and feasibility of a Japanese electronic S3-NIV questionnaire in routine respiratory care
Supplemental material for Development and feasibility of a Japanese electronic S3-NIV questionnaire in routine respiratory care by Koichi Nishimura, Masaaki Kusunose, Toru Oga, Shonosuke Tajima, Tomoya Ogawa, Fumiko Watanabe, Sawako Okamoto, and Tomoki Kimura in Digital Health.
Supplemental material
Supplemental material - Development and feasibility of a Japanese electronic S3-NIV questionnaire in routine respiratory care
Supplemental material for Development and feasibility of a Japanese electronic S3-NIV questionnaire in routine respiratory care by Koichi Nishimura, Masaaki Kusunose, Toru Oga, Shonosuke Tajima, Tomoya Ogawa, Fumiko Watanabe, Sawako Okamoto, and Tomoki Kimura in Digital Health.
Footnotes
Acknowledgements
The authors would like to thank Mr. Yusuke Nagae and Mr. Takuya Mishima (Department of Rehabilitation, Tosei General Hospital) for administering the cognitive debriefing, Dr. Takashi Hajiro (Department of Respiratory Medicine, Tenri Hospital) for his contribution to the initial forward translation, Ms. Naomi Shima (Freelance Translator, Otsu, Japan) for her work on the back translation, and Mr. Kazuhito Nakayasu (Data Research Section, Kondo Inc.) for the statistical examination. The electronic version of the S3-NIV questionnaire (English version) is available here (
) with the permission of Dr. Dan Adler, the original developer.
Ethical considerations
The cognitive debriefing phase was approved by the Tosei General Hospital Medical Ethics Committee (Approval No. 828). The multicenter electronic feasibility study was approved by the ethics committees of all participating institutions, including the NCGG Ethics Committee (Approval No. 1573).
Consent to participate
Written informed consent was obtained from all patients before the study.
Author contributions
The authors meet the authorship criteria recommended by the International Committee of Medical Journal Editors. They take responsibility for the integrity of the work as a whole, contributed to the writing and reviewing of the manuscript, and gave final approval for the version to be published. All authors made significant contributions to the work, including the conception, study design, execution, data acquisition, analysis, and interpretation, or a combination of these areas. They participated in drafting, revising, or critically reviewing the article, provided final approval of the version to be published, agreed on the journal to which the article was submitted, and accepted accountability for all aspects of the work.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was partly supported by the Research Funding for Longevity Sciences (25-23) from the National Center for Geriatrics and Gerontology (NCGG), Japan.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
Anonymized participant data will be made available upon reasonable request to the corresponding author.
Declaration of generative AI and AI-assisted technologies in the writing process
We state that generative AI-based language tools (ChatGPT by OpenAI) were used only to assist with revision of sentence structure, clarity, grammar, and overall readability of the English text, whereas all intellectual content, scientific interpretation, and final decisions remained entirely the responsibility of the authors.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
