Impact assessment on patients’ satisfaction and healthcare professionals’ commitment of software supporting Directly Observed Treatment,Short-course: A protocol proposal

Abstract

Doctors, nurses, and other healthcare professionals use software that affects the patients. Directly Observed Treatment, Short-course is the name given to the tuberculosis control strategy recommended by the World Health Organization. The main goal of this work is to propose a protocol for evaluating the impact of healthcare software supporting Directly Observed Treatment, Short-course on patients, healthcare professionals, and services. The proposed protocol consists of a set of instruments and steps. The instruments are reliable and validated existing questionnaires to be applied before and after using the software tool. The literature points out the need for standards on the software assessment. This is particularly critical when software affects patients directly. The present protocol is a universal tool to assess the impact of software used to support the fight against the tragedy of tuberculosis where a rigorous evaluation of IT in healthcare is highly recommended and of great importance.

Keywords

e-health healthcare policy healthcare professional training IT healthcare evaluation quality control

Introduction

There is evidence for the positive impact on the patients and healthcare professionals of the investment in healthcare information systems (HIS).¹ When the software is considered a medical device^2–4 and the expenditure in healthcare has a tendency to grow,⁵ a rigorous evaluation of software in healthcare is needed. Studies on the evaluation of the health systems^6–8,13 have identified that there still are key challenges to overcome and have concluded that a broadly accepted framework for software evaluation in healthcare is necessary. This evaluation must focus, not only, but essentially on users.^9,10 Accordingly, we justify the need of instruments, a model and a methodological approach to assess the impact of the use of software on the patients, healthcare professionals and services.⁶

About one-third of the world’s population has latent tuberculosis (TB) infection. Over 95 percent of TB deaths occur in low- and middle-income countries.¹¹ TB causes one-fifth of all deaths of people with HIV infection.¹¹ DOTS or Directly Observed Treatment, Short-course is the internationally recommended strategy for TB control that has been recognized as a highly efficient and cost-effective strategy.¹¹ Our main goal is to study the relationship between the adoption of software and the impact on the patient, on the professional, and on the services that deliver the DOTS. In this way, we propose a protocol for DOTS software evaluation.

The article, in the next section, introduces the concepts related with the approach, namely, the perspectives of assessment of satisfaction of software. The following sections detail the research question and objectives of the study, as well as the methods applied. The proposed protocol presents a set of instruments and a set of steps. The article follows with a discussion of the proposal and ends with summarizing the conclusions and future research work.

Background

It is possible to identify innumerous studies for evaluating healthcare information technology.^7,8,12 Ammenwerth et al.⁶ sum up the evaluation questions as follows: (1) Which information technology should be selected and installed? (2) What is the usability of the information technology? (3) What are the technical and system features, for example, performance or software quality, of the information technology that affects its use? (4) Do users accept the information technology and use it as intended? If not, why not? (5) How does the information technology affect structural or process quality (time saving, data quality, clinical workflow, and patient administration) with regard to different users (physicians, nurses and administrative staff)? Does it work effectively? If not, why not? (6) What are the effects of an information technology on the quality of care? (7) Are the patients satisfied with the information technology? and (8) What are the investment and operational costs of information technology? Is it cost-effective?

We will use Ammenwerth et al.’s¹⁴ definition of HIS evaluation: “the act of measuring or exploring attributes of a HIS (in planning, development, implementation, or operation), the result of which informs a decision to be made concerning that system in a specific context.” All along the years, it is possible to register the shift from pure technical evaluations to more complex models, including the importance of the human factor,¹⁵ and the social and organizational issues.^10,16 Payton et al.¹⁰ and Paul et al.⁹ emphasize the key importance of including the patient–user perspective in the evaluation of the systems. Regarding the questions of “What are effects of an information technology on the quality of care?” and “Are the patients satisfied with the information technology?” we can identify several key works, summed up in Table 1. It is possible to conclude the absence of a consistent instrument, model and methodological approach to evaluate the users’—patients and healthcare staff—perspective of HIS. If the use of software in healthcare has the purpose to improve the quality of healthcare services and the level of effectiveness, then it must be evaluated from these two perspectives and therefore we need a consistent protocol for that assessment.

Table 1.

Key works for evaluation of health information systems’ satisfaction.

Author(s) (year)	Title	Goals	Methods	Findings/conclusions
Rojas-Mendizabal et al. (2013)¹⁷	Toward a model for Quality of Experience (QoE) and Quality of Service (QoS) in e-health ecosystems	Proposal of a health ecosystem involving the quality of experience and quality of service in e-health systems	Literature revision	Their view is that the deployment of e-health projects centered only on technical requirements tends to provide marginal results. They pose that the starting stage must be the understanding of the patient and medical specialist needs, the understanding of the context and its social reality
Lazakidou et al. (2012)¹⁸	Patient satisfaction as part of quality assurance in e-health	Justifying the need of including patients’ satisfaction as part of quality assurance in e-health	Literature revision	Assessment of patient satisfaction with the process of care is important as a measure of quality care. The study of the effect of different providers’ characteristics in patients’ satisfaction would also be useful in improving the process of healthcare
Currell and Urquhart (2003)¹⁹	Nursing record systems: effects on nursing practice and healthcare outcomes (Review)	To assess the effects of nursing record systems on nursing practice and patient outcomes	Eight trials involving 1497 people included randomized controlled trials, questionnaires, interviews, patient satisfaction surveys and data quality comparison	No evidence was found of effects on practice attributable to changes in record systems. The research undertaken so far may have suffered both from methodological problems and faulty hypotheses. Qualitative nursing research to explore the relationship between practice and information use could be used as a precursor to the design and testing of nursing information systems

Research question

The research question is as follows: How can the impact of healthcare software supporting DOTS on patients, healthcare professionals, and services be assessed?

The goal of this research is to propose a protocol for DOTS software evaluation from the perspective of its impact on patients, healthcare professionals and services.

Methods

The first step was to perform a literature revision to identify the existing key works to evaluate the impact of healthcare software either on patients or on software professionals. This literature revision allowed us to identify the need of approaches to use similar instruments and methods to evaluate and compare different software tools. The second step was to perform a second literature review on instruments to measure: patient satisfaction, healthcare professional’s satisfaction, and TB service evaluation. The third step was the definition of criteria to select instruments for the protocol. The fourth step was the definition of criteria for the selection of the steps to include in the proposed protocol. The fifth and final step was the development of the protocol.

The protocol

The proposed protocol is composed of a set of instruments and a set of steps. The adoption of the instruments follows the following criteria: (1) instruments must be based on studies demonstrating their fundamentals of validity, namely, the number of the sample, internal consistency reliability, convergent validity, discriminant validity and predictive validity; (2) instruments used in several other studies published in renown publications; and (3) the instruments must assess patient satisfaction, and the impact on healthcare professionals and on the TB services.

The adoption of the steps for the present protocol follows the following criteria: (1) the steps should allow the use of the protocol in different geographies; (2) the protocol should include all procedures of the selected instruments; and (3) the protocol should indicate how to use the results of the application of each instrument.

Only few instruments meet these demanding requirements. Despite the large number of existing instruments and studies, only few met these criteria. Based on a thorough literature review, the selected instruments are the key renowned instruments that met the criteria and indicated in the literature.

The instruments

To evaluate the patients’ satisfaction, we propose the use of the following questionnaires: Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS)²⁰ and the Patient Satisfaction Questionnaire Short-Form (PSQ-18).²¹ The intent of the CAHPS Hospital Survey, also known as Hospital CAHPS or HCAHPS, is to provide a standardized survey instrument and data collection methodology for measuring patients’ perspectives of hospital care. The HCAHPS survey contains 21 patient perspectives on care and patient rating items that encompass nine key topics: (1) communication with doctors, (2) communication with nurses, (3) responsiveness of hospital staff, (4) pain management, (5) communication about medicines, (6) discharge information, (7) cleanliness of the hospital environment, (8) quietness of the hospital environment, and (9) transition of care. The survey also includes four screener questions and seven demographic items, which are used for adjusting the mix of patients across hospitals and for analytical purposes. The survey is 32-question long. The PSQ-18 has been validated for use in different settings. It was developed through rigorous research and abbreviated from much larger questionnaires,^22,23 maintaining internal consistency and reliability.²¹ PSQ-18, Likert scale questionnaire, proposes seven dimensions of patient satisfaction directed toward their healthcare professionals: (1) general satisfaction, (2) technical quality, (3) interpersonal manner, (4) communication, (5) financial aspects, (6) time spent with doctor, and (7) accessibility and convenience. Each domain is tested through different related questions, which is of substantial benefit when one aims to identify a particular area to improve on. The general satisfaction has strong correlation with the other domains, and thus, it is important to improve all.

Morisky Medication Adherence Scale, version with 8 questions (MMAS-8) evaluates the adherence to the medication.²⁴ Different tools have been used to evaluate and assess patients’ adherence to medication, and there is no “gold standard” measure of medication adherence.²⁵ MMAS-8 is proposed to evaluate the impact of the software on medication adherence.

To assess the impact of a software on the healthcare professionals and services, we propose the use of the Organizational Commitment Questionnaire (OCQ), the Computer System Usability Questionnaire (CSUQ), and the organizational dimensions and the performance of the services of the basic attention centers in the management of TB, Tuberculosis Primary Care Assessment Tool (TPCAT). Human resources in healthcare are defined as “the heart of the health systems of the countries,” “the most effective part of the health care services,” and “a critical component of the health policies.”²⁶ According to the World Health Organization (WHO), healthcare professionals are the people who protect and improve the health level of the society.²⁷ To assess whether a software affects the work of the healthcare professionals, we propose the OCQ.^28–32 Organizational commitment can be defined as the strong feeling of an individual as a part of the organization and defining himself or herself in that way.³³ Affective commitment expresses the emotional attachment of the employees. Numerous studies have examined the correlates of commitment. Positive associations were found between affective commitment and motivation, organizational citizenship behavior, job satisfaction, performance, and productivity.^34–36 The OCQ³² relates to the measurement of affective commitment and counts among the most frequently used measurement instruments.^37,38 The OCQ consists of 15 items, of which 6 are negatively poled. Positive associations were found with job satisfaction³⁹ and with the readiness to remain in the organization,⁴⁰ and negative correlations were found with the intention to leave the organization.^41,42 Maier and Woschée⁴³ prove the construct validity of the OCQ with the help of a confirmatory factor analysis by showing that the OCQ represents a construct that can be empirically distinguished from other work attitudes (job satisfaction and job involvement).

To assess the satisfaction with the usability of software in health, we propose the use of the CSUQ.⁴⁴ CSUQ was validated with 825 employees who worked at nine IBM development sites: Atlanta, Austin, Bethesda, Boca Raton, Dallas, Raleigh, Rochester, San Jose, and Tucson. The questionnaire works well in non-laboratory settings.⁴⁴

Concerning the healthcare professionals, we did not identify any reference tool to evaluate healthcare professionals’ satisfaction. However, there are several works to evaluate nurses and doctors.⁴⁵ Two works must be referred—Beaglehole and Dal Poz,⁴⁶ with the Public Health Workforce: Challenges and Policy Issues, and Mollahaliloğlu et al.,⁴⁷ with the Healthcare Employee Satisfaction Survey.

Finally, to evaluate the impact of software in the services, we will evaluate the organizational dimensions and the performance of the services of the basic attention centers in the management of TB by using the questionnaire developed by Tereza Villa and Ruffino-Netto (48,49) (TPCAT) Villa and Ruffino-Netto used an instrument of the Primary Care Assessment Tool (PCAT). This tool was formulated and validated to evaluate the critical issues of the primary care in industrialized countries, developed by the John Hopkins University (50 –55). This instrument was adapted and validated for Brazil by Macinko, Almeida, and Oliveira (55,56). The same was adapted to evaluate the TB attention by Villa and Ruffino-Netto (49).

The steps

The protocol consists of five steps. It starts with the definition of the sample (Step 1) and the adjustment of the instruments (Step 2), followed by their validity (Step 3). The next step is the application of the instruments (Step 4). The analysis of the results and expected outcomes is the fifth step (Step 5). Figure 1 presents the steps of the protocol.

Figure 1.

Five steps of the protocol.

The first step concerns the definition of the sample and where to apply the instruments. Statistical approaches are used to determine samples and the definition of the inclusion and exclusion criteria in the study. Having a control group is mandatory in order to guarantee that the software is the only factor that affects the final results. Steps 2 and 3 apply when the instruments need to be adapted to a particular geography and language. The instruments first need to be translated to the mother-tongue and tested before being used in the field. A “forward–backward” procedure should be used to translate the instruments. Retro-translation and contrast analysis to reduce the bias involved in the translation must also be applied. HCAHPS has several domains (constructs) regarding hospital internment and hospital discharge. As the majority of the TB patients do not need internment, and consequently have no discharge, these constructs are not used. We suggest three series of pre-test of the instruments with a sample of patients and professionals based on the similarity of the demographic characteristics of the sample that will be used. The pre-tests will allow (1) the replacement of several expressions by others more comprehensible for the patients, (2) the improvement of the procedures of application of the instruments leading to a normalization of the application of the questionnaires, (3) the developing of a checklist and procedures’ manual for the team that will apply the instruments, and (4) the training of the team. The application of the pre-tests should be recorded (video) and carefully post-analyzed by the research team. Patients participating in these pre-tests must sign a consent term. The next step, Step 3, consists of making the internal consistency analysis of the adapted instruments. Construct validity estimates the ability of an instrument to measure the underlying construct of interest.⁵⁷ Exploratory factor analysis (EFA) has traditionally been employed by researchers as a tool to determine the number of underlying dimensions in a data set by grouping variables that are correlated.⁵⁸ The technique of Confirmatory Factor Analysis analyzes a priori measurement models in which both the number of factors and their correspondence with the indicators are explicitly specified.⁵⁹ Reliability can be assessed by Cronbach’s alpha internal consistency reliability coefficients. Values greater than 0.6 are considered generally satisfactory, and those greater than 0.8 indicate high internal consistency. Step 4 consists of the application of the instruments. This step must follow a specific timeline. Using the (adjusted) instruments either in patients or in healthcare professionals, before and after the introduction of a software, and by comparing the results, it is possible to verify whether the use of the software has any impact, and whether it is positive or negative. The ideal scenario should be using the instruments for each patient and health professional before and after the use of the software. However, when applying the instruments, different patients will be in different moments of the treatment. Some will be at the beginning and some will be at the end, which makes it difficult to apply the instruments, exactly to the same patients, both before and after the use of the software. Therefore, the only possible solution is to apply to existent patients in the beginning and to existent patients in the end even if they are not the same. Thus, we propose that the first evaluation moment for all the patients, regardless of their status of treatment, should be in the first month. Then, we will introduce the software, training the professionals by teaching them how to use the system. After 3 months of consistent use of the system, we can consider that the professional is totally adapted to the tool, having overtaken the learning curve and adaptation. Figure 2 presents the timeline to adopt.

Figure 2.

Feasible timeline application of the protocol.

After the 3-month use of the software, a new application of the instruments, added with the CSUQ⁴⁴ to verify whether the usability has some correlation with the other results, should occur. Comparing the first measures with the second, we would find out whether the software had some impact on the patients, healthcare professionals, and the services. All the procedures of each instrument must be followed. Results come from the comparison between the first and the second moments of evaluation (Step 5). PSQ-18 and HCAHPS allow the patients’ satisfaction to be measured. A similar development is expected between the results of these two instruments, either positive or negative. MMAS-8 will offer the impact on the medication adherence. A progression in line with PSQ-18 and HCAHPS is also expected. The evolution in the professionals’ attitude is measured using the OCQ. CSUQ will help in understanding whether the evolution is related to the software. If OCQ demonstrates a positive evolution, it is expected that CSUQ shows a good level of satisfaction with DOTS software. The contrary may also be a possible outcome, that is, a negative evolution in OCQ with a correspondent negative evolution in the CSUQ. However, if these two curves are not aligned, we can still conclude that software has no impact on the professionals’ commitment. Finally, using (TPCAT), we expect a result in line with OCQ.

Discussion

The number of different instruments identified in the literature was significant. When several instruments met the criteria, the number of publications, degree, and generality of use were the final factors for the decision. This applies to the HCAHPS, CSUQ, and OCQ. The number of instruments may be an adverse factor in the application of the protocol. Each instrument has its own procedures and some are extensive. The number of steps is also considerable, but using validated instruments, most of them, already adapted and used in several geographies, allow us to minor the impact of Steps 2 and 3. The validity of the protocol relies on the validity of the instruments and on the standard procedures selected. It is a protocol that allows the practitioners in the field to use the protocol to measure the impact of the software on the patients, professionals, and services. It is a complete 360° tool, but grounded in reliable and solid sources. Creating one single validated instrument and reducing the number of steps are the goals of future work.

Conclusion

The patients’ and healthcare professionals’ perspective is a key factor in the evaluation of the healthcare software. The protocol proposal represents a new approach to the problem of the evaluation of the software by patients and health professionals. Several existing studies use different instruments and different methodological approaches to evaluate patients’ and professionals’ satisfaction. Therefore, we can conclude the relevance of the development of an instrument, a model, and a methodological approach that could be used in the evaluation of the impact of the healthcare software either on the patients’ satisfaction or on the professionals’ commitment. With a set of instruments and a set of steps and procedures organized in a protocol, it will be possible to define a way of standardizing the assessment of software for supporting DOTS. The steps allow the use of the protocol in different geographies, and the validated instruments reduce bias. Future studies should be designed to (1) identify the key determinant factors in the evaluation of the impact of a DOTS software for patients, healthcare professionals, and services; (2) develop and test a new instrument based on those factors and demonstrate its validity; and (3) propose a scale to determine the “good,” “bad,” and “fair” impact of a DOTS supporting software on the patients and professionals, allowing the comparison of different software applications.

Footnotes

Acknowledgements

The authors would like to thank the team of the Hospital das Clínicas (HC), Centro Saúde Escola (CSE), Simioni (SI) and Vila Virginia (VV), that participated with a clinical team in this research and provided the conditions for this work. They also acknowledge the city of Ribeirão Preto and CEP/Conet (CAAE: 44813815.6.0000.5440) for authorizing this research.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES, process 88881.068176 / 2014-01), Foundation of the Education Ministry of Brazil, and the Brazilian Ministry of Health (process 796767/2013).

References

Cline

Luiz

. Information technology systems in public sector health facilities in developing countries: the case of South Africa. BMC Med Inform Decis Mak 2013; 13: 13.

Miller

Gardner

. Recommendations for responsible monitoring and regulation of clinical software systems. J Am Med Informatics Assoc 1997; 4(6): 442–57.

Cooper

Pauley

. Healthcare software assurance. AMIA Annu Symp Proc 2006; 2006: 166–70.

Curts

. FDA-regulated validation in clinical and nonclinical environments. IEEE Eng Med Biol Mag 2007; 26(1): 91–93, 101.

World Health Organization. Global health expenditure database, http://www.who.int/health-accounts/ghed/en/ (accessed 15 July 2016).

Ammenwerth

Gräber

Herrmann

et al . Evaluation of health information systems—problems and challenges. Int J Med Inform 2003; 71(2–3): 125–135.

Ammenwerth

de Keizer

. An inventory of evaluation studies of information technology in health care: trends in evaluation research 1982–2002. Methods Inf Med 2005; 44: 44–56.

Yusof

Papazafeiropoulou

Paul

et al . Investigating evaluation frameworks for health information systems. Int J Med Inform 2008; 77(6): 377–385.

Paul

Ezz

Kuljis

. Healthcare information systems: a patient-user perspective. Health Syst 2012; 1(2): 85–95.

10.

Payton

Pare

Le Rouge

et al . Health care IT: process, people, patients and interdisciplinary considerations. J Assoc Inf Syst 2011; 12(2): Article 4.

11.

World Health Organization. Tuberculosis fact sheet no. 104, 2015, http://www.who.int/mediacentre/factsheets/fs104/en/ (accessed 15 July 2016).

12.

Van der Loo

. Overview of published assessment and evaluation studies. In: Van Gennip

EMSJ

Talmon

(eds) Assessment and evaluation of information technologies in medicine, vol. 17. Amsterdam: IOS Press, 1995, pp. 261–282.

13.

Yusof

Paul

Stergioulas

. Health information systems evaluation: a focus on clinical decision supports system. Stud Health Technol Inform 2005; 116: 855–860.

14.

Ammenwerth

Brender

Nykanen

et al . Visions and strategies to improve evaluation of health information systems: reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inform 2004; 73(6): 479–491.

15.

Gremy

Fessler

Bonnin

. Information systems evaluation and subjectivity. Int J Med Inform 1999; 56: 13–23.

16.

Kaplan

Shaw

. Future directions in evaluation research: people, organizational, and social issues. Methods Inf Med 2004; 43(3): 215–231.

17.

Rojas-Mendizabal

Serrano-Santoyo

Conte-Galvan

et al . Toward a model for quality of experience and quality of service in e-health ecosystems. Procedia Technol 2013; 9: 968–974.

18.

Lazakidou

Ilioudi

Tsironi

. Patient satisfaction as part of quality assurance in E-health. Int J Reliab Qual E-Healthcare 2012; 1(1): 1–12.

19.

Currell

Urquhart

. Nursing record systems: effects on nursing practice and health care outcomes. In: Currell

(ed.). The cochrane database of systematic reviews. Chichester: John Wiley & Sons, Ltd, 2003, p. CD002099.

20.

National Quality Forum. Hospital Consumer Assessment of Healthcare Providers and Systems (H-CAHPS), 2013, http://www.hcahpsonline.org/home.aspx

21.

Marshall

Hays

. The Patient Satisfaction Questionnaire Short-Form (PSQ-18). St Monica, CA: Rand Corporation, 1994, http://www.rand.org/content/dam/rand/pubs/papers/2006/P7865.pdf

22.

Marshall

Hays

Sherbourne

et al . The structure of patient satisfaction with outpatient medical care. Psychol Assess 1993; 5(4): 477–483.

23.

Ware

Snyder

Wright

et al . Defining and measuring patient satisfaction with medical care. Eval Program Plann 1983; 6(3–4): 247–263.

24.

Morisky

Ang

Krousel-Wood

et al . Predictive validity of a medication adherence measure in an outpatient setting. J Clin Hypertens (Greenwich) 2008; 10: 348–354.

25.

Donnan

MacDonald

Morris

. Adherence to prescribed oral hypoglycaemic medication in a population of patients with Type 2 diabetes: a retrospective cohort study. Diabet Med 2002; 19(4): 279–284.

26.

Hongoro

McPake

. How to bridge the gap in human resources for health. Lancet 2004; 364(9443): 1451–1456.

27.

World Health Organization. The World Health Report 2006—working together for health, 2006, http://www.who.int/whr/2006/en/

28.

Kanning

Hill

. Validation of the Organizational Commitment Questionnaire (OCQ) in six languages. J Bus Media Psychol 2013; 4(2): 11–22.

29.

Mowday

Steers

Porter

. The measurement of organizational commitment. J Vocat Behav 1979; 14(2): 224–247.

30.

Porter

Crampon

Smith

. Organizational commitment and managerial turnover: a longitudinal study. Organ Behav Hum Perform 1976; 15(1): 87–98.

31.

Porter

Steers

Mowday

et al . Organizational commitment, job satisfaction, and turnover among psychiatric technicians. J Appl Psychol 1974; 59(5): 603–609.

32.

Porter

Smith

. The etiology of organizational commitment (unpublished manuscript). Irvine, CA: University of California, Irvine, 1970.

33.

Meyer

Kam

Goldenberg

et al . Organizational commitment in the military: application of a profile approach. Mil Psychol 2013; 25(4): 381–401.

34.

Dávila

García

. Organizational identification and commitment: correlates of sense of belonging and affective commitment. Span J Psychol 2013; 15(1): 244–255.

35.

Dello Russo

Vecchione

Borgogni

. Commitment profiles, job satisfaction, and behavioral outcomes. Appl Psychol 2013; 62(4): 701–719.

36.

Gutierrez

Candela

Carver

. The structural relationships between organizational commitment, global job satisfaction, developmental experiences, work values, organizational support, and person-organization fit among nursing faculty. J Adv Nurs 2012; 68(7): 1601–1614.

37.

Mathieu

Bruvold

Ritchey

. Subcultural research on organizational commitment with the 15 OCQ invariant instrument. J Pers Sell Sales Manag 2000; 20(3): 129–138.

38.

Mathieu

Zajac

. A review and meta-analysis of the antecedents, correlates, and consequences of organizational commitment. Psychol Bull 1990; 108(2): 171–194.

39.

Caught

Shadur

Rodwell

. The measurement artifact in the Organizational Commitment Questionnaire. Psychol Rep 2000; 87(3 Pt 1): 777–788.

40.

Steers

. Antecedents and outcomes of organizational commitment. Adm Sci Q 1977; 22(1): 46–56.

41.

Pierce

Dunham

. Organizational commitment: pre-employment propensity and initial work experiences. J Manage 1987; 13(1): 163–178.

42.

Cooke

. Discriminant validity of the organizational commitment questionnaire. Psychol Rep 1997; 80(2): 431–441.

43.

Maier

Woschée

R-M

. Die affektive Bindung an das Unternehmen (1Teile dieser Arbeit wurden auf der 1. Tagung der Fachgruppe Arbeits- und Organisationspsychologie der DGPs. September 1999, in Marburg präsentiert; Die dargestellten Ergebnisse stammen aus der Längsschnittstudie). Zeit Arb Organ A&O 2002; 46(3): 126–136.

44.

Lewis

. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact 1995; 7(1): 57–78.

45.

While

Barriball

. Job satisfaction among nurses: a literature review. Int J Nurs Stud 2005; 42(2): 211–227.

46.

Beaglehole

Dal Poz

. Public health workforce: challenges and policy issues. Hum Resour Health 2003; 1(1): 4.

47.

Mollahaliloğlu

Kosdak

Taşkaya

. Health care employee satisfaction survey. Ankara, 2010, http://sbu.saglik.gov.tr/Ekutuphane/kitaplar/healthcare_employee_satisfaction_survey_turkey.pdf (accessed 15 July 2016).

48.

Villa

TCS

Ruffino-Netto

. uberculose - Pesquisas Operacionais. 1st ed. São Paulo: FUNPEC - Editora, 2009.

49.

Villa

TCS

Ruffino-Netto

. Questionário para avaliação de desempenho de serviços de atenção básica no controle da TB no Brasil. J Bras Pneumol 2009; 35(6).

50.

Barbara

. Primary care: concept, evaluation, and policy. New York: Oxford University Press, 1992.

51.

Cassady

Starfield

Hurtado

et al . Measuring consumer experiences with primary care. Pediatrics 2000; 105(4): 998–1003.

52.

Starfield

. Is primary care essential? Lancet 1994; 344(8930): 1129–1133.

53.

Leiyu

Starfield

. Validating the Adult Primary Care Assessment Tool. J Fam Pract 2001; 50(2): 161–175.

54.

Harzheim

Starfield

Rajmil

et al . Consistência interna e confiabilidade da versão em português do Instrumento de Avaliação da Atenção Primária (PCATool-Brasil) para serviços de saúde infantil [Internal consistency and reliability of Primary Care Assessment Tool (PCATool-Brasil) for child. Cad Saude Publica 2006; 22(8): 1649–1659.

55.

Macinko

Almeida

Oliveira

. Avaliação das características organizacionais dos serviços de atenção básica em Petrópolis: teste de uma metodologia [Evaluation of the primary care services organization in Petrópolis: a methodological test]. Saude & Debate 2003; 27(65): 243–256.

56.

Almeida

Macinko

. Validação de uma metodologia de avaliação rápida das características organizacionais e do desempenho dos serviços de atenção básica do sistema único de saúde (SUS) em nível local. Brasília: Organização Mundial da Saúde (OMS) Organização Pan-Americana da Saúde (OPAS) Ministério da Saúde do Brasil, 2006.

57.

Ellenbecker

Byleckie

. Home healthcare nurses’ job satisfaction scale: refinement and psychometric testing. J Adv Nurs 2005; 52(1): 70–78.

58.

Tabachnick

Fidell

. Using multivariate statistics. 6th ed. Pearson, 2012, 1024 pp., http://www.amazon.com/Using-Multivariate-Statistics-Barbara-Tabachnick/dp/0205849571/ref=sr_1_1?s=books&;ie=UTF8&qid=1439317188&sr=1–1 (accessed 15 July 2016).

59.

Kline

. Principles and practice of structural equation modeling. 3rd ed. The Guilford Press, 2010, 427 pp., http://www.amazon.com/Principles-Practice-Structural-Equation-Methodology/dp/1606238760 (accessed 15 July 2016).