Two-Unit Reliability Analysis of Questionnaires Used in a Regulatory System

Abstract

An interjudge reliability test was conducted to evaluate the questionnaires used in the surveillance of residential care institutions. Because the reliability test was carried out as part of the routine surveillance program and not as part of a controlled experiment, it was subject to deviations from the optimal reliability test model. However, this nonpure design provided an opportunity to not only examine the reliability of the items in the surveillance tool, but also to gain a better understanding of the use of a reliability test in an "imperfect" field setting. Two different surveyor teams administered the 257 questions on the questionnaires to a representative sample of 32 institutions on two separate occasions. In order to explain the variance in the reliability scores, a multivariate analysis was conducted for two units of analysis: the surveillance questions and the institutions. Based on the results of the reliability test, changes were introduced to improve the questionnaires and their administration.

Get full access to this article

View all access options for this article.

References

Brown, M.D. 1988. Functional assessment of the elderly. Journal of Gerontological Nursing 94:13-7.

Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20:37-46.

Donabedian, A. 1991. Reflections on effectiveness of quality assurance . In Striving for quality of care, edited by R. H. Palmer , A. Donabedian , and G. J. Pover. Ann Arbor, MI: Health Administration Press.

Dynia, A. , R. Fleishman , A. Shapira , and V. Shirazi. 1992. The elderly as a source of information in monitoring the quality of institutional care. In Proceedings of the Ninth International Conference of the Israel Society for Quality Assurance, 610-6.

Fillenbaum, G.G. , and M.A. Singer. 1989. The development, validity and reliability of the OARS Multidimensional Functional Assessment Questionnaire. Journal of Gerontology 36:428-34.

Fleishman, R. , A. Tomer , and R. Schwartz. 1990. Standards in long-term care facilities . International Journal of Health Care Quality Assurance 3:4-15.

Fleishman, R. , G. Mizrahi , M. Bar-Giora , J. Mandelson , and F. Yuz. 1995. Surveillance of institutions for the semi-independent and frail elderly: A guide for surveyors. M-31-95. Jerusalem: JDC-Brookdale Institute.

Fleishman, R. , G. Mizrahi , A. Dynia , D. Walk , V. Shirazi , and A. Shapira. 1994. Improving regulation of care. The International Journal of Quality in Health Care 6:61-71.

Gibson, D.M. , and W. Aitkenhead. 1983. The elderly respondent. Research on Aging 5.283-96.

10.

Gwynne-Schmidt, M. 1975. Interviewing the "old-old." The Gerontologist 15:544-7.

11.

Hyman, H.H. , W.J. Cobb , J.J. Feldman , C.W. Hart , and C.H. Stember. 1954. Interviewing in social research. Chicago: University of Chicago Press.

12.

Kahn, R.L. , and C.F. Cannell. 1957. The dynamics of interviewing: Theory, technique and cases. New York: John Wiley.

13.

Kerr, D.M. , L. Kent , and T.C.M. Lam. 1985. Measuring program implementation with a classroom observation instrument. Evaluation Review 9:469-82.

14.

Kessner, D.M. , and C.E. Kalk. 1973. A strategy for evaluating health services . Washington, DC: Institute of Medicine, National Academy of Sciences.

15.

Lazarsfeld, P.F. 1972. Qualitative analysis. Boston : Allyn & Bacon.

16.

Maccoby, E.E. , and N. Maccoby. 1959. The interview: A tool of social science . In Handbook of social psychology, Vol. 1, edited by G. Lindzey , 449-87. Reading, MA: Addison-Wesley.

17.

Morris, J.N. , C. Hawes , B.E. Freis , C.D. Phillips , V. Mor , S. Katz , K. Murphy , M.L. Drugovich , and A.S. Friedlob. 1990. Designing the national resident assessment instrument for nursing institutions. The Gerontologist 30:293-307.

18.

Morton-Williams, J. 1993. Interviewer approaches. Aldershot : Dartmouth.

19.

Nunnally, J.C. 1978. Psychometric theory. 2nd ed. New York: McGraw-Hill.

20.

Nunnally, J.C. , and A. Durham. 1975. Validity, reliability and special problems of measurement in evaluating research. In Handbook of evaluation research, Vol. 1, edited by E. Straening and M. Guttentog , 283-352. Beverly Hills, CA: Sage.

21.

Perreault, W.D. , and L.E. Leigh. 1989. Reliability of nominal data based on qualitative judgements. Journal of Marketing Research 26:135-48.

22.

Prescott, P.A. , K.L. Soeken , and J.W. Ryan. 1989. Measuring patient intensity: A reliability study. Evaluation & the Health Professions 12:255-69.

23.

Rust, R.T. , and B. Cooil. 1994. Reliability measures for qualitative data: Theory and implication. Journal of Marketing Research 31:1-14.

24.

Suchman, E. 1967. Evaluation research. New York : Russell Sage.

25.

Sudman, S. , and N.M. Bradburn. 1974. Response effect in surveys. Chicago: Aldine.

26.

Spitznagel, E.L. , and J.E. Helzer. 1985. A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry 42:725-8.

27.

Zinn, J.S. , R. Lavizza-Mourey , and L. Taylor. 1993. Measuring satisfaction with care in the nursing home setting: The Nursing Home Resident Satisfaction Scale. Journal of Applied Gerontology 12:452-65.