Occupational therapists are interested in clients' performance of activities of daily living in real-life contexts. To maximize the validity of inferences made from assessments, occupational therapists conduct observations in everyday environments: at home, school, and work and in the community. Although these observations in context are arguably more valid, they are potentially less controlled, which affects the reliability of the data obtained. Occupational therapists may wonder about the evidence of reliability when evaluating observational assessments for use or question reliability considerations in developing protocols. This article explores the concept of reliability in the context of observational assessments. We compare two theoretical approaches to reliability; apply the theory to the example of occupational performance, as viewed by occupational therapists; and argue the relevance of internal consistency, observer (rater) reliability, and sample reliability. Finally, the authors examine in-depth factors influencing the sample reliability of observational assessment scores, weighed against the utility of assessments and generalizability of scores.
American Occupational Therapy Association. (2002). Occupational therapy practice framework: Domain and process. Bethesda, MD: Author.
2.
BaumC.ChristiansenC. (1997). The occupational therapy context: Philosophy—principles—practice. In ChristiansenC.BaumC. (Eds.), Occupational therapy: Enabling function and well-being (2nd ed., pp. 26–45). Thorofare, NJ: SLACK Incorporated.
3.
BrentnallJ.BundyA. C.KayF. S. (in press). The effect of the length of observation on test of playfulness scores. OTJR: Occupation, Participation and Health, 28, 133–140.
4.
CairnsR. B.GreenJ. A. (1979). How to assess personality and social patterns: Observations or ratings? In CairnsR. B. (Ed.), The analysis of social interactions: Methods, issues, and illustrations (pp. 209–226). Hillsdale, NJ: Lawrence Erlbaum.
5.
ChapparoC.RankaJ. (1997). A description of constructs and structure. InOccupational performance model (Australia), Monograph 1 (pp. 1–23). Sydney, Australia: Occupational Performance Network.
6.
ChristiansenC.BaumC. (1997). Person-environment occupational performance: A conceptual model for practice. In ChristiansenC.BaumC. (Eds.), Occupational therapy: Enabling function and well-being (2nd ed., pp. 46–70). Thorofare, NJ: SLACK Incorporated.
7.
ConeJ. D. (1977). The relevance of reliability and validity for behavioral assessment. Behavior Therapy, 8, 411–426.
8.
ConeJ. D. (1997). Issues in functional analysis in behavioral assessment. Behavior Research and Therapy, 35(3), 259–275.
9.
ConeJ. D.FosterS. L. (1982). Direct observation in clinical psychology. In KendallP. C.ButcherJ. N. (Eds.), Handbook of research methods in clinical psychology (pp. 311–354). New York: John Wiley & Sons.
10.
EpsteinS. (1979). The stability of behavior: I. On predicting most of the people much of the time. Journal of Personality and Social Psychology, 37, 1097–1126.
11.
EpsteinS. (1980). The stability of behavior: II. Implications for psychological research. American Psychologist, 35, 790–806.
12.
FisherA. G. (2006). Assessment of Motor and Process Skills. Vol. 1: Development, Standardization, and Administration Manual (6th ed.). Fort Collins, CO: Three Star Press.
13.
GardnerF. (2000). Methodological issues in the direct observation of parent—child interaction: Do observational findings reflect the natural behavior of participants?Clinical Child and Family Psychology Review, 2, 185–198.
14.
Groth-MarnatG. (2003). Handbook of psychological assessment (4th ed.). Hoboken, NJ: John Wiley & Sons.
15.
Guide for the Uniform Data Set for Medical Rehabilitation (including the FIM instrument): Version 5.1. (1997). Buffalo, NY: State University of New York at Buffalo.
16.
HartmannD. P.BarriosB. A.WoodD. D. (2004). Principles of behavioral observation. In HaynesS. N.HeibyE. M. (Eds.), Behavioral assessment (vol. 3, pp. 108–127). Hoboken, NJ: John Wiley & Sons.
17.
HartmannD. P.WoodD. D. (1990). Observational methods. In BellackA. S.HersenM.KazdinA. E. (Eds.), International handbook of behavior modification and therapy (2nd ed., pp. 107–149). New York: Plenum Press.
18.
HeibyE. M.HaynesS. N. (2004). Introduction to behavioral assessment. In HaynesS. N.HeibyE. M. (Eds.), Behavioral assessment (vol. 3, pp. 3–18). Hoboken, NJ: John Wiley & Sons.
19.
HolmM. B.RogersJ. C.StoneR. G. (2003). Persontask-environment interventions: A decision-making guide. In CrepeauE. B.CohnE. S.SchellB. A. B. (Eds.), Willard and Spackman's occupational therapy (10th ed., pp. 460–480). Philadelphia: Lippincott, Williams & Wilkins.
20.
JohnstonJ. M.PennypackerH. S. (1980). Strategies and tactics of human behavioral research. Hillsdale, NJ: Lawrence Erlbaum Associates.
21.
LawM. (1987). Measurement in occupational therapy: Scientific criteria for evaluation. Canadian Journal of Occupational Therapy, 54, 133–138.
22.
LawM.CooperB. A.StrongS.StewartD.RigbyP.LettsL. (1996). The person-environment-occupation model: A transactive approach to occupational performance. Canadian Journal of Occupational Therapy, 63, 9–23.
23.
LeyendeckerB.LambM. E.ScholmerichA. (1997). Studying mother-infant interaction: The effects of context and length of observation in two subcultural groups. Infant Behavior and Development, 20, 325–337.
24.
MudfordO. C.BealeI. L.SinghN. N. (1990). The representativeness of observational samples of different durations. Journal of Applied Behavior Analysis, 23, 323–331.
25.
NelsonR. O.HayL. R.HayW. M. (1977). Comments on Cone's “the relevance of reliability and validity for behavioral assessment.”Behavior Therapy, 8, 427–430.
26.
PellegriniA. D. (1996). Observing children in their natural worlds: A methodological primer. Mahwah, NJ: Lawrence Erlbaum Associates.
27.
RigbyP.LettsL. (2003). Environment and occupational performance: Theoretical considerations. In LettsL.RigbyP.StewartD. (Eds.), Using environments to enable occupational performance (pp. 17–32). Thorofare, NJ: SLACK Incorporated.
28.
SattlerJ. M. (1992). Assessment of children (Revised and updated 3rd ed.). San Diego: Author.
29.
SeiferR.SameroffA. J.BarrettL. C.KrafchukE. (1994). Infant temperament measured by multiple observations and mother report. Child Development, 65, 1478–1490.
30.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Thousand Oaks, CA: SAGE Publications.
31.
SkardG.BundyA. C. (2008). Play and playfulness: What to look for. In ParhamL. D.FazioL. S. (Eds.), Play in occupational therapy for children (2nd ed., pp. 71–93). St Louis, MO: Mosby.
32.
StewartD.LawM. (2003). The environment: Paradigms and practice in health, occupational therapy and inquiry. In LettsL.RigbyP.StewartD. (Eds.), Using environments to enable occupational performance (pp. 3–15). Thorofare, NJ: SLACK Incorporated.
33.
StoolmillerM.EddyJ. M.ReidJ. B. (2000). Detecting and describing preventive intervention effects in a universal school-based randomized trial targeting delinquent and violent behavior. Journal of Consulting and Clinical Psychology, 68, 296–306.
34.
StreinerD. L.NormanG. R. (2003). Health measurement scales: A practical guide to their development and use (3rd ed.). New York: Oxford University Press.
SuenH. K.RzasaS. E. (2004). Psychometric foundations of behavioral assessment. In HaynesS. N.HeibyE. M. (Eds.), Behavioral assessment (vol. 3, pp. 37–56). Hoboken, NJ: John Wiley & Sons.
37.
WachsT. D. (1987). Short-term stability of aggregated and nonaggregated measures of parental behavior. Child Development, 58, 796–797.
38.
YarrowM. R.WaxlerC. Z. (1979). Observing interaction: A confrontation with methodology. In CairnsR. B. (Ed.), The analysis of social interactions: Methods, issues, and illustrations (pp. 209–276). Hillsdale, NJ: Lawrence Erlbaum.
