The interrater agreement of visual judgments made from single-subject data was examined. Seventy-nine raters were given 21 single-subject graphs. Thirty-nine of the raters examined graphs containing single-subject data arrayed in the traditional format. The remaining 40 subjects reviewed AB graphs that were supplemented by a trend line. As measured by intraclass correlation coefficients, interrater agreement was higher for the trend line group than for the group relying only on visual analysis. There was a statistically significant correlation between the change in slope across the phases of the AB design and a score reflecting disagreement among raters in the visual analysis group. This relationship between change in slope and rater disagreement was not present in the trend line group. The results suggest that the low interrater agreement often associated with visual analysis of single-subject data may be improved by simple supplements to visually inspected charts.
BaerD. M. (1977). Perhaps it would be better not to know everything. Journal of Applied Behavior Analysis, 10, 167–172.
2.
BaileyD. B. (1984). Effects of lines of progress and semilogarithmic charts on ratings of charted data. Journal of Applied Behavior Analysis, 17, 359–365.
3.
BarlowD. H.HayesS. C.NelsonR. O. (1984). The scientist practitioner. Research and accountability in clinical and educational settings. New York: Pergamon.
4.
BarlowD. H.HersenM. (1984). Single-case experimental designs: Strategies for studying behavior change (2nd Ed.). New York: Pergamon.
5.
BartkoJ. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 460–472.
6.
BerkR. A. (1979). Generalizability of behavioral observations: A clarification of interobserver agreement and interobserver reliability. American Journal of Mental Deficiency, 83, 460–472.
7.
BloomM.FischerJ. (1982). Evaluating practice: Guidelines for the accountable professional. Englewood Cliffs, NJ: Prentice Hall.
8.
DeProsperoA.CohenS. (1979). Inconsistent visual analyses of intra-subject data. Journal of Applied Behavioral Analysis, 12, 573–579.
9.
EdgingtonE. S. (1982). Nonparametric tests for single-subject multiple schedule experiments. Behavior Assessment, 4, 83–91.
10.
ElashoffJ. D.ThoresenC. E. (1978). Choosing a statistical method for analysis of an intensive experiment. In KratochwillT. R. (Ed.), Single-subject research: Strategies for evaluating change (pp. 287–311). New York, Academic Press.
11.
FurlongM. J.WampoldB. (1981). Visual analysis of single-subject studies by school psychologists. Psychology in the Schools, 18, 80–86.
12.
FurlongM. J.WampoldB. (1982). Intervention effects and relative variation as dimensions in experts use of visual inference. Journal of Applied Behavior Analysis, 15, 415–421.
13.
GibsonG.OttenbacherK. (1988). Characteristics influencing the visual analysis of single-subject data: An empirical analysis. Journal of Applied Behavioral Science, 24, 298–313.
14.
GriggN. C.SnellM. E.LoydB. (1989). Visual analysis of student evaluation data: A qualitative analysis of teacher decision making. Journal of the Association for Persons with Severe Handicaps, 14, 23–32.
15.
HochbergY.BenjaminiY. (1990). More powerful procedures for multiple significance testing. Statistics in Medicine, 9, 811–818.
16.
HojemM. A.OttenbacherK. (1988). Empirical investigation of visual inspection versus trend-line analysis of single-subject data. Physical Therapy, 68, 983–988.
17.
HuitemaB. E. (1986). Statistical analysis and single-subject designs: Some misunderstandings. In PolingA.FuquaR. W. (Eds.). Research methods in applied behavior analysis (pp. 209–232). New York: Plenum.
18.
JonesR. R.VaughtR.WeinrottM. (1977). Time-series analysis in operant research. Journal of Applied Behavior Analysis, 10, 151–166.
19.
JonesR. R.WeinrottM.VaughtR. S. (1978). Effects of serial dependency on the agreement between visual and statistical inferences. Journal of Applied Behavior Analysis, 11, 277–283.
20.
KazdinA. E. (1982). Single-case research designs: Methods for clinical and applied settings. New York: Oxford University Press.
21.
KazdinA. E. (1983). Single-case research in child psychiatry. Journal of the American Academy of Child Psychiatry, 22, 422–432.
22.
KnappT. J. (1983). Behavior analysts' visual appraisal of behavior change in graphic display. Behavioral Assessment, 5, 155–164.
23.
LedolterJ. (1983). The study of time-series [Review]. Contemporary Psychology, 28, 151–158.
24.
MichaelJ. (1974). Statistical inference for individual organism research: Mixed blessing or curse?Journal of Applied Behavior Analysis, 7, 647–653.
25.
OdomS. L. (1988). Research in early childhood special education: Methodologies and paradigms. In OdomS. L.KarnesM. B. (Eds.) Early intervention for infants and children with handicaps (p. 1–22). Baltimore: Paul H. Brooks.
26.
ParsonsonB. S.BaerD. M. (1978). The analysis and presentation of graphic data. In KratochwillT. R. (Ed.), Single-subject research: Strategies for evaluating change (pp. 101–165). New York: Academic Press.
27.
ParsonsonB. S.BaerD. M. (1986). The graphic analysis of data. In PolingA.FuquaR. W. (Eds.), Research methods in applied behavior analysis: Issues and advances (pp. 157–186). New York: Plenum.
28.
SharpleyL. (1986). Fallibility in the visual assessment of behavioral interventions: Time-series statistics to analyze time-series data. Behavior Change, 3, 26–33.
29.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
30.
SidmanM. (1960). Tactics of scientific research. New York: Basic Books.
31.
TawneyJ. W.GastD. L. (1984). Single-subject research in special education. Columbus, OH: Charles E. Merrill.
32.
TindalG.DenoS. (1983). Factors influencing the agreement between visual and statistical analysis of time-series data. Washington, D.C.: Office of Special Education and Rehabilitative Services. (ERIC ED 236 194).
33.
WampoldB.FurlongM. J. (1981). The heuristics of visual inference. Behavioral Assessment, 3, 79–82.
34.
WhiteO. R.HaringN. G. (1980). Exceptional teaching (2nd Ed.). Columbus, OH: Charles E. Merrill.
35.
WilkinsonL. (1986). SYSTAT: The system for statistics. Evanston, IL: SYSTAT.
36.
WoleryM.BillingsleyF. F. (1982). The application of Reusky's Rn test to slope and level changes. Behavioral Assessment, 4, 93–103.
37.
WoleryM.HarrisS. R. (1982). Interpreting results of single-subject research designs. Physical Therapy, 62, 445–452.
