This paper describes the development and psychometric properties of the Assessment Rating Profile-Revised (ARP-R), an instrument designed to evaluate the acceptability of assessment methods. First, an overview of the initial exploratory work on the original scale, the Assessment Rating Profile (APR; Kratochwill & Van Someren, 1984), is provided. Research examining the reliability and validity of the ARP is reviewed and psychometric inadequacies are highlighted. Next, the results of a study examining the psychometric properties of the revised measure are reported. The results of the study indicated that the ARP-R demonstrated good internal consistency, with coefficients ranging from .94 to .99. Test-retest reliability across 1-, 3-, 6-, and 12-month intervals rangedfrom .82 to .85, suggesting adequate stability over time. Confirmatory factor analytic procedures indicated a reasonable fit of the data to the proposed one-factor model, “General Assessment Acceptability.” Lastly, suggestions are made regarding use of the ARP-R in applied and research settings.
Get full access to this article
View all access options for this article.
References
1.
Allinder, R. M., & Oats, R. G. (1997). Effects of acceptability on teachers' implementation of curriculum-based measurement and student achievement in mathematical computation. Remedial and Special Education, 18, 113-120.
2.
Alwin, D. F., & Jackson, D. J. (1980). Measurement models for response enrors in surveys: Issues and applications. In K. F Schuessler (Ed.), Sociological methodology (pp. 68-119). San Francisco: Jossey-Bass.
3.
American Speech-Language-Hearing Association (1990). Guidelines for practice in early intervention. Rockville, MD: American Speech-Language-Hearing Association.
4.
Anderson, J. C., & Gerbing, D. W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fitindices for maximum likelihood confinmatory factor analysis. Psychometrka, 49, 155-173.
5.
Bender, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246.
6.
Bentler, P. M., & Bonnett, D. G. (1980). Significance tests and goodness-of-fit in the covariance structures. Psychological Bulletin, 88,588-606.
7.
Bollen, K. A. (1989). Structural equations with latent variables. New York, NY: John Wsley & Sons.
8.
Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37, 62-83.
9.
Carmines, E., & McIver, J. (1981). Analyzing models with unobserved variables: Analysis of covariance structures. In G. Bohrnstedt & E. Borgatta (Eds.), Social Measurement: Current Issues (pp. 65-115). Beverly Hills, CA: Sage.
10.
Chou, C. P., Bentler, P. M., & Satorra, A. (1991). Scaled test statistics and robust standard errors for nonnormal data in covariance structure analysis: A Monte Carlo study. British Journal of Mathematical and Statistical Psychology, 44, 347-357.
11.
Clark, L., & Elliott, S. N. (1988). The influence of treatment strength information on knowledgeable teachers' pretreatment evaluations of social skills training methods. Professional School Psychology, 3, 241-251.
12.
Cross Calvert, S., & McMahon, R. (1987). The treatment acceptability of a behavioral parent training program and its components. Behavior Therapy, 2, 165-179.
13.
de Mesquita, P. B., & Zoilman, A. (1995). Teachers' preferences for academic intervention strategies in mathematics: Implications for instructional consultation. Journal of Educational and Psychological Consultation, 6, 159-174.
14.
DEC Task Force on Recommended Practices (1993). DEC recommended practices: Indicators of quality in programs for infants and young children with special needs and theirfamilies. Reston, VA: Council for Exceptional Children.
15.
Dunteman, G. H. (1989). Principal components analysis. Newbury Park, CA: Sage Publishing.
16.
Eckert, T. L., & Hintze, J. M. (in press). Behavioral conceptions and applications of acceptability: Issues related to service delivery and research methodology. School Psychology Quarterly.
17.
Eckert, T. L., Hintze, J. M., & Shapiro, E. S. (1997). School psychologists' acceptability of psychological assessment measures: The acceptability of behavioral and traditional assessment. School Psychology Quarterly, 12, 150-169.
18.
Eckert, T. L., & Shapiro, E. S. (1999). Methodological issues in analog acceptability research: Are teachers' acceptability ratings of assessment methods influenced by experimental design?School Psychology Review, 27, 5-16.
19.
Eckert, T. L., Shapiro, E. S., & Lutz, J. G. (1995). Teachers' ratings of the acceptability of curriculum-based assessment methods. School Psychology Review, 24, 499-513.
20.
Elliott, S. N., Witt, J. C., & Kratochwill, T. R. (1991). Selecting, implementing, and evaluating classroom interventions. In G. Stoner, M. R. Shinn, and H. M. Walker (Eds.), Interventionsfor achievement and behavior problems (pp. 99-135). Silver Springs, MD: National Association of School Psychologists.
21.
Fassinger, R. E. (1987). Use of structural equation modeling in counseling psychology research. Journal of Counseling Psychology, 34, 425-436.
22.
Fincham, F. D., & Spettell, C. (1984). The acceptability of dry bed training and urine alarm training as treatments of nocturnal enuresis. Behavior Therapy, 15, 388-394.
23.
Fuchs, L. S., & Fuchs, D. (1992). Identifying a measure for monitoring student reading progress. School Psychology Review, 21, 45-58.
24.
Gable, R. K., & Wolf, M. B. (1993). Instrument development in the affective domain. Boston, MA: Kluwer Academic Publishers.
25.
Gerbing, D. W., & Anderson, J. C. (1987). Improper solutions in the analysis of covariance structures: Their interpretability and a comparison of alternate respecifications. Psychometrika, 52, 99-111.
26.
Greenspan, S. I., & Meisels, S. J. (1994). Toward a new vision for developmental assessment of infants and young children. Zero to Three, 14, 170-177.
27.
Hayes, S. N., Nelson, R. O., & Jarrett, R. B. (1987). The treatment utility of assessment. American Psychologist, 42, 963-974.
28.
Hunsley, J. (1993). Treatment acceptability of symptom prescription techniques. Journal of Counseling Psychology, 40, 139-143.
29.
Hu, L., Bentler, P. M., & Kano, Y. (1992). Can test statistics in covariance structure analysis be trusted?Psychological Bulletin, 2, 351-362.
30.
Jacob-Timm, S., & Hartshorne, T. (1994). Ethics for law and school psychologists. New York: John Wiley & Sons, Inc.
31.
Joreskog, K. G., & Sorbom, D. (1993). Lisrel 8 user's reference guide. Chicago: Scientific Software International.
32.
Kazdin, A. E. (1980). Acceptability of alternative treatments for deviant child behavior. Journal of Applied Behavior Analysis, 13, 259-273.
33.
Kaufman, A. S., & Kaufman, N. L. (1985). Kaufmnan Test of Educational Achievement. Circle Pines, MN: American Guidance System.
34.
Kim, J. O., & Mueller, C. W. (1978). Factor analysis: Statistical methods and practical issues. Newbury Park, CA: Sage Publishing.
35.
Kratochwill, T. R., & Van Someren, K. (1984). Assessment rating profile. Unpublished manuscript, University of Wisconsin-Madison.
36.
Kutsick, K. A., Gutkin, T. B., & Witt, J. C. (1991). The impact of treatment development process, intervention type, and problem severity on treatment acceptability as judged by classroom teachers. Psychology in the Schools, 28, 325-331.
37.
Lindsley, O. R. (1992). Why aren't effective teaching tools widely adopted?Journal of Applied Behavior Analysis, 25, 21-26.
38.
Long, J. S. (1983). Confirmatoryfactor analysis. Beverly Hills, CA: Sage.
39.
Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-ft indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391-410.
40.
Martens, B. K., Witt, J. C., Elliott, S. N., & Darveaux, D. (1985). Teacher judgments concerning the acceptability of school-based interventions. Professional Psychology, 16, 191-198.
41.
Meller, P. J., Martens, B. K., & Hurwitz, S. (1990). Variables influencing perceptions of liability: A case for treatment acceptability assessment. School Psychology Quarterly, 5, 237-255.
42.
Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30, 955-966.
43.
Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23, 13-23.
44.
Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50, 741-749.
45.
Mulaik, S. A., James, L. R., Van Alstine, J., Bennett, N., Lnd, S., & Stilwell, C. D. (1988). Evaluation of goodness-of-fit indices for structural equation models. Psychological Bulletin, 105, 430-445.
46.
National Center for Educational Statistics (1994). Digest of education statistics: 1994. Washington, DC: Government Printing Office.
47.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York, NY: McGraw-Hill.
48.
Prasse, D. A. (1990). Best practices in legal and ethical considerations. In A. Thomas & J. Grimes (Eds.), Best Practices in School Psychology-II (pp. 469-505). Washington, DC: National Association of School Psychologists.
49.
Reimers, T. M., & Lee, J. (1991). Parental acceptability of treatments for children's hypocholesterolemia. Journal of Behavioral Medicine, 14, 225-239.
50.
Reimers, T. M., & Wacker, D. P. (1988). Parents' ratings of the acceptability of behavioral treatment recommendations made in an outpatient clinic: A preliminary analysis of the influence of treatment effectiveness. Behavioral Disorders, 14, 7-15.
51.
Salvia, J., & Ysseldyke, J. E. (1995). Assessment. Boston, MA: Houghton Mifflin Company.
52.
Shapiro, E. S., & Eckert, T. L. (1994). Acceptability of curriculum-based assessment among school psychologists. Journal of School Psychology, 32, 167-183.
53.
Tanaka, J. S. (1987). “How big is enough?”: Sample size and goodness of fit in structural equation models with latent variables. Child Development, 58, 134-146.
54.
Tarnowski, K. J., Rasnake, L. K., Mulick, J. A., & Kelly, P. A. (1989). Acceptability of behavioral interventions for self-injurious behavior. American Journal on Mental Retardation, 93, 575-580.
55.
Tarnowski, K. J., Simonian, S. J., Bekeny, P., & Park, A. (1992). Acceptability of treatments for child behavioral disturbance: Race, socioeconomic status, and multicomponent treatment effects. Child and Family Behavior Therapy, 14, 25-37.
56.
Telzrow, C. F. (1995). Best practices in facilitating intervention adherence. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology (pp. 501-510). Washington, D.C.: The National Association of School Psychologists.
57.
Von Brock, M. B., & Elliott, S. N. (1987). The influence of treatment effectiveness information on the acceptability of classroom interventions. Journal of School Psychology, 25, 131-144.
58.
Wechsler, D. (1991). Wechsler Individual Achievement Test: Manual. New York, NY: Psychological Corporation.
59.
Wheaton, B., Muthen, B. Alwin, D., & Summers, G. (1977). Assessing reliability and stability in panel models. In D. Heise (Ed.), Sociological Measurement (pp. 84-136). San Francisco, CA: Jossey-Bass.
60.
Witt, J. C., & Elliott, S. N. (1985). Acceptability of classroom intervention strategies. In T. R. Kratochwill (Ed.), Advances in school psychology (Vol. 4, pp. 251-288). Hillsdale, NJ: Lawrence Erlbaum.
61.
Witt, J. C., & Martens, B. K. (1983). Assessing the acceptability of behavioral interventions used in classrooms. Psychology in the Schools, 20, 510-517.
62.
Wolins, L. (1995). A Monte Carlo study of constrained factor analysis using maximum likelihood and unweighted least squares. Educational and Psychological Measurement, 55, 545-557.
