Since the 1940s, measurement specialists have called for an empirical validation technique that combines con tent- and construct-related evidence. This study investi gated the value of such a technique. A self-assessment instrument designed to cover four traditional foreign language skills was administered to 1,404 college-level foreign language students. Four subject-matter experts were asked to provide item dissimilarity judgments, us ing whatever criteria they thought appropriate. The data from the students and the experts were examined sepa rately using multidimensional scaling followed by clus ter and discriminant analyses. Results showed that the structure of the data underlying both the student and ex pert scaling solutions corresponded closely to that speci fied in the instrument blueprint. In addition, using canonical correlation, a comparison of the two scaling solutions revealed a high degree of similarity in the two solutions.
Arabie, P., Carroll, J.D., & DeSarbo, W.J. (1987). Three-way scaling and clustering. Newbury Park CA: Sage.
3.
Bachman, L.F. (1990). Fundamental considerations in language testing . Oxford: Oxford University Press .
4.
Bachman, L.F., & Palmer, A.S. (1981). The construct validity of the FSI oral interview . Language Learning, 31, 67-86.
5.
Bachman, L.F., & Palmer, A.S. (1982). The construct validity of some components of communicative proficiency. TESOL Quarterly, 16, 449-465.
6.
Barrows, T., Ager, S.M., Bennett, M.F., Braun, H.I., Clark, J.D.L., Harris, L.G., & Klein, S.F. (1981). College students' knowledge and beliefs: A survey of global understanding. New Rochelle NY: Change Magazine Press.
7.
Birckbichler, D.W., Corl, K.A., & Deville, C.W. (1993). The dynamics of placement testing: Implications for articulation and program revision. In D. P. Benseler (Ed.), The dynamics of language program direction (pp. 155-171). Boston: Heinle & Heinle.
8.
Cramer, E.M., & Nicewander, W.A. (1979). Some symmetric, invariant measures of multivariate association. Psychometrika, 44, 43-54.
9.
Crocker, L.M., Miller, D., & Franks, E.A. (1989). Quantitative methods for assessing the fit between test and curriculum. Applied Measurement in Education, 2, 179-194.
10.
Cronbach, L.J. (1971). Test validation. In R. L. Thomdike (Ed.), Educational measurement (2nd ed.; pp. 443-507). Washington DC: American Council on Education.
Davidson, F., & Henning, G.A. (1985). A self-rating scale of English difficulty: Rasch scalar analysis of items and rating categories. Language Testing , 2, 164-179.
13.
Davison, M.L. (1985). Multidimensional scaling versus components analysis of test intercorrelations. Psychological Bulletin,97, 94-105.
Davison, M.L., & Srichantra, N. (1988). Acquiescence in components analysis and multidimensional scaling of self-rating items. Applied Psychological Measurement, 12, 339-351.
16.
Ebel, R.L. (1977). Comments on some problems of employment testing . Personnel Psychology, 30, 55-63.
17.
Fitzpatrick, A.R. (1983). The meaning of content validity. Applied Psychological Measurement, 7, 3-13.
18.
Guion, R.M. (1977). Content validity: The source of my discontent . Applied Psychological Measurement, 1, 1-10.
19.
Gulliksen, H. (1950). Intrinsic validity. American Psychologist , 5, 511-517.
20.
Heilenman, L.K. (1990). Self-assessment of second language ability: The role of response effects. Language Testing, 7, 174-201.
21.
Heilenman, L.K. (1991). Self-assessment and placement: A review of the issues. In R. V. Teschner (Ed.), Issues in language program direction: Assessing foreign language proficiency of undergraduates (pp. 93-114). Boston: Heinle & Heinle.
22.
Heiser, W.J., & Meulman, J. (1983). Constrained multidimensional scaling, including confirmation. Applied Psychological Measurement, 7, 381-404.
23.
Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
LeBlanc, R., & Painchaud, G. (1985). Self-assessment as a second language placement instrument. TESOL Quarterly, 19, 673-687.
26.
Levine, M.S. (1977). Canonical analysis and factor comparison. Beverly Hills: Sage.
27.
Likert, R. (1974). A method of constructing an attitude scale. In G. M. Maranell (Ed.), Scaling: A sourcebook for behavioral scientists (pp. 233-243). Chicago: Aldine.
28.
Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30, 955-966.
29.
Messick, S. (1989a). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5-11.
30.
Messick, S. (1989b). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.; pp. 13-103). New York: American Council on Education.
Moss, P.A. (1995). Themes and variations in validity theory. Educational Measurement: Issues and Practice, 14(2), 5-13.
33.
Napior, D. (1972). Nonmetric multidimensional techniques for summated ratings. In R. N. Shepard, A. K. Romney, & S. B. Nerlove (Eds.), Multidimensional scaling: Theory and applications in the behavioral sciences. Volume I: Theory (pp. 157-178). New York: Seminar Press.
34.
Oltman, P.K., Stricker, L.J., & Barrows, T.S. (1990). Analyzing test structure by multidimensional scaling . Journal of Applied Psychology, 75, 21-27.
35.
Oskarsson, M. (1989). Self-assessment of language proficiency: Rationale and applications. Language Testing, 6, 1-13.
36.
Osterlind, S.J. (1989). Constructing test items. Hingham MA: Kluwer-Nijhoff.
37.
Pedhazur, E.J. (1982). Multiple regression in behavioral research: Explanation and prediction (2nd ed.). New York: Holt, Rinehart and Winston.
38.
Reilly, R.R., & Warech, M.A. (1991). The validity and fairness of alternatives to cognitive tests. Unpublished manuscript, AT&T.
39.
Ross, R.T. (1934). Optimal orders in the method of paired comparisons. Journal of ExperimentalPsychology, 25, 414-424.
Shepard, L.A. (1993). Evaluating test validity. In L. Darling-Hammond (Ed.), Review of research in education, 19 (pp. 405-450). Washington DC: American Educational Research Association .
42.
Sireci, S.G. (1995, April). The central role of content representation in test validity. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
43.
Sireci, S.G., & Geisinger, K.F. (1992). Analyzing test content using cluster analysis and multidimensional scaling. Applied Psychological Measurement, 16, 17-31.
44.
Sireci, S.G., & Geisinger, K.G. (1995). Using subject matter experts to assess content representation: A MDS analysis. Applied Psychological Measurement, 19, 241-255.
45.
Spector, P.E. (1992). Summated rating scale construction: An introduction . Newbury Park CA: Sage.
46.
Spence, I. (1982). Incomplete experimental designs for multidimensional scaling. In R. G. Golledge & J. N. Rayner (Eds.), Proximity and preference: Problems in the multidimensional analysis of large data sets (pp. 29-46). Minneapolis : University of Minnesota Press.
47.
Spence, I. (1983). Monte carlo simulation studies. Applied Psychological Measurement, 7, 405-425.
48.
Stewart, D.K., & Love, W.A. (1968). A general canonical correlation index. Psychological Bulletin,70, 160-163.
49.
Tatsuoka, M.M. (1988). Multivariate analysis: Techniques for educational and psychological research (2nd ed.). New York: Macmillan.
Thompson, B. (1984). Canonical correlation analysis: Uses and interpretation . Beverly Hills CA: Sage.
52.
Thomdike, R.L. (1982). Applied psychometrics. Boston : Houghton-Mifflin.
53.
Tucker, L.T. (1961, November). Factor analysis of relevance judgments: An approach to content validity. Paper presented at the Invitational Conference on Testing Problems, Princeton NJ. [Reprinted in A. Anastasi (Ed.), Testing problems in perspective, 1966, (pp. 577-586). Washington DC: American Council on Education.]
54.
von Elek, T. (1985). A test of Swedish as a second language: An experiment in self-assessment. In Y. Lee, A. Fok, R. Lord, & G. Low (Eds.), New directions in language testing (pp. 47-57). Oxford: Oxford University Press.
55.
Williams, L.J., Cote, J.A., & Buckley, M.R. (1989). Lack of method variance in self-reported affect and perceptions at work: Reality or artifact?Journal of Applied Psychology, 74, 462-468.
56.
Yalow, E.S., & Popham, W.J. (1983). Content validity at the crossroads. Educational Researcher, 12(8), 10-14,21.
