Test equivalence can be evaluated in terms of four aspects: psychometric, behavioral, experiential, and individual differences (i.e., relativity of equivalence) (Honaker, 1988). This study examined the psychometric properties of the Attitude Towards Computerized Assessment Scale (ATCAS) designed to assess one of these criteria, namely, experiential equivalence. Specifically, the ATCAS was developed to explore examinees' emotional, perceptual and attitudinal reactions towards computerized relative to conventional testing methods. The validity and reliability of the ATCAS is addressed by presenting research findings from two independent studies. An exploratory factor analysis (EFA) of the intercorrelations of responses on the 13-items comprising the ATCAS was conducted in Study 1 using data from 92 undergraduate psychology students. The results of the EFA yielded two qualitatively different aspects of the individual's perceptions and experience of computer-based testing (CBT): Ease of Use and CBT-Confidence. The second study used data collected from an Australian Defence Force (ADF) sample to further explore and then confirm the factor structure of the ATCAS. The second study was divided into two phases. Based on the responses of 201 ADF applicants, an EFA of the ATCAS supported the presence of the Ease of Use and CBT-Confidence factors. Based on the responses of 331 ADF applicants, a confirmatory factor analysis (CFA) of ATCAS items supported the two-factor model. In studies 1 and 2, internal consistency was evidenced for the ATCAS and its subscales with values ranging from .76 to .92. Implications for the study of experiential equivalence are discussed, while areas for future research are also considered.
Get full access to this article
View all access options for this article.
References
1.
American Psychological Association (APA). (1986). Guidelines for computer-based tests and interpretations.Washington, DC: Author.
2.
AnastasiA. (1990). Psychological testing (6th ed.). New York: Macmillan.
BerntF. M.BugbeeA. C., & ArceoR. D. (1990). Factors influencing student resistance to computer administered testing. Journal of Research on Computing in Education, 22(3), 265–275.
6.
BocijP., & GreasleyA. (1999). Can computer-based testing achieve quality and efficiency in assessment?International Journal of Educational Technology, 1(1).
7.
BresolinM. J. (1984). A comparative study of computer administration of the Minnesota Multiphasic Personality Inventory. Applied Psychological Measurement, 1, 543–549.
8.
BurkeM. J.NormandJ., & RajuN. S. (1987). Examinee attitudes toward computer-administered ability testing. Computers in Human Behavior, 3(2), 95–107.
9.
ChinC. H. L.DonnS., & ConryR. F. (1991). Effects of computer-based tests on the achievement, anxiety, and attitudes of Grade 10 science students. Educational and Psychological Measurement, 51(3), 735–745.
10.
FedericoP. A. (1992). Assessing semantic knowledge using computer-based and paper-based media. Computers in Human Behavior, 8(2–3), 169–181.
11.
FulcherG. (1999). Computerizing an English language placement test. ELT Journal, 53(4), 289–299.
12.
GigliottiR. J.SmergliaV. L.FalkR. F., & NeiswanderN. (1994). Computer-based testing in sociology: A description and evaluation. Teaching Sociology, 22, 32–39.
13.
GoldbergA. L. (2000). Test-level, item-level, and experiential differences on computerized and paper-and-pencil versions of a practice Graduate Record Exam (GRE). Unpublished doctoral dissertation, Boston College.
14.
HankP., & SchwnekmezgerP. (1993). Computer-assisted versus pencil-and-paper based self-monitoring: An analysis of experiential and psychometric equivalence. In FahrenbergJ. & MyrtekM. (Eds.), Ambulatory assessment: Computer-assisted psychological and psychophysiological methods in monitoring field studies (pp. 85–99). Seattle: Hogrefe & Huber.
15.
HedlJ. J.O'NeilH. F., & HansenD. N. (1973). Affective reactions toward computer-based intelligence testing. Journal of Consulting and Clinical Psychology, 40(2), 217–222.
16.
HoferP. (1985). Developing standards for computerized testing. Computers in Human Behavior, 1, 301–315.
17.
HoferP. J., & GreenB. F. (1985). The challenge of competence and creativity in computerized psychological testing. Journal of Consulting and Clinical Psychology, 53(6), 826–838.
18.
HonakerL. M. (1988). The equivalency of computerized and conventional MMPI administration: A critical review. Clinical Psychology Review, 8, 561–577.
19.
KaplanR. M., & SaccuzzoD. P. (1997). Psychological testing: Principles, applications and issues (4th ed.). Pacific Grove, CA: Brooks/Cole.
20.
LeggS. M., & BuhrD. C. (1992). Computerized adaptive testing with different groups. Educational Measurement: Issues and Practice, 11(2), 23–27.
21.
McDonaldA. S. (2002). The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments. Computers & Education, 39(3), 299–312.
22.
MoeK. C., & JohnsonM. F. (1988). Participants' reactions to computerized testing. Journal of Educational Computing Research, 4, 79–86.
23.
NunnallyJ. C. (1967). Psychometric theory.New York: McGraw-Hill.
24.
OgilvieR. W.TruskT. C., & BlueA. V. (1999). Student's attitudes towards computer testing in a basic science course. Medical Education, 33(11), 828–831.
25.
ParshallC. G., & KromreyJ. D. (1993, April 12–16). Computer testing versus paper-and-pencil testing an analysis of examinee characteristics associated with mode effect. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.
PowersD. E., & O'NeillK. (1993). Inexperienced and anxious computer users: Coping with a computer-administered test of academic skills. Educational Assessment, 1, 153–173.
28.
RasulisR.SchuldbergD., & MurtaghM. (1996). Computer-administered testing with the Rotter Incomplete Sentences Blank. Computers in Human Behavior, 12(4), 497–513.
29.
Richman-HirschW. L.Olson-BuchananJ. B., & DrasgowF. (2000). Examining the impact of administration medium on examinee perceptions and attitudes. Journal of Applied Psychology, 85(6), 880–887.
30.
RozenskyR. H.HonorL. F.RasinskiK.TovianS. M., & HerzG. I. (1986). Paper-and-pencil versus computer administered MMPI'S: A comparison of patient's attitudes. Computers in Human Behavior, 2, 111–116.
31.
SchaefferG. A.ReeseC. M.SteffenM.McKinleyR. L., & MillsC. N. (1993). Field test of a computer-based GRE General Test (GRF Board Rep. No. 88–08P). Princeton, NJ: Educational Testing Service.
32.
SchmittN.GillilandS. W.LandisR., & DevineD. (1993). Computer-based testing applied to selection of secretarial applicants. Personnel Psychology, 46(1), 149–165.
33.
SchulenbergS. E., & YutrzenkaB. A. (1999). The equivalence of computerized and paper-and-pencil psychological instruments: Implications for measures of negative affect. Behavior Research Methods, Instruments and Computers, 31(2), 315–321.
34.
SegallD. O., & MorenoK. E. (1999). Development of the computerized adaptive testing version of the Armed Services Vocational Aptitude Battery. In DrasgowF. & Olsen-BuchananJ. B. (Eds.), Innovations in computerized assessment (pp. 35–65). Mahwah, NJ: Erlbaum.
35.
SingletonC. H.HomeJ., & ThomasK. (1999). Computerized baseline assessment of literacy. Journal of Research in Reading, 22(1), 67–80.
36.
SmithB. L. (2003). Conventional versus computer-based administration of measures of cognitive ability: An analysis of psychometric, behaviourial, experiential and relativity of equivalence. Unpublished doctoral dissertation, University of Wollongong.
37.
SteeleD. J.Johnson PalenskyJ. E.LynchT. G.LacyN. L., & DuffyS. W. (2001). Learning preferences, computer attitudes, and test performance: Response to computer-aided instruction. Medical Education, 36, 225–232.
38.
StriekerL. J., & WilderG. Z. (2001). Examinees' attitudes about the TOEFL-CBT, possible determinants, and relationships with test performance (Research Rep. No. 01–01). Princeton, NJ: Educational Testing Service.
39.
TabachnickB. G., & FidellL. S. (2001). Using multivariate statistics (4th ed.). Boston, MA: Allyn and Bacon.
40.
TonidandelS.QuinonesM. A., & AdamsA. A. (2002). Computer-adaptive testing: The impact of test characteristics on perceived performance and test takers' reactions. Journal of Applied Psychology, 87(2), 320–332.
41.
VispoelW. P.RocklinT. R., & WangT. (1994). Individual differences and test administration procedures: A comparison of fixed-item, computerized-adaptive, and self-adapted testing. Applied Measurement in Education, 7(1), 53–79.
42.
WebsterJ., & CompeauD. (1996). Computer-assisted versus paper-and-pencil administration of questionnaires. Behavior Research Methods, Instruments and Computers, 28(4), 567–576.
43.
ZandvlietD., & FarragherP. (1997). A comparison of computer-administered and written tests. Journal of Research on Computing in Education, 29(4), 423–438.
