Principles of evidential reasoning have often been discussed in the context of educational and psychological measurement with respect to construct validity and validity arguments. More recently, Mislevy proposed the metaphor of assessment as an evidentiary argument about students’ learning and abilities given their behavior in particular circumstances. An assessment argument consists of a claim one wants to make, typically about student learning, and evidence that supports that claim. From this perspective, the quality of our assessments are a function of both whether we have built our arguments about the right types of claims and whether we have collected sufficient persuasive evidence to support our claims.
Purpose
This paper examines limitations of the dominant practice in educational assessment of the 20th century, which focuses on relatively simple claims and often rely on a single piece of evidence. This paper considers future educational assessment in terms of principles of evidential reasoning, focusing the discussion on the changes to the claims our assessments must support, the types of evidence needed to support these claims, and the statistical tools available to evaluate our evidence vis-à-vis the claims. An expanded view of assessment is advanced in which assessments based on multiple evidence sources from contextually rich situated learning environments, including unconventional data regarding human competencies, improve our ability to make valid inferences and decisions about all education stakeholders.
Conclusions
For educational assessment to have the positive impact we intend on educational outcomes, future assessments must leverage technological and computational developments, as well as more contemporary models of human cognition, to build robust complex evidential arguments about the critical competencies that are likely to determine individuals’ success in 21st century society.
Get full access to this article
View all access options for this article.
References
1.
American Psychological Association, American Educational Research Association, & National Council on Measurement in Education (APA/AERA/NCME). (1999). Standards on Educational and Psychological Testing. Retrieved from http://www.apa.org/science/programs/testing/standards.aspx
2.
BehrensJ. T., MislevyR. J., BauerM., WilliamsonD. M., & LevyR. (2004). Introduction to evidence-centered design and lessons learned from its application in a global e-learning program. International Journal of Testing, 4, 295–302.
3.
BennettR. E. (2010). Cognitively based assessment of, for, and as learning: A preliminary theory of action for summative and formative assessment. Measurement: Interdisciplinary Research and Perspectives, 8, 70–91.
4.
BennettR. E. (2011). Formative assessment: A critical review. Assessment in Education: Principles, Policy and Practice, 18, 5–25.
Casner-LottoJ. (2006). Are they really ready to work?: Employers’ perspectives on the basic knowledge and applied skills of new entrants to the 21st century U.S. workforce. New York: Conference Board.
7.
ChenY. H., GorinJ. S., ThompsonM. S., & TatsoukaK. K. (2008). Cross-cultural validity of the TIMSS-1999 Mathematics Test: Verification of a cognitive model. International Journal of Testing, 8(3), 251–271.
8.
ConatiC., GerinerA., & Van LehnK. (2002). Using Bayesian networks to manage uncertainty in student modeling. User Modeling & User-Adapted Interaction, 72(4), 371–417.
9.
CorterJ., & TatsuokaK. K. (2004). Patterns of diagnosed mathematical content and process skills in TIMSS-R across a sample of 20 countries. American Educational Research Journal, 41(4), 901–926.
10.
CroccoM. S., & CostiganA. T. (2007). The narrowing of curriculum and pedagogy in the age of accountability: Urban educators speak out. Urban Education, 42, 512–535.
11.
CronbachL. J. (1989). Construct validation after thirty years. In CronbachL. J. (Ed.), Intelligence: Measurement, theory, and public policy: Proceedings of a symposium in honor of Lloyd G. Humphreys (pp. 147–171). Champaign, IL: University of Illinois Press.
12.
CrossD. R., & ParisS. G. (1987). Assessment of reading comprehension: Matching test purposes and test properties. Educational Psychologist, 22(3&4), 313–332.
13.
de BoeckP., & WilsonM. (Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York: Springer.
14.
de la TorreJ., & DouglasJ. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.
15.
De MarsC. E. (2007). Changes in rapid-guessing behavior over a series of assessments. Educational Measurement, 12(1), 23–45.
16.
ElliottS. N., & RoachA. T. (2007). Alternate assessments of students with significant disabilities: Alternative approaches, common technical challenges. Applied Measurement in Education, 20(3), 301–333.
17.
Formative Assessment for Students and Teachers (FAST) State Collaborative on Assessment and Student Standards (SCASS). (2008, October). Attributes of effective formative assessment. Paper prepared for the Formative Assessment for Teachers and Students State Collaborative on Assessment and Student Standards of the Council of Chief State School Officers. Washington, DC: Council of Chief State School Officers.
18.
GeeJ. P. (2007). Reflections on assessment from a sociocultural-situated perspective. In MossP. A. (Ed.), Evidence and decision making (pp. 362–375). Oxford: Blackwell Publishing.
19.
GeeJ. P. (2010). Human action and social groups as the natural home of assessment: Thoughts on 21st century learning and assessment. In ShuteV. J. & BeckerB. J. (Eds.), Innovative assessment for the 21st century: Supporting educational needs (pp. 13–39). New York: Springer Science.
20.
GoldsteinJ., & BehuniakP. (2011). Assumptions in alternate assessment: An argument-based approach to validation. Assessment for Effective Intervention, 36(3), 179–191
21.
GordonE. W (2007). Intellective competence. Voices in Urban Education: Towards Proficiency, 14.
22.
GorinJ. S. (2006, April). Using alternative data sources to inform item difficulty modeling. Paper presented at the Annual Meeting of the National Council on Educational Measurement, San Francisco, CA.
23.
GorinJ. S. (2007). Test construction and diagnostic testing. In LeightonJ. P. & GierlM. J. (Eds.), Cognitive diagnostic assessment in education: Theory and practice (pp. 173–204). Cambridge: Cambridge University Press.
24.
IvieJ. L., KupzykK. A., & EmbrestonS. E. (2004). Final report of Cognitive Components Study— Predicting strategies for solving multiple-choice quantitative reasoning items: An eyetracker study. Princeton, NJ: Educational Testing Services and Lawrence, KS: University of Kansas.
25.
KaneM. (1992). An argument-based approach to validation. Psychological Bulletin, 112, 527–535.
26.
KaufmanA. S. (1979). Intelligent testing with the WISC-R. New York: Wiley.
27.
LeeY-H., ChenH. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53, 359–379.
28.
LevyR. (2011a). Evidentiary reasoning in diagnostic classification models. Measurement: Interdisciplinary Research and Perspectives, 7(1), 36–41.
29.
LevyR. (2011b). Posterior predictive model checking for conjunctive multidimensionality in item response theory. Journal of Educational and Behavioral Statistics, 36(5), 672–694.
30.
LordF. M., & NovickM. R. (1968). Statistical theories of mental test score. Reading, MA: Addison-Wesley Publishing Inc.
31.
LurieS. (2011). Towards greater clarity in the role of ambiguity in clinical reasoning. Medical Education, 45(4), 326–328.
32.
MessickS. (1989). Validity. In LinnR. L. (Ed.) Educational Measurement (3rd ed., pp. 13–103). New York: American Council on Education/Macmillan Publishing.
33.
MislevyR. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439–483.
34.
MislevyR. J. (2012). Four metaphors we need to understand assessment. Princeton, NJ: The Gordon Commission on the Future of Assessment in Education.
35.
MislevyR. J., & LevyR. (2007). Bayesian psychometric modeling from an evidence-centered design perspective. In RaoC. R. and SinharayS. (Eds.), Handbook of statistics, Volume 26 (pp. 839–865). North-holland: Elsevier.
36.
MislevyR. J., & YinC. (2009). If language is a complex adaptive system, what is language assessment. Language Learning, 59(1), 249–267.
37.
National Research Council (NRC). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
38.
RodriguezM. C. (2009). Psychometric considerations for alternate assessment based on modified academic achievement standards. Peabody Journal of Education, 84, 595–602.
39.
RuppA., TemplinJ., & HensonR. (2010). Diagnostic measurement: Theory, methods, and applications. New York: Guilford Press.
40.
SanchezJ. G., ChristophersonR., EcheagarayM. E. C., GibsonD. C., AtkinsonR. K., & BurlesonW. (2011). How to do multimodal detection of affective states?ICALT, 654–655.
41.
SchaferW. D., & LissitzR. W. (2009). Alternate assessments based on alternate achievement standards: Policy, practice, and potential. Baltimore, MD: Brookes Publishing Company.
42.
SchnipkeD. L. & ScramsD. J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In MillsC. N., PotenzeM., FremerJ. J., & WardW. (Eds.), Computer-based testing: Building the foundation for future assessments (pp. 237–266). Hillsdale, NJ: Lawrence Erlbaum Associates.
43.
SchumD. A. (2001). The evidential foundations of probabilistic reasoning.Evanston, IL: Northwestern University Press.
44.
ShermisM. D., & BursteinJ. (2003). Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Lawrence Erlbaum.
45.
ShuteV. J. (2011). Stealth assessment in computer-based games to support learning. In TobiasS. & FletcherJ. D. (Eds.), Computer games and instruction (pp. 503–524). Charlotte, NC: Information Age Publishing.
46.
SilvaE. (2008). Measuring skills for the 21st century.Washington, DC: Education Sector Reports.
47.
ToulminS. E. (1958). The uses of argument. Cambridge: Cambridge University Press.
48.
WilliamsonD. M., BauerM., SteinbergL. S., MislevyR. J., BehrensJ. T., & DeMarkS. F. (2004). Design rationale for a complex performance assessment. International Journal of Testing, 4, 303–332.
49.
WiseS., & DeMarsC. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19–38.
50.
WodrichD. L., & SchmittA. J. (2006). Patterns of learning disorders: Working systematically from assessment to intervention. New York: Guilford.
51.
WylieE. C. (2012, June). Using learning progressions to support item and task development for formative purposes in middle school mathematics. Presentation made at the Chief Council of State School Officers (CCSSO) National Conference on Student Assessment, Minneapolis, MN.
52.
XuD. L., YangJ. B., & WangY. M. (2006). The ER approach for multi-attribute decision analysis under interval uncertainties. European Journal of Operational Research, 174(3), 1914–1943.
53.
YangJ. B., & XuD. L. (2002). On the evidential reasoning algorithm for multiple attribute decision analysis under uncertainty. IEEE Transactions on Systems, Man and Cybernetics Part A: Systems and Humans, 32(3), 289–304.
54.
YsseldykeJ. E., & OlsenK. R. (1997). Putting alternate assessments into practice: What to measure and possible sources of data (Synthesis Report No. 28). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://www.cehd.umn.edu/nceo/OnlinePubs/archive/Synthesis/Synthesis28.htm
55.
ZwickR. (2007). College admissions in twenty-first-century America: The role of grades, tests, and games of chance. Harvard Educational Review, 77(4), 419–429.
