This study investigated the score exchangeability of concept maps with problem-solving essays. Of interest was whether sixth-grade students' concept maps predicted their scores on essay responses that used concept map content. Concept maps were hypothesized to be alternatives to performance assessments for content-area domain knowledge in science. Both special (n = 16) and general (n = 16) education students participated in a science unit that covered three ecosystem cycles: (a) the water cycle, (b) the nitrogen cycle, and (c) the carbon dioxide-oxygen cycle. Assessment tasks required students to draw three concept maps and respond to two essay questions. Results showed that concept maps and essay tasks lacked consistent exchangeability. Findings indicated low predictive validity between map scores and essay scores and, thus, maps should not be considered alternative content-area domain assessments.
Get full access to this article
View all access options for this article.
References
1.
Baker, E.L. (1994). Learning-based assessments of history understanding . Educational Psychologist, 29(2), 97-106.
2.
Baxter, G.P., & Shavelson, R.J. (1994). Performance assessments: Benchmarks and surrogates. International Journal of Educational Research, 21, 279-298.
3.
Cawley, J.F. (1994). Science for students with disabilities. Remedial and Special Education, 15(2), 67-71.
4.
De La Paz, S., & Graham, S. (1997). Strategy instruction in planning: Effects on the writing performance and behavior of students with learning difficulties. Exceptional Children, 63, 167-81.
5.
Dietel, R.J., Herman, J.L., & Knuth, R.A. (1991). What does research say about assessment?Oak Brook, IL: North Central Regional Educational Laboratory .
6.
Fuchs, L.S. (2003). Assessing intervention responsiveness. Learning Disabilities Research and Practice, 18, 172-186.
7.
Goldberg, G.L., & Roswell, B.S. (2001). Are multiple measures meaningful? Lessons from a statewide performance assessment. Applied Measurement in Education, 14, 125-150.
8.
Harniss, M.K., Dickson, S.V., Kinder, D., & Hollenbeck, K.L. (2001). Textual problems and instructional solutions: Strategies for enhancing learning from published history textbooks . Reading & Writing Quarterly, 17, 127-150.
9.
Jentzsch, C., & Tindal, G. (1991). Analytic scoring of writing (Training Module No. 8). Eugene: University of Oregon .
10.
Joehnk, D., Tindal, G., Nolet, V., & Hollenbeck, K. (2003). A flowchart scoring update . Available online: http://brt.uoregon.edu.
11.
Klein, D.C.D., Chung, G.K.W.K., Osmundson, E., Herl, H.E., & O'Neil, H.F. (2002). Examining the validity of knowledge mapping as a measure of elementary students' scientific understanding (Center for the Study of Evaluation Technical Report 557). Los Angeles: UCLA, Center for Research on Evaluation, Standards, and Student Testing.
12.
Koretz, D., & Hamilton, L. (2001). The performance of students with disabilities on New York's revised regents' comprehensive examination in English (Center for the Study of Evaluation Technical Report 540). Los Angeles: UCLA, Center for Research on Evaluation, Standards, and Student Testing.
13.
Mayer, R.E. (1989). Models for understanding. Review of Educational Research, 59, 43-64.
14.
Means, B., & Knapp, M.S. (1991). Introduction: Rethinking teaching for disadvantaged students. In B. Means, C. Chelemer, and M. S. Knapp (Eds.), Teaching advanced skills to at-risk students: Views from research and practice (pp. 1-26). San Francisco : Jossey-Bass.
15.
Messick, S. (1982). The values of ability testing: Implications of multiple perspectives about criteria and standards. Educational Measurement: Issues and Practice, 1(3), 9-12, 20, 26.
16.
Messick, S. (1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In H. Wainer & H. Braun (Eds.), Test validity (pp. 33-45). Hillsdale, NJ: Erlbaum.
17.
Messick, S. (1994). The interplay of evidence and consequences in the validity of performance assessments. Educational Researcher , 23(2), 13-23.
18.
Messick, S. (1996). Validity of performance assessments. In G.W. Phillips (Ed.), Technical issues in large-scale performance assessment (pp. 1-18). Washington, DC: National Center for Education Statistics.
19.
Novak, J.D. (1990). Concept maps and Venn diagrams: Two metacognitive tools to facilitate meaningful learning. InstructionalScience , 19, 29-52.
20.
Novak, J.D. (1991). Clarify with concept maps: A tool for students and teachers alike. The Science Teacher, 58(7), 44-49.
21.
Novak, J.D. (1993). How do we learn our lesson? Talking students through the process. The Science Teacher, 60(3), 50-55.
22.
Novak, J.D., & Gowin, D.B. (1984). Learning how to learn. Cambridge, UK: Cambridge University Press.
23.
O'Neil, H.F., Chung, G.K.W.K., & Herl, H.E. (1999). Computer-based collaborative knowledge mapping to measure team processes and team outcomes (Center for the Study of Evaluation Technical Report 502). Los Angeles: UCLA, Center for Research on Evaluation, Standards, and Student Testing.
24.
Odom, A.L., & Barrow, L.H. (1995). Development and application of a two-tier diagnostic test measuring college biology students' understanding of diffusion and osmosis after a course of instruction. Journal of Research in Science Teaching, 32, 45-61.
25.
Osmundson, E., Chung, G.K.W.K., Herl, H.E., & Klein, D.C.D. (1999). Knowledge mapping in the classroom: A tool for examining the development of students' conceptual understandings (Center for the Study of Evaluation Technical Report 507). Los Angeles : UCLA, Center for Research on Evaluation, Standards, and Student Testing.
26.
Parker, R.M. (1990). Power, control, and validity in research. Journal of Learning Disabilities, 23, 613-620.
27.
Pearson, P.D., & Garavaglia, D.R. (1997). Improving the information value of performance items in large-scale assessments (NAEP Validity Studies) . Palo Alto, CA: American Institute for Research.
28.
Pearson, P.D., Vyas, S., Sensale, L.M., & Kim Y. (2001). Making our way through the assessment and accountability maze: Where do we go now?Clearing House, 74, 175-182.
29.
Phillips, S.E. (1996). Legal defensibility of standards: Issues and policy perspectives. Educational Measurement: Issues and Practice , 15(2), 5-13, 19.
30.
Ruiz-Primo, M.A., Schultz, S.E., Li, M., & Shavelson, R.J. (1998). Comparison of the reliability and validity of scores from two concept-mapping techniques (Center for the Study of Evaluation Technical Report 492). Los Angeles : UCLA, Center for Research on Evaluation, Standards, and Student Testing.
31.
Ruiz-Primo, M.A., Schultz, S.E., & Shavelson, R.J. (1997). Concept map-based assessment in science: Two exploratory studies (Center for the Study of Evaluation Technical Report 436). Los Angeles: UCLA, Center for Research on Evaluation, Standards, and Student Testing.
32.
Ruiz-Primo, M.A., & Shavelson, R.J. (1996). Problems and issues in the use of concept maps in science assessment. Journal of Research in Science Teaching, 33, 569-600.
33.
Ruiz-Primo, M.A., Shavelson, R.J., & Schultz, S.E. (1997). On the validity of concept map-based assessment interpretations: An experiment testing the assumption of hierarchical concept maps in science (Center for the Study of Evaluation Technical Report 455) . Los Angeles: UCLA, Center for Research on Evaluation, Standards, and Student Testing.
34.
Sager, S. (1998). Statview. Cary, NC: SAS Institute.
35.
Sexton, M., Harris, K.R., & Graham, S. (1998). Self-regulated strategy development and the writing process: Effects on essay writing and attributions. Exceptional Children, 64, 295-311.
36.
Shavelson, R.J., & Ruiz-Primo, M.A. (1998). On the assessment of science achievement: Conceptual underpinnings for the design of performance assessments. Report of year 2 activities (Center for the Study of Evaluation Technical Report 491). Los Angeles: UCLA, Center for Research on Evaluation, Standards, and Student Testing.
37.
Shavelson, R.J., Lang, H., & Lewin, B. (1994). On concept maps as potential "authentic" assessments in science (Center for the Study of Evaluation Technical Report 388). Los Angeles: UCLA, Center for Research on Evaluation, Standards, and Student Testing.
38.
Sinatra, R.C. (2000). Teaching learners to think, read, and write more effectively in content subjects. Clearing House, 73, 266-273.
39.
Sloane, F.C., & Kelly, A.E. (2003). Issues in high-stakes testing programs. Theory into Practice, 42, 12, 17.
40.
Speece, D.L., Case, L.P., & Molloy, D.E. (2003). Responsiveness to general education: The first gate to learning disabilities identification. Learning Disabilities Research and Practice, 18, 147-156.
41.
Sugrue, B., Webb, N., & Schlackman, J. (1998). The interchangeability of assessment methods in science (Center for the Study of Evaluation Technical Report 474). Los Angeles: UCLA, Center for Research on Evaluation, Standards, and Student Testing.
42.
Thompson, S., Thurlow, M., & Whetstone, P. (2001). Recommendations for addressing standards and assessments on state and district IEP forms (Policy Directions No. 12). Minneapolis: University of Minnesota, National Center on Educational Outcomes.
43.
Twyman, T. (2003). Effects of a conceptually-frame, problem/solution/effect graphic organizer on content comprehension and problem-solving skills for seventh grade social studies students. Unpublished doctoral dissertation, University of Oregon, Eugene.
44.
van Dijk, T.A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press.
45.
Vaughn, S., Linan-Thompson, S., & Hickman, P. (2003). Response to instruction as a means of identifying students with reading/learning disabilities. Exceptional Children , 69, 391-409.
46.
Webb, N.M., Schlackman, J., & Sugrue, B. (2000). The dependability and interchangeability of assessment methods in science (Center for the Study of Evaluation Technical Report 515) . Los Angeles: UCLA, Center for Research on Evaluation, Standards, and Student Testing.
