Construction of a paper-and-pencil Test of Basic Scientific Literacy based on selected literacy goals recommended by the American Association for the Advancement of Science
Restricted accessResearch articleFirst published online October, 1996
Construction of a paper-and-pencil Test of Basic Scientific Literacy based on selected literacy goals recommended by the American Association for the Advancement of Science
We describe the construction of a paper-and-pencil Test of Basic Scientific Literacy (TBSL) specifically designed for high-school leavers entering technikon and university in South Africa. The `true-false-don't know' scientific literacy test-items are based on a pool of 472 items developed previously from selected literacy goals recommended by the AAAS in Science for all Americans. Test-items were pilot-tested on 625 technikon and university students and were included in the 110-item TBSL on the basis of item discrimination, item difficulty and student feedback. The TBSL consists of three subtests based on Jon Miller's three constitutive dimensions of scientific literacy: the nature of science (22 items); science content knowledge (72 items); and the impact of science and technology on society (16 items). About 260 South Africa-based members of various South African professional science and engineering associations participated in setting a performance standard for each of the three dimensions of scientific literacy. These standards were validated using a `contrasting groups' approach. The internal consistency of the individual TBSL subtests, and the reliability of mastery-nonmastery classification decisions based on the performance standard, was found to be about 0.80. The reliability of the overall 110-item TBSL was 0.95.
Get full access to this article
View all access options for this article.
References
1.
Shamos, M. , 1988, The lesson every child need not learn . The Sciences, 28(4), 14-20.
2.
American Association for the Advancement of Science , 1989, Science for all Americans (Washington, DC: AAAS).
3.
National Science Teachers Association , 1992, Scope, Sequence and Coordination of Secondary School Science, Vol. 1: The Content Core (Washington, DC: National Science Teachers Association).
4.
The methods and procedures used in the development of the scientific literacy test were outlined in Laugksch, R. C. , and Spargo, P. E. , 1993, Development of a scientific literacy test based on AAAS literacy goals: methods and procedures . Paper presented at the 1993 International Conference on the Public Understanding of Science and Technology, Chicago Academy of Sciences, Chicago, IL.
5.
Laugksch, R. C. , and Spargo, P. E. , 1996, Development of a pool of scientific literacy test-items based on selected AAAS literacy goals . Science Education, 80(2), 121-143.
6.
Berk, R. A. , 1984, A Guide to Criterion-referenced Test Construction (Baltimore: Johns Hopkins University Press).
7.
DeBoer, G. E. , 1991, A History of Ideas in Science Education (New York: Teachers College Press).
8.
Roberts, D. A. , 1983, Scientific Literacy: Towards a Balance for Setting Goals for School Science Programs (Ottawa, Canada: Minister of Supply and Services).
9.
Roberts, D. A. , 1983, Scientific Literacy: Towards a Balance for Setting Goals for School Science Programs (Ottawa, Canada: Minister of Supply and Services).
10.
Jenkins, E. W. , 1992, School science education: towards a reconstruction . Journal of Curriculum Studies, 24(3), 229-246.
11.
See, for example, Shortland, M. , 1988, Advocating science: literacy and public understanding . Impact of Science on Society, 152, 305-316.
12.
Champagne, A. B. , and Lovitts, B. E. , 1989, Scientific literacy: a concept in search of definition. This Year in School Science. Scientific Literacy, edited by A. B. Champagne , B. E. Lovitts and B. J. Callinger (Washington, DC: AAAS), pp. 1-14.
13.
Miller, J. D. , 1983, Scientific literacy: a conceptual and empirical review . Daedalus, 112(2), 29-48.
14.
For example, Arons, A. B. , 1983, Achieving wider scientific literacy . Daedalus, 112(2), 91-122.
15.
Laetsch, W. M. , 1987, A basis for better understanding of science. Communicating Science to the Public, edited by D. Evered and M. O'Connor (London: Wiley), pp. 1-10.
16.
Miller, J. D. , 1989, Scientific literacy . Paper presented at the annual meeting of the American Association for the Advancement of Science, San Francisco, CA.
17.
Meichtry, Y. J. , 1993, The impact of science curricula on student views about the nature of science . Journal of Research in Science Teaching, 30(5), 429-443.
18.
Cooley, W. W. , and Klopfer, L. E. , 1961, Test on Understanding Science. Form W (Princeton, NJ: Educational Testing Service).
19.
Rubba, P. A. , and Anderson, H. O. , 1978, Development of an instrument to assess secondary school students' understanding of the nature of scientific knowledge . Science Education, 62(4), 449-458.
20.
Kimball, M. E. , 1968, Understanding the nature of science: a comparison of scientists and science teachers . Journal of Research in Science Teaching, 5, 110-120.
21.
Rubba, P. A. , and Anderson, H. O. , 1978, Development of an instrument to assess secondary school students' understanding of the nature of scientific knowledge . Science Education, 62(4), 449-458.
22.
Anonymous , 1989, Conference proceedings, edited paper collections, books and bibliographic sources on children's conceptions in science . International Journal of Science Education, 11(5), 597 .
23.
The STS movement is outlined, for example, by Yager, R. E. , 1993, Science-Technology-Society as reform . School Science and Mathematics, 93(3), 145-151.
24.
Aikenhead, G. S. , and Ryan, A. G. , 1992, The development of a new instrument: `Views on Science-Technology-Society' (VOSTS) . Science Education, 76(5), 477-491.
25.
Aikenhead, G. S. , and Ryan, A. G. , 1992, The development of a new instrument: `Views on Science-Technology-Society' (VOSTS) . Science Education, 76(5), 480 .
26.
Lord, T. R. , and Rauscher, C. , 1991, A sampling of basic life science literacy in a college population . The American Biology Teacher, 53(7), 449-424.
27.
Cannon, J. , and Jinks, J. , 1992, A cultural literacy approach to assessing general scientific literacy . School Science and Mathematics, 92(4), 196-200.
28.
See Hirsch, E. D., Jr , 1988, Cultural Literacy: What Every American Needs to Know (New York: Vintage Books).
29.
Miller, J. D. , 1992, Toward a scientific understanding of the public understanding of science and technology . Public Understanding of Science, 1(1), 23-26.
30.
Miller, J. D. , 1992, Toward a scientific understanding of the public understanding of science and technology . Public Understanding of Science, 1(1), 23-26.
31.
Durant, J. R. , Evans, G. A. , and Thomas, G. P. , 1992, The public understanding of science in Britain: the role of medicine in the popular representations of science . Public Understanding of Science, 1(2), 161-182.
32.
INRA (Europe) and Report International , 1993, Europeans, Science and Technology. Report No. EUR15461 (Brussels: Commission of the European Communities, Directorate-General XIII)
33.
Ge, T. , Liu, W. , and Li, D. , 1995, Public scientific literacy in China and international comparisons . Paper presented at the 1995 International Conference on Public Understanding of Science, Beijing, China.
34.
cited by Zhongliang Zhang and Jiansheng Zhang , 1993, A survey of public scientific literacy in China . Public Understanding of Science, 2(1), 21-38.
35.
Fjæstad, B. , 1994, The 1994 Swedish survey on public understanding of science . Paper presented at the 1994 International Conference on the Public Understanding of Science and Technology, Science Museum, London, UK.
36.
See references contained in Einsiedel, E. F. , 1994, Mental maps of science: knowledge and attitudes among Canadian adults . International Journal of Public Opinion Research, 6(1), 35-44.
37.
Bureau of Industry Economics , 1995, Science Understanding and Awareness: How Australians Rank in International Surveys. Occasional Paper 30 (Canberra: Australian Government Publishing Service).
38.
Kim, H.-S. , 1995, Does the Korean public understand science and technology? Paper presented at the 1995 International Conference on Public Understanding of Science, Beijing, China.
39.
Miller, J. D. , 1992, Toward a scientific understanding of the public understanding of science and technology . Public Understanding of Science, 1(1), 23-26.
40.
Miller, J. D. , 1993, Theory and measurement in the public understanding of science: a rejoinder to Bauer and Schoon . Public Understanding of Science, 2(3), 235-243.
41.
National Science Board , 1993, Science & Engineering Indicators—1993 (NSB 93-1) (Washington, DC: US Government Printing Office).
42.
Durant, J. R. , Evans, G. A. , and Thomas, G. P. , 1989, The public understanding of science . Nature, 340(6228), 11-14.
43.
Miller, J. D. , 1993, Theory and measurement in the public understanding of science: a rejoinder to Bauer and Schoon . Public Understanding of Science, 2(3), 235-243.
44.
National Science Board , 1993, Science & Engineering Indicators—1993 (NSB 93-1) (Washington, DC: US Government Printing Office).
45.
National Science Board , 1991, Science & Engineering Indicators—1991 (NSB 91-1) (Washington, DC: US Government Printing Office).
46.
Doran, R. L. , Lawrenz, F. , and Helgeson, S. , 1994, Research on assessment in science. Handbook of Research on Science Teaching and Learning, edited by D. L. Gabel (Toronto: Macmillan), p. 394.
47.
See, for example, Champagne, A. B. , and Newell, S. T. , 1992, Directions for research and development: alternative methods of assessing scientific literacy . Journal of Research in Science Teaching, 29(8), 841-860.
48.
The approach to test students entering tertiary education has been chosen for practical and for conceptual reasons. Firstly, for a number of reasons it is less of a logistical tour de force to test South African students at tertiary educational institutions than at high-school. Secondly, and more importantly, high-school leavers at tertiary educational institutions represent a self-selected group of `successful' high-school leavers. It is therefore not unreasonable to believe that this latter group is more likely to exhibit successful outcomes of secondary education, such as, for example, being minimally scientifically literate, than less `successful' high-school leavers who do not enter tertiary education.
49.
For example, it has been suggested on the basis of research that the importance of science topic knowledge in reasoning should not be underestimated. See Linn, M. C. , Clement, C. , Pulos, S. , and Sullivan, P. , 1989, Scientific reasoning during adolescence: the influence of instruction in science knowledge and reasoning strategies . Journal of Research in Science Teaching, 26(2), 171-187.
50.
Walberg, H. J. , 1983, Scientific literacy and economic productivity in international perspective . Daedalus, 112(2), 1-28.
51.
Nitko, A. J. , 1984, Defining `criterion-referenced test'. A Guide to Criterion-Referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 8-28.
52.
Hambleton, R. K. , and Rogers, H. J. , 1991, Advances in criterion-referenced measurement. Advances in Educational and Psychological Testing: Theory and Applications, edited by R. K. Hambleton and J. N. Zaal (Boston, MA: Kluwer Academic), pp. 3-42.
53.
In norm referenced-tests, test scores are referenced internally, that is performance is interpreted with reference to a `norm' group. In criterion-referenced tests, test scores are referenced externally, that is performance is interpreted with reference to clear attainment `targets' for content specifications. See Hambleton, R. K. , and Rogers, H. J. , 1991, Advances in criterion-referenced measurement. Advances in Educational and Psychological Testing: Theory and Applications, edited by R. K. Hambleton and J. N. Zaal (Boston, MA: Kluwer Academic), pp. 3-42.
54.
Ebel, R. L. , and Frisbie, D. A. , 1991, Essentials of Educational Measurement (5th edn) (Englewood Cliffs, NJ: Prentice Hall).
55.
These strategies are summarized in this paper in the section on item validity. Additional details are given in Laugksch, R. C. , and Spargo, P. E. , 1996, Development of a pool of scientific literacy test-items based on selected AAAS literacy goals . Science Education, 80(2), 121-143.
56.
Doran, R. L. , Lawrenz, F. , and Helgeson, S. , 1994, Research on assessment in science. Handbook of Research on Science Teaching and Learning, edited by D. L. Gabel (Toronto: Macmillan), pp. 388-442.
57.
Shavelson, R. J. (Guest Ed.), 1994, Performance assessment . International Journal of Educational Research, 21(3), 235-350.
58.
The following chapters in SFAA were used in constructing the item pools for Miller's dimensions of scientific literacy: `The Nature of Science' for the nature-of-science dimension; `The Physical Environment', `The Living Environment' and `The Human Organism' for the science-content knowledge dimension; and `The Nature of Technology' for the impact-of-science-and-technology-on-society dimension. The science content areas covered by the various SFAA chapters corresponding to the science-content dimension are the earth/space and physical/chemical sciences, the life sciences, and the health sciences, respectively. For further details see Laugksch, R. C. , and Spargo, P. E. , 1996, Development of a pool of scientific literacy test-items based on selected AAAS literacy goals . Science Education, 80(2), 121-143.
59.
Berk, R. A. , 1984, Conducting the item analysis. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 97-143.
60.
South African technikons are tertiary educational institutions where the emphasis in teaching and research is placed more on the practical, rather than theoretical, aspects of numerous varied disciplines. Technikons are now widely regarded as equivalent in status to a university.
61.
Population group categories used here, that is African, Coloured, Indian and White, follow former South African legislative practice. The categories are used to highlight differences between the `racial' groups created by Apartheid policies and their use does not intend to give any legitimacy to these categories. Indian students were excluded from these two ϰ2 analyses because of the students' very low total frequency (< 15).
62.
There were no significant differences in the composition of the four pilot test-forms with respect to population group and gender for each of the `instructed' and `uninstructed' groups (unpublished data). Hence the data of the four pilot test-forms were combined for each criterion group.
63.
both cited by Berk, R. A. , 1984, Conducting the item analysis. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 97-143.
64.
Laugksch, R. C. , and Spargo, P. E. , 1996, Development of a pool of scientific literacy test-items based on selected AAAS literacy goals . Science Education, 80(2), 121-143. For further details of item-objective congruence see the section in this paper on item validity.
65.
Hambleton, R. K. , 1984, Validating the test scores. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 199-230.
66.
Berk, R. A. , 1984, Conducting the item analysis. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 97-143.
67.
Once Fellows had identified key ideas of `undisputed significance to science' in the entire SFAA chapter allocated to each of them, the Fellows were asked to place each idea into the following three categories of importance: `very important', `important', and `relatively less important'. For each item, the number of ratings in the three categories were summed; the item's final importance rating was determined by the category with the highest number of `votes'. Tied rankings resulted in the middle, that is `important', category being chosen for that item.
68.
The difficulty of finding students enrolled at tertiary educational institutions in South Africa who have taken no science subject at all in senior high-school was pointed out earlier.
69.
Zeller, R. A. , 1994, Validity. The International Encyclopedia of Education: Vol. 11 (2nd edn), edited by T. Husen and T. N. Postlethwaite (Oxford: Pergamon Press), pp. 6569-6576,
70.
Shepard, L. A. , 1993, Evaluating test validity . Review of Educational Research, 19, 405-450.
71.
Laugksch, R. C. , and Spargo, P. E. , 1996, Development of a pool of scientific literacy test-items based on selected AAAS literacy goals . Science Education, 80(2), 121-143.
72.
Hambleton, R. K. , 1984, Validating the test scores. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 199-230.
73.
Hambleton, R. K. , 1984, Validating the test scores. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 199-230.
74.
Further details of the judgemental review procedure are given in Laugksch, R. C. , and Spargo, P. E. , 1996, Development of a pool of scientific literacy test-items based on selected AAAS literacy goals . Science Education, 80(2), 121-143.
75.
Zeller, R. A. , 1994, Validity. The International Encyclopedia of Education: Vol. 11 (2nd edn), edited by T. Husen and T. N. Postlethwaite (Oxford: Pergamon Press), p. 6572.
76.
American Association for the Advancement of Science , 1993, Benchmarks for Science Literacy (New York: Oxford University Press).
77.
Ahlgren, A. , 1993, Creating benchmarks for science education . Educational Leadership, 50(5), 46-49.
78.
Shepard, L. A. , 1984, Setting performance standards. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 169-198.
79.
This professional science association has recently extended its scope and has been renamed the Southern African Institute of Ecologists and Environmental Scientists.
80.
Berk, R. A. , 1986, A consumer's guide to setting performance standards on criterion-referenced tests . Review of Educational Research, 56(1), 137-172.
81.
cited by Shepard, L. A. , 1984, Setting performance standards. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 169-198.
82.
Scientific literacy is defined as `a desired level of depth and breadth of scientific understanding appropriate to the interests and needs of the person being taught, set within the context of the developmental, educational, economic and political needs and interests of a country at a given point in time'. This definition is adapted from Champagne, A. B. , and Lovitts, B. E. , 1989, Scientific literacy: a concept in search of definition. This Year in School Science. Scientific Literacy, edited by A. B. Champagne . B. E. Lovitts and B. J. Callinger (Washington, DC: AAAS), p. 12.
83.
In almost every content area there were some test-items for which the probability assigned by different judges were observed to vary widely. The median is less sensitive than the mean to large fluctuations in the value of a variable, and consequently the median rather than the mean was used in the determination of the cut-off score per content area. See Zar, J. H. , 1984, Biostatistical Analysis (2nd edn) (Englewood Cliffs, NJ: Prentice-Hall).
84.
The lower end-point of the two-sided 95 per cent confidence interval around the median overall cut-off score for each area, rounded downwards to the nearest integer, was used in the calculation of the performance standard. For details of the calculation of the confidence interval around the median see Zar, J. H. , 1984, Biostatistical Analysis (2nd edn) (Englewood Cliffs, NJ: Prentice-Hall).
85.
The influence of blind guessing on the scores of a true-false test diminishes as the test increases in length. On a one-item true-false test an individual has a 50 per cent chance of getting a perfect score, but on a two-item test it drops to 25 per cent, and on a five-item test to 3 per cent. The chance of obtaining a score equal to the required performance standard on each subtest by blind guessing is about one in 10 000, one in 1013, and one in 1000 for the NSST, SCKST, and ISTSST, respectively. The chance of meeting the performance standard on all subtests by blind guessing is consequently about one in 1020. See Ebel, R. L. , and Frisbie, D. A. , 1991, Essentials of Educational Measurement (5th edn) (Englewood Cliffs, NJ: Prentice Hall).
86.
cited by Shepard, L. A. , 1984, Setting performance standards. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 169-198.
87.
The frequencies were insufficiently large to use a ϰ2 analysis, and hence the Fisher's Exact Test was used. See Everett, B. S. , 1992, The Analysis of Contingency Tables (2nd edn) (London: Chapman & Hall).
88.
In this paper, the Yates continuity correction is used for the analysis of all 2×2 contingency tables.
89.
The term `Black' refers to persons classified as African, Coloured or Indian.
90.
The majority of African students are English Second Language speakers, and this difference may indicate that the TBSL is potentially not language-robust.
91.
Berk, R. A. , 1984, Selecting the index of reliability. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 231-266.
92.
Hambleton, R. K. , and Novick, M. R. , 1973, Toward an integration of theory and method for criterion-referenced tests . Journal of Educational Measurement, 10, 159-170.
93.
The Kuder-Richardson 20 coefficient is calculated by [ILLEGIBLE] where k is the number of test-items, p is the proportion of students answering a question correctly, and V is the variance of total test scores. For further details see Livingston, S. A. , 1988, Reliability of test results. Educational Research, Methodology, and Measurement, edited by J. P. Keeves (Oxford: Pergamon Press), pp. 386-392.
94.
Taken together, however, the three subtests measure scientific literacy.
95.
Thorndike, R. L. , 1990, Reliability. The International Encyclopedia of Educational Evaluation, edited by H. J. Walberg and G. D. Haertel (Oxford: Pergamon Press), pp. 260-273.
96.
Carmines, E. G. , and Zeller, R. A. , 1979, Reliability and Validity Assessment. Quantitative Applications in the Social Sciences Series No. 17 (Beverly Hills, CA: Sage).
97.
For further details see Berk, R. A. , 1984, Selecting the index of reliability. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 231-266.
98.
Lord, F. M. , and Novick, M. R. , 1968, Statistical theories of Mental Test Scores (Reading, MA: Addison-Wesley), p. 251.
99.
Unpublished data.
100.
Carmines, E. G. , and Zeller, R. A. , 1979, Reliability and Validity Assessment. Quantitative Applications in the Social Sciences Series No. 17 (Beverly Hills, CA: Sage).
101.
The measurement coefficient of variation expresses an estimate of the `true' standard deviation of a subtest score (i.e., the standard error of measurement) as a percentage of the relevant mean subtest score.
102.
Subkoviak, M. J. , 1984, Estimating the reliability of master-nonmastery classification. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 267-291.
103.
Hambleton, R. K. , and Rogers, H. J. , 1991, Advances in criterion-referenced measurement. Advances in Educational and Psychological Testing: Theory and Applications, edited by R. K. Hambleton and J. N. Zaal (Boston, MA: Kluwer Academic), pp. 3-42.
104.
cited by Subkoviak, M. J. , 1984, Estimating the reliability of master-nonmastery classification. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 267-291.
105.
Subkoviak, M. J. , 1984, Estimating the reliability of master-nonmastery classification. A Guide to Criterion-referenced Test Construction, edited by R. A. Berk (Baltimore: Johns Hopkins University Press), pp. 267-291.
106.
The bounds of p0 are 0 and +1.00; the upper limit of p0 occurs only if all individuals are consistently classified.
107.
Hambleton; R. K. , and Rogers, H. J. , 1991, Advances in criterion-referenced measurement. Advances in Educational and Psychological Testing: Theory and Applications, edited by R. K. Hambleton and J. N. Zaal (Boston, MA: Kluwer Academic), pp. 3-42.
108.
Participating students at the five tertiary educational institutions came from a variety of educational and socioeconomic backgrounds (unpublished data), and were registered for first degrees and diplomas in commerce and management, in the human and natural sciences, and in engineering.
109.
Plans are afoot to further refine this test. Envisaged work includes factor analysis of the test-items, as well as interviewing respondents to explore, firstly, how they understood the questions, and, secondly, what they had in mind when they gave their answers.
110.
Shortland, M. , 1988, Advocating scientific literacy: literacy and public understanding . Impact of Science on Society, 152, 305-316.
