Determination of Differential Bundle Functioning (DBF) of Numeracy and Literacy Tests Administered to Grade 3 Learners in South Africa

Abstract

The study is based on data gathered for the Assessment Modelling Initiative (AMI), part of the District Development Support Programme (DDSP) project conducted by the Human Sciences Research Council (HSRC) in 2002, in conjunction with the Department of Education (DoE) and provincial departments of education of the Eastern Cape, KwaZulu-Natal, Northern Cape, and Limpopo. The project was funded by the United States Agency for International Development (USAID) and managed by Research Triangle Institute (RTI) International. The Joint Education Trust (JET) developed the Grade 3 numeracy test and adapted the literacy test from the International Association for the Evaluation of Educational Achievement (IEA) survey. The aim of AMI was to develop assessment materials in literacy and numeracy for learners in Grade 3 that could be used by teachers in classroom instruction and for learner assessment. The purpose of the analysis was to investigate Differential Item Functioning (DIF) and Differential Bundle Functioning (DBF) in these tests and to determine item difficulty, discrimination, reliability, and evidence of construct validity. Participants were 4,000 male and 4,000 female Grade 3 learners selected using the stratified random sampling method from 13,425 Grade 3 learners who participated in the DDSP project. The Simultaneous Item Bias Test (SIBTEST) was used for detecting DIF and bias. SIBTEST was also used to detect bundle bias or DBF. There were significant gender differences in mean scores of both numeracy and literacy tests. Female participants performed consistently better than male participants in both tests. Both the numeracy and the literacy tests possessed appropriate psychometric properties of difficulty, discrimination, reliability, and some evidence of construct validity. Significant estimates of beta (test for DIF and DBF) were obtained for literacy but not for numeracy. The literacy test exhibited DIF at item level and DBF at bundle level. Although a myriad of factors could have caused DIF and DBF, the use of tests translated into various languages may have introduced bias. Further analysis should be done to identify other factors that could have caused DIF and DBF.

Keywords

Assessment Modelling Initiative (AMI)differential bundle functioning (DBF)differential item functioning (DIF)item bias literacy test numeracy test non-uniform DIF Simultaneous Item Bundle Functioning (SIBTEST)test bias uniform DIF

Get full access to this article

View all access options for this article.

References

Ackerman

T. A.

(1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29, 67–91.

Boughton

K. A.

Gierl

M. J.

Khaliq

S. N.

(2000). Differential Bundle Functioning on Mathematics and Science Achievement Tests: A Small Step Toward Understanding Differential Performance. Paper presented at the Annual Meeting of the Canadian Society for Studies in Education (CSSE), Edmonton, Alberta, Canada.

Crocker

Algina

(1986). Introduction to classical and modern test theory. Orlando, Florida. Holt, Rinehart and Winston, Inc.

Douglas

J. A.

Roussos

L. A.

Stout

(1996). Item bundle DIF hypothesis testing: Identifying suspect bundles and assessing their DIF. Journal of Educational Measurement, 33, 465–484.

Durrheim

Baillie

Johnstone

(2008). The development and validation of a measure of racial justice perceptions. South African Journal of Psychology, 38, 615–632.

Fry

Greenop

Turnbull

Bowman

(2009). The effect of education and gender on emotion-based decision-making. South African Journal of Psychology, 39, 122–132.

Gierl

M. J.

Khaliq

S. N.

(2001). Identifying sources of differential item and bundle functioning on translated achievement tests: confirmatory analysis. Journal of Educational Measurement, 38, 164–187.

Hambleton

R. K.

Jones

R. W.

(1994). Comparison of empirical and judgemental procedures for detecting differential item functioning. Educational Research Quarterly, 18, 21–36.

Hambleton

R. K.

Rogers

H. J.

(1989). Detecting potentially biased test items: Comparison of IRT area and Mantel-Haenszel methods. Applied Measurement in Education, 2, 313–334.

10.

Holland

P. W.

Thayer

D. T.

(1988). Differential item functioning and the Mantel Haenszel procedure. In Wainer

Braun

H. I.

(eds), Test validity (pp. 129–145). Hillsdale, NJ: Lawrence Erlbaum.

11.

Joseph

Van Lill

(2008). Investigating subscale differences among race and language group of the Occupational Personality Profile. South African Journal of Psychology, 38, 501–514.

12.

H. H.

Stout

(1996). A new procedure for detection of crossing DIF. Psychometrika, 61, 647–677.

13.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

14.

Martin

M. O.

Mullis

I. V. S.

Gonzalez

E. J.

Kennedy

A. M.

(2003). Trends in children's reading literacy achievement 1991–2001: IEA's Repeat in nine countries of the 1991 Reading Literacy Study. Chestnut Hill, MA: Boston College.

15.

Mellenbergh

G. J.

(1982). Contingency table models for assessing item bias. Journal of Educational Statistics, 7, 105–108.

16.

Nandakumar

(1993). Simultaneous DIF Amplification and Cancellation: In Shealy & Stout's Test for DIF. Journal of Educational Measurement, 30, 293–311.

17.

Pine

S. M.

(1977). Application of item response theory to the problem of test bias. In Weiss

D. J.

(ed.), Application of computerized adaptive testing; Research report No. 77. Minneapolis: University of Minnesota, Psychometric Methods Program: pp. 37–43.

18.

Pinefield

R. D.

Lam

T. C. M.

(2000). Assessing differential item functioning in performance assessment: Review and recommendations. Educational Measurement: Issues and Practice, 19, 5–15.

19.

Rousos

Stout

(1996). A multi-dimensionality-based DIF analysis paradigm. Applied Psychological Measurement, 20, 355–371.

20.

Shealy

Stout

W. F.

(1993a). A model-based standardization approach that separates true bias/DIF from group differences and detects test bias/DIF as well as item bias/DIF. Psychometrika, 58, 159–194.

21.

Shealy

Stout

W. F.

(1993b). An item response theory model for test bias and differential test functioning. In Holland

P. W.

Wainer

(eds), Differential item functioning (pp. 197–239). Hillsdale, NJ: Lawrence Erlbaum.

22.

Stout

Rousos

(1999). Dimensionality-based DIF/DBF package [Computer program]. Champaign-Urbana, IL: William Stout Institute for Measurement, University of Illinois.

23.

Swaminathan

Rogers

H. J.

(1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361–370.

24.

Thissen

Steinberg

Wainer

(1993). Detection of differential item functioning using the parameters of item response models. In Holland

P. W.

Wainer

(eds), Differential item functioning (pp. 67–113). Hillsdale, NJ: Lawrence Erlbaum Associates.