To find population proficiency distributions, a two-level hierarchical linear model
may be applied to large-scale survey assessments such as the National Assessment of
Educational Progress (NAEP). The model and parameter estimation are developed and a
simulation was carried out to evaluate parameter recovery. Subsequently, both a
hierarchical and a simple model were applied to NAEP reading data. The impact of
using a hierarchical model was found to be relatively modest in this case, mostly due
to modest clustering. Several other applications and future studies are
discussed.
AdamsR. J.WilsonM. R.WuM. L. (1997). Multilevel item response models:
An approach to errors in variables regression. Journal of
Educational and Behavioral Statistics, 22,
46–75.
2.
AitkinM.AitkinI. (2005). Comparison of direct estimation
with the conditioning model and plausible value imputation.
Paper prepared for U.S. Department of Education,
Office of Educational Research and Improvement, National Center for
Education Statistics (NCES) Washington, DC.
3.
AllenN. L.DonoghueJ. R.SchoepsT. L. (2001). The NAEP 1998 technical report No. NCES
2001-509. Washington, DC:
National Center for Educational
Statistics.
4.
BirnbaumA. (1968). Some latent trait models and
their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.). Statistical theories of mental test score.
Reading, MA:
Addison-Wesley.
5.
DempsterA. P.LairdN. M.RubinD. B. (1977). Maximum likelihood from
incomplete data via the EM algorithm. Journal of the Royal
Statistical Society, Series B, 39,
1–38.
6.
JohnsonM. S. (2002). A Bayesian hierarchical model for
multi-dimensional performance assessments. Paper
presented at the 2002 annual meeting of the national council of measurement in
education, New Orleans, Louisiana
7.
JohnsonM. S.JenkinsF. (2005). A Bayesian hierarchical model for
large-scale educational surveys: An application to the National Assessment of
Educational Progress. ETS RR-04-38. Princeton,
NJ: Educational Testing
Service.
8.
LordF. M. (1980). Applications of item response theory to
practical testing problems. Hillsdale, NJ:
Lawrence Erlbaum Associates,
Inc.
9.
MastersG. N. (1982). A Rasch model for partial credit
scoring. Psychometrika, 47,
149–174.
10.
MazzeoJ.DonoghueJ. R.LiD.JohnsonM. (2006). Marginal estimation in NAEP:
Current operational procedures and AM. Report prepared
for National Center for Education Statistics (NCES) under task 2.2.8 of the “NAEP
in the New Millennium, Continuity and Innovation” contract
11.
MislevyR. (1993). Should “multiple imputations” be
treated as “multiple indicators?”. Psychometrika,
58, 79–85.
12.
MislevyR. J. (1984). Estimating latent
distributions. Psychometrika,
49, 359–381.
13.
MislevyR. J. (1985). Estimation of latent group
effects. Journal of the American Statistical
Association, 80,
993–997.
14.
MurakiE. (1992). A generalized partial credit
model: Application of an EM algorithm. Applied
Psychological Measurement, 14,
59–71.
15.
PfeffermanD.SkinnerC. J.HolmesD. J.GoldsteinH.RashbashJ. (1998). Weighting for unequal selection
probabilities in multilevel models. Journal of Royal
Statistics Society, B, 60,
23–40.
16.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications
and data analysis method. (2nd ed.).
Thousand Oaks, CA:
Sage.
17.
RaudenbushS. W.FotiuR. P.CheongY. F. (1999). Synthesizing results from the
trial state assessment. Journal of Educational and
Behavioral Statistics, 24,
413–438.
18.
SinharayS. (2003). Bayesian item fit analysis for
dichotomous item response models (ETS RR-03-34).
Princeton, NJ: Educational Testing
Service
19.
StapletonJ. H. (1995). Linear statistical models.
New York: John
Wiley.
20.
ThomasN. (1993). Asymptotic corrections for
multivariate posterior moments with factored likelihood functions.
Journal of Computations and Graphics Statistics,
2(3),309–322.
21.
van der LindenW. J.HambletonR. K. (1997). Handbook of modern item response
theory. New York:
Springer-Verlag.
22.
WolterK. M. (1985). Introduction to variance
estimation. New York:
Springer-Verlag.
