Complex survey data are collected by means other than simple random samples. This creates two analytical issues: nonindependence and unequal selection probability. Failing to address these issues results in underestimated standard errors and biased parameter estimates. Using data from the nationally representative Head Start Family and Child Experiences Survey (FACES; 1997 and 2000 cohorts), three diverse multilevel models are presented that illustrate differences in results depending on addressing or ignoring the complex sampling issues. Limitations of using complex survey data are reported, along with recommendations for reporting complex sample results.
Bell-EllisonB. A.KromreyJ. D.2007. “Alternatives for Analysis of Complex Sample Surveys: A Comparison of SAS, SUDAAN, and AM Software.”Paper presented at SAS Global Forum, Orlando, FL, April16.
2.
BerglundP. A.2002. “Analysis of Complex Sample Survey Data: Using the SURVEYMEANS and SURVEYREG Procedures and Macro Coding.”Paper presented at SAS User’s Group International (SUGI), Orlando, FL, April14.
3.
BiemerP. P.ChristS. L. (2008). “Weighting Survey Data.”. In International Handbook of Survey Methodology (pp. 317–341), edited by DeLeeuwE. D.New York, NY: Lawrence Erlbaum.
BroganD. J. (1998). “Pitfalls of Using Standard Statistical Software Packages for Sample Survey Data.”. In ArmitageP.ColtonT. (Eds.), Encyclopedia of Biostatistics. New York, NY: Wiley.
6.
Bulotsky-ShearerR. J.WenX.FariaA.Hahs-VaughnD. L.KorfmacherJ. (under review). National profiles of classroom quality and family involvement: A multilevel examination of proximal influences on Head Start children's school readiness.
7.
CampbellC.MeyerM.1978. “Some Properties of t Confidence Intervals for Survey Data.”Paper presented at American Statistical Association Annual Meeting, Washington, DC.
8.
CavinEdward S.OhlsJames C.1990. “An Application of Balanced Repeated Replication (BRR) Variance Estimation to Program Evaluation.”Evaluation Review14:206–13.
9.
DuMouchelW. H.DuncanG. J. (1983). “Using Sample Survey Weights in Multiple Regression Analyses of Stratified Samples.”. Journal of the American Statistical Association, 78, 535–43.
10.
FayR. E.1989. Theoretical application of weighting for variance Calculation. Proceedings of the Section on Survey Research Methods of the American Statistical Association (pp. 212–217).
11.
FrankelM. R.1971. Inference From Survey Samples. Ann Arbor, MI: Institute for Social Research.
12.
Hahs-VaughnD. L. (2005). “A Primer for Using and Understanding Weights with National Datasets.”. Journal of Experimental Education, 73, 221–48.
13.
Hahs-VaughnD. L. (2006a). “Analysis of Data From Complex Samples.”. International Journal of Research & Method in Education, 29, 163–81.
14.
Hahs-VaughnD. L. (2006b). “Utilization of Sample Weights in Single Level Structural Equation Modeling.”. Journal of Experimental Education, 74, 163–90.
15.
Hahs-VaughnD. L. (2006c). “Weighting Omissions and Best Practices When Using Large-Scale Data in Educational Research.”Association for Institutional Research Professional File 101, 1–9.
16.
HindmanA. H.SkibbeL. E.MillerA.ZimmermanM. (2010). “Ecological Contexts and Early Learning: Contributions of Child, Family, and Classroom Factors during Head Start, to Literacy and Mathematics Growth through First Grade.”Early Childhood Research Quarterly, 25, 235–50.
17.
HippJ. R.BauerD. J.2006. “Local solutions in the estimation of growth mixture models.”Psychological Methods, 11:36–53.
18.
HofferthS. L. (2005). “Secondary Data Analysis in Family Research.”Journal of Marriage and Family, 67, 891–907.
19.
HongG.RaudenbushS. W. (2005). “Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics.”Educational Evaluation and Policy Analysis, 27, 205–24.
20.
HoxJ. J.KreftI. G. G. (1994). “Multilevel Analysis Methods.”Sociological Methods and Research, 22, 283–99.
21.
HuangG.SalvucciS.PengS.OwingsJ. (1996). National Educational Longitudinal Study of 1988 (Nels:88) Research Framework and Issues. Arlington, VA: Synetics for Management Decisions.
22.
KaltonG. (1983a). Introduction to Survey Sampling. Thousand Oaks, CA: Sage.
23.
KaltonG. (1983b). “Models in the Practice of Survey Sampling.”International Statistical Review, 51, 175–88.
24.
KaltonG. (1989). “Modeling Considerations: Discussion from a Survey Sampling Perspective.” In Panel Surveys, edited by KasprzykD.DuncanG.KaltonG.SinghM.New York, NY: Wiley.
25.
KaplanD.FergusonA. J. (1999). “On the Utilization of Sample Weights in Latent Variable Models.”Structural Equation Modeling, 6, 305–21.
26.
KishL. (1965). Survey Sampling. New York, NY: Wiley.
27.
KishL. (1992). “Weighting for Unequal Pi.”Journal of Official Statistics, 8, 183–200.
28.
KishL.FrankelM. R.1973. “Inference from Complex Samples.”Paper presented at annual meeting of the Royal Statistical Society, October17.
29.
KishL.FrankelM. R. (1974). “Inference from Complex Samples.”Journal of the Royal Statistical Society, Series B, 36, 1–37.
30.
KornE. L.GraubardB. I. (1995). “Examples of Differing Weighted and Unweighted Estimates from a Sample Survey.”American Statistician, 49, 291–305.
31.
KovarJ. G.RaoJ. N. K.WuC. F. J. (1988). “Bootstrap and Other Methods to Measure Errors in Survey Estimates.”The Canadian Journal of Statistics, 16, 25–45.
32.
LandisR. J.LepkowskiJ. M.EklundS. A.StehouwerS. A. (1982). “A Statistical Methodology for Analyzing Data from a Complex Survey: The First National Health and Nutrition Examination Survey.” In Vital and Health Statistics. Hyattsville, MD: National Center for Health Statistics.
33.
LeeE. S.ForthoferR. N.LorimorR. J. (1989). Analyzing Complex Survey Data. Newbury Park, CA: Sage.
34.
LemeshowS.LevyP. S. (1979). “Estimating the Variance of Ratio Estimates in Complex Sample Surveys with Two Primary Units per Stratum: A Comparison of Balanced Repeated Replication and Jackknife Techniques.”Journal of Statistical Computing and Simulations, 8, 191–205.
35.
LumleyT. (2004). “Analysis of Complex Survey Samples.”Journal of Statistical Software, 9:1–19.
36.
MahalanobisP. C.1946. On large-scale sample surveys. Philosophical Transactions of the Royal Society of London, series B, 231, 329–451.
37.
McWayneC. M.Hahs-VaughnD. L.CheungK.WrightL. G. (submitted for review). National profiles of school readiness skills for Head Start children: An investigation of stability and change. Early Childhood Research Quarterly.
38.
McCallR. B.AppelbaumM. I. (1991). “Some Issues of Conducting Secondary Analyses.”Developmental Psychology, 27, 911–17.
39.
McCarthyP. J.1966. Replication: An approach to the analysis of data from complex surveys. Public Health Service Publication Number 1000, series 2, number 14. Washington, DC: U.S. Department of Health, Education, and Welfare.
MuthenB. O.SatorraA. (1995). “Complex Sample Data in Structural Equation Modeling.”Sociological Methodology, 25, 267–316.
42.
O’BrienR. W.D’ElioM. A.Vaden-KiernanM.MageeC.YounoszaiT.IKeaneM. J.ConnellD. C.HaileyL. (2002). A Descriptive Study of Head Start Families: FACES Technical Report I. Washington, DC: Department of Health and Human Services Administration for Children and Families Head Start Bureau.
43.
PengS. S.2000. “Technical Issues in Using NCES Data.”Paper presented at AIR/NCES National Data Institute on the Use of Postsecondary Databases, Gaithersburg, MD.
44.
PfeffermannD.SkinnerC. J.HolmesD. J.GoldsteinH.RasbashJ. (1998). “Weighting for Unequal Selection Probabilities in Multilevel Models.”Journal of the Royal Statistical Society B, 60, 23–40.
45.
PikeG. R. (2008). “Using Weighting Adjustments to Compensate for Survey Nonresponse.”Research in Higher Education, 49, 153–171.
46.
PotthoffR. F.WoodburyM. A.MantonK. G. (1992). “‘Equivalent sample size’ and ‘equivalent degrees of freedom’ refinements for inference using survey weights under superpopulation models.”Journal of the American Statistical Association, 87, 383–96.
47.
RaudenbushS. W.BrykA. S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods. Thousand Oaks, CA: Sage.
RustK. F.RaoJ. N. K. (1996). “Variance Estimation for Complex Surveys Using Replication Techniques.”Survey Methods in Medical Research, 5, 283–310.
50.
SchenkerN.GentlemanJ. F. (2001). “On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals.”The American Statistician, 55, 182–7.
51.
SkinnerC. J. (1989). “Domain Means, Regression and Multivariate Analysis.” In Analysis of Complex Surveys (pp. 59–88), edited by SkinnerC. J.HoltD.SmithT. M. F.. New York, NY: Wiley.
52.
SkinnerC. J.HoltD.SmithT. M. F., eds. 1989. Analysis of Complex Samples. New York, NY: Wiley.
53.
StapletonL. M. (2002). “The Incorporation of Sample Weights into Multilevel Structural Equation Models.”Structural Equation Modeling, 9, 475–502.
54.
StapletonL. M. (2006). “An Assessment of Practical Solutions for Structural Equation Modeling with Complex Sample Data.”Structural Equation Modeling, 13, 28–58.
55.
StapletonL. M.ThomasS. L. (2008). “The Use of National Datasets for Teaching and Research.” In O’ConnellA. A.McCoachD. B. (Eds.), Multilevel Modeling of Educational Data (pp. 11-59). Charlotte, NC: Information Age Publishing.
56.
ThomasS. L.HeckR. H. (2001). “Analysis of Large-Scale Secondary Data in Higher Education Research: Potential Perils Associated with Complex Sampling Designs.”Research in Higher Education, 42, 517–40.
57.
ThomasS. L.HeckRonald H.BauerK. W. (2005). “Weighting and Adjusting for Design Effects in Secondary Data Analyses.”New Directions for Institutional Research, 12, 51–72.
58.
WenX.Bulotskey-ShearerR. J.Hahs-VaughnD. L.KorfmacherJ., (submitted for review). Examination of Head Start program quality: Combining classroom quality and parent involvement to understand children's vocabulary, literacy, and mathematics achievement trajectories. Early Childhood Research Quarterly.
59.
WestJ.RathburnA.2004. “ECLS-K Technical Issues.”Paper presented at the American Educational Research Association Institute on Statistical Analysis for Education Policy, San Diego, CA.
60.
WolterK. (1985). Introduction to Variance Estimation.
New York, NY: Springer-Verlag.
61.
ZillN.KimK.SorongonA.HerbisonR.ClarkC. (2006). Head Start Family and Child Experiences Survey (FACES) 2000 Cohort: User's Guide. Washington, DC: United States Department of Health and Human Services; Administration for Children and Families; Office of Planning, Research and Evaluation.
62.
ZillN.KwangK.SorongonA.HerbisonR.ClarkC. (2005). Head Start Family and Child Experiences Survey (FACES) 1997 Cohort: User’s Guide. Ann Arbor, MI: Inter-University Consortium for Political and Social Research at the Institute for Social Research, University of Michigan and the United States Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation.
