×
Laukaityte, Inga; Wiberg, Marie

Importance of sampling weights in multilevel modeling of international large-scale assessment data. (English) Zbl 1508.62020

Commun. Stat., Theory Methods 47, No. 20, 4991-5012 (2018).
Summary: Multilevel modeling is an important tool for analyzing large-scale assessment data. However, the standard multilevel modeling will typically give biased results for such complex survey data. This bias can be eliminated by introducing design weights which must be used carefully as they can affect the results. The aim of this paper is to examine different approaches and to give recommendations concerning handling design weights in multilevel models when analyzing large-scale assessments such as TIMSS (The Trends in International Mathematics and Science Study). To achieve the goal of the paper, we examined real data from two countries and included a simulation study. The analyses in the empirical study showed that using no weights or only level 1 weights sometimes could lead to misleading conclusions. The simulation study only showed small differences in estimation of the weighted and unweighted models when informative design weights were used. The use of unscaled or not rescaled weights however caused significant differences in some parameter estimates.

MSC:

62D05 Sampling theory, sample surveys

Keywords:

informative weights; two-stage sampling; rescaling weights; simulation study

Software:

sampling; Mplus; MLwiN; R
Cite Review PDF
Full Text: DOI

References:

[1] Asparouhov, T. 2006. General Multilevel modeling with sampling weights. Communications in statistics. Theory and methods 35 (3):439-60. doi:10.1080/03610920500476598. · Zbl 1084.62053
[2] Asparouhov, T., and Muthen & Muthen. 2004. Weighting for unequal probability of selection in multilevel modeling. Mplus Web Notes: No. 8. Retrieved from: https://www.statmodel.com/download/webnotes/MplusNote81.pdf. Accessed 03 July 2014.
[3] Asparouhov, T. 2005. Sampling weights in latent variable modeling. Structural Equation Modeling 12 (3):411-34. doi:10.1207/s15328007sem1203_4.
[4] Cai, T. 2013. Investigation of ways to handle sampling weights for multilevel model analyses. Sociological methodology 43 (1):178-219. doi:10.1177/0081175012460221. · doi:10.1177/0081175012460221
[5] Carle, A. C. 2009. Fitting multilevel models in complex survey data with design weights: Recommendations. BMC Medical Research Methodology 9 (49). Retrieved from: http://www.biomedcentral.com/1471-2288/9/49. Accessed 03 July 2014.
[6] Chantala, K., and C. Suchindran. 2006. Adjusting for unequal selection probability in multilevel models: A comparison of software packages. In JSM proceedings, survey research methods section. Alexandria, VA: American Statistical Association. 2815-24. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.627.4453&rep=rep1&type=pdf. Accessed 03 July 2014.
[7] Danielsen, A. G., N. Wiium, B. U. Wilhelmsen, and B. Wold. 2010. Perceived support provided by teachers and classmates and students’ self-reported academic initiative. Journal of School Psychology 48 (3):247-67. doi:10.1016/j.bbr.2011.03.031. · doi:10.1016/j.jsp.2010.02.002
[8] Howell, D. C. 2008. The analysis of missing data. In Handbook of social science methodology, ed. W. Outhwaite and S. Turner, (208-224). London, GB: Sage.
[9] Howie, S. J., and T. Plomp. 2004. The importance of national options in IEA International Comparative Studies: Exploring the effects of language proficiency upon secondary students’ performance in Timss’99 mathematics in South Africa. Proceedings of the IRC-2004. Retrieved from: http://www.iea.nl/fileadmin/user_upload/IRC/IRC_2004/Papers/IRC2004_Howie_Plomp.pdf. Accessed 03 July 2014.
[10] IEA. 2006. PIRLS 2006 international database. Retrieved from: https://timssandpirls.bc.edu/pirls2006/user_guide.html. Accessed 03 July 2014.
[11] IEA. (2007). TIMSS 2007 international database. Retrieved from: https://timssandpirls.bc.edu/TIMSS2007/idb_ug.html. Accessed 03 July 2014.
[12] IEA. 2011. TIMSS 2011 international database. Retrieved from: http://timssandpirls.bc.edu/timss2011/international-database.html. Accessed 03 July 2014.
[13] IDB Analyzer (Version 2). 2009. [computer software]. Hamburg, Germany: IEA Data Processing and Research center.
[14] Jenkins, F. 2008. Multilevel analysis with informative weights. In JSM Proceedings, Survey Research Methods Section. Alexandria, VA: American Statistical Association. 2226-33. Retrieved from https://www.amstat.org/sections/SRMS/Proceedings/y2008/Files/301419.pdf. Accessed 03 July 2014.
[15] Jianjun, W. 2000. Relevance of the hierarchical linear model to TIMSS data analyses. opinion papers. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA.
[16] Joncas, M. 2008. TIMSS 2007 sampling weights and participation rates. In TIMSS 2007 technical report, ed. J. F. Olson, M. O. Martin, and I. V. S. Mullis, 153-92. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
[17] Kim, J. S., C. J. Anderson, and B. Keller. 2013. Multilevel analysis of assessment data. In Chapman and Hall/CRC statistics in the social and behavioral: Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis, ed. L. Rutkowski, M. von Davier, D. Rutkowski, 389-424. Boca Raton, FL, USA: Chapman & Hall/CRC Press. Retrieved from http://www.ebrary.com. Accessed 13 October 2015.
[18] Kish, L. 1965. Survey sampling. New York, NY: Wiley. · Zbl 0151.23403
[19] Kyriakides, L., and C. Charalambous. 2004. Extending the scope of analyzing data of IEA studies: Applying multilevel modelling techniques to analyse TIMSS data. Proceedings of the IRC-2004. Retrieved from http://www.iea.nl/fileadmin/user_upload/IRC/IRC_2004/Papers/IRC2004_Kyriakides_Charalambous.pdf
[20] Lamb, S., and S. Fullarton. 2002. Classroom and school factors affecting mathematics achievement: A comparative study of Australia and the United States using TIMSS. Australian Journal of Education 46 (2):154-71. doi:10.1177/000494410204600205. · doi:10.1177/000494410204600205
[21] Marsh, H. W., U. Trautwein, O. Lüdtke, O. Köller, and J. Baumert. 2005. Academic self-concept, interest, grades and standardized test scores: Reciprocal effects models of causal ordering. Child development 76 (2):397-416. doi:10.1111/j.1467-8624.2005.00853.x. · doi:10.1111/j.1467-8624.2005.00853.x
[22] Mohammadpour, E., and M. Ghafar. 2012. Mathematics achievement as a function of within- and between school differences. Scandinavian Journal of Educational Research, 58 (2):189-221. doi:10.1080/00313831.2012.725097
[23] Muthén, L. K., and B. O. Muthén. 1998-2011. [computer software]. MPLUS Version 7. Los Angeles, CA: Muthén & Muthén.
[24] Olson, J. F., M. O. Martin, and I. V. S. Mullis. (Ed.). 2008. TIMSS 2007 technical report. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
[25] Pauli, C., K. Reusser, and U. Grob. 2007. Teaching for understanding and/or self-regulated learning? A video-based analysis of reform-oriented mathematics instruction in Switzerland. International Journal of Educational Research 46:294-305. doi:10.1016/j.ijer.2007.10.004. · doi:10.1016/j.ijer.2007.10.004
[26] Pfeffermann, D. 1993. The role of sampling weights when modeling survey data. International Statistical Review 61 (2):317-37. doi:10.2307/1403631. · Zbl 0779.62009 · doi:10.2307/1403631
[27] Pfeffermann, D., C. J. Skinner, D. J. Holmes, H. Goldstein, and J. Rasbash. 1998. Weighting for unequal selection probabilities in multilevel models. Journal of the Royal Statistical Society, Series B 60:23-40. doi:10.1111/1467-9868.00106. · Zbl 0909.62006 · doi:10.1111/1467-9868.00106
[28] R Development Core Team. 2014. [Computer software]. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from: http://www.R-project.org/. Accessed 03 July 2014.
[29] Rasbash, J., C. Charlton, W. J. Browne, M. Healy, and B. Cameron. 2005. [Computer software]. MLwiN Version 2.02. Centre for Multilevel Modelling, University of Bristol.
[30] Rabe-Hesketh, S., and A. Skrondal. 2006. Multilevel modelling of complex survey data. Journal of the Royal Statistical Society, Series A 169 (4):805-27. doi:10.1111/j.1467-985X.2006.00426.x. · doi:10.1111/j.1467-985X.2006.00426.x
[31] Rubin, D. B. 1987. Multiple imputations for non-response in surveys. New York, NY: Wiley. · Zbl 1070.62007 · doi:10.1002/9780470316696
[32] Rutkowski, L., E. Gonzalez, M. Joncas, and M. von Davier. 2010. International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher 39 (142):142-51. doi:10.3102/0013189X10363170. · doi:10.3102/0013189X10363170
[33] Sabah, S., and H. Hammouri. 2010. Does subject matter? Estimating the impact of instructional practices and resources on student achievement in science and mathematics: Findings from TIMSS 2007. Evaluation & Research in Education 23 (4):287-99. doi:10.1080/09500790.2010.509782.
[34] Schafer, J. L. 1997. Analysis of incomplete multivariate data. London, GB: Chapman & Hall. · Zbl 0997.62510 · doi:10.1201/9781439821862
[35] Schafer, J. L., and J. W. Graham. 2002. Missing data: Our view of the state of the art. Psychological Methods 7 (2):147-77. doi:10.1037//1082-989X.7.2.147. · doi:10.1037/1082-989X.7.2.147
[36] Snijders, T. A. B., and R. J. Bosker. 2012. Multilevel analysis: An introduction to basic and advanced multilevel modeling. 2nd ed. London: Sage. · Zbl 1296.62008 · doi:10.1016/S0147-1767(03)00054-3
[37] Stapleton, L. 2013. Incorporating sampling weights into single- and multilevel analyses. In Chapman and Hall/CRC statistics in the social and behavioral: Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis, ed. L. Rutkowski, M. von Davier, and D. Rutkowski, 363-88. Boca Raton, FL, USA: Chapman & Hall/CRC Press. Retrieved from http://www.ebrary.com. Accessed 13 October 2015.
[38] Tillé, Y., and A. Matei. 2013. Sampling: Survey sampling. R package version 2.6. Retrieved from: http://CRAN.R-project.org/package=sampling. Accessed 03 July 2014.
[39] Webster, B. J., and D. L. Fisher. 2010. Accounting for variation in science and mathematics achievement: A multilevel analysis of Australian data third international mathematics and science study (TIMSS). School Effectiveness and School Improvement: An International Journal of Research, Policy and Practice 11 (3):339-60. doi:10.1076/0924-3453(200009)11:3;1-G;FT339.
[40] Zaccarin, S., and C. Donati. 2008. The effects of sampling weights in multilevel analysis of PISA data. Working Paper n. 119. Retrieved from: http://www2.units.it/nirdses/sito_inglese/working
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.