×

Partial-mastery cognitive diagnosis models. (English) Zbl 1478.62343

Summary: Cognitive diagnosis models (CDMs) are a family of discrete latent attribute models that serve as statistical basis in educational and psychological cognitive diagnosis assessments. CDMs aim to achieve fine-grained inference on individuals’ latent attributes, based on their observed responses to a set of designed diagnostic items. In the literature CDMs usually assume that items require mastery of specific latent attributes and that each attribute is either fully mastered or not mastered by a given subject. We propose a new class of models, partial mastery CDMs (PM-CDMs), that generalizes CDMs by allowing for partial mastery levels for each attribute of interest. We demonstrate that PM-CDMs can be represented as restricted latent class models. Relying on the latent class representation, we propose a Bayesian approach for estimation. We present simulation studies to demonstrate parameter recovery, to investigate the impact of model misspecification with respect to partial mastery and to develop diagnostic tools that could be used by practitioners to decide between CDMs and PM-CDMs. We use two examples of real test data – the fraction subtraction and the English tests – to demonstrate that employing PM-CDMs not only improves model fit, compared to CDMs, but also can make substantial difference in conclusions about attribute mastery. We conclude that PM-CDMs can lead to more effective remediation programs by providing detailed individual-level information about skills learned and skills that need to study.

MSC:

62P10 Applications of statistics to biology and medical sciences; meta analysis
62H25 Factor analysis and principal components; correspondence analysis

Software:

GDINA; CDM

References:

[1] Airoldi, E. M., Blei, D. M., Erosheva, E. A. and Fienberg, S. E. (2014). Handbook of Mixed Membership Models and Their Applications. CRC Press, Boca Raton, FL. · Zbl 1369.62003
[2] Blei, D. M. and Lafferty, J. D. (2007). A correlated topic model of Science. Ann. Appl. Stat. 1 17-35. · Zbl 1129.62122 · doi:10.1214/07-AOAS114
[3] Blei, D. M., Ng, A. Y. and Jordan, M. I. (2003). Latent Dirichlet allocation. J. Mach. Learn. Res. 3 993-1022. · Zbl 1112.68379
[4] Bolt, D. M. and Lall, V. F. (2003). Estimation of compensatory and noncompensatory multidimensional item response models using Markov chain Monte Carlo. Appl. Psychol. Meas. 27 395-414. · doi:10.1177/0146621603258350
[5] Chen, J. and de la Torre, J. (2013). A general cognitive diagnosis model for expert-defined polytomous attributes. Appl. Psychol. Meas. 37 419-437.
[6] Chen, Y., Liu, J., Xu, G. and Ying, Z. (2015). Statistical analysis of \(Q\)-matrix based diagnostic classification models. J. Amer. Statist. Assoc. 110 850-866. · Zbl 1373.62565 · doi:10.1080/01621459.2014.934827
[7] Chen, Y., Culpepper, S. A., Chen, Y. and Douglas, J. (2018). Bayesian estimation of the DINA Q matrix. Psychometrika 83 89-108. · Zbl 1402.62302 · doi:10.1007/s11336-017-9579-4
[8] Chiu, C.-Y., Douglas, J. A. and Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika 74 633-665. · Zbl 1179.62087 · doi:10.1007/s11336-009-9125-0
[9] Chiu, C.-Y., Köhn, H.-F., Zheng, Y. and Henson, R. (2016). Joint maximum likelihood estimation for diagnostic classification models. Psychometrika 81 1069-1092. · Zbl 1367.62313 · doi:10.1007/s11336-016-9534-9
[10] Culpepper, S. A. (2015). Bayesian estimation of the dina model with Gibbs sampling. J. Educ. Behav. Stat. 40 454-476.
[11] Culpepper, S. A. and Chen, Y. (2019). Development and application of an exploratory reduced reparameterized unified model. J. Educ. Behav. Stat. 44 3-24.
[12] Culpepper, S. A. and Hudson, A. (2018). An improved strategy for Bayesian estimation of the reduced reparameterized unified model. Appl. Psychol. Meas. 42 99-115.
[13] DeCarlo, L. T. (2011). On the analysis of fraction subtraction data: The DINA model, classification, class sizes, and the Q-matrix. Appl. Psychol. Meas. 35 8-26.
[14] DeCarlo, L. T. (2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Appl. Psychol. Meas. 36 447-468.
[15] de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. J. Educ. Meas. 45 343-362.
[16] de la Torre, J. (2009). DINA model and parameter estimation: A didactic. J. Educ. Behav. Stat. 34 115-130.
[17] de la Torre, J. (2011). The generalized DINA model framework. Psychometrika 76 179-199. · Zbl 1284.62775 · doi:10.1007/s11336-011-9207-7
[18] de la Torre, J. and Chiu, C.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika 81 253-273. · Zbl 1345.62167 · doi:10.1007/s11336-015-9467-8
[19] de la Torre, J. and Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika 69 333-353. · Zbl 1306.62527 · doi:10.1007/BF02295640
[20] DiBello, L. V., Stout, W. F. and Roussos, L. A. (1995). Unified cognitive psychometric diagnostic assessment likelihood-based classification techniques. In Cognitively Diagnostic Assessment (P. D. Nichols, S. F. Chipman and R. L. Brennan, eds.) 361-390. Erlbaum Associates, Hillsdale, NJ.
[21] Embretson, S. E. and Yang, X. (2013). A multicomponent latent trait model for diagnosis. Psychometrika 78 14-36. · Zbl 1284.62703 · doi:10.1007/s11336-012-9296-y
[22] Erosheva, E. A. (2004). Partial membership models with application to disability survey data. In Statistical Data Mining and Knowledge Discovery 117-134. CRC Press/CRC, Boca Raton, FL.
[23] Erosheva, E. A. (2005). Comparing latent structures of the grade of membership, Rasch, and latent class models. Psychometrika 70 619-628. · Zbl 1306.62409 · doi:10.1007/s11336-001-0899-y
[24] Erosheva, E. A., Fienberg, S. E. and Joutard, C. (2007). Describing disability through individual-level mixture models for multivariate binary data. Ann. Appl. Stat. 1 502-537. · Zbl 1126.62101 · doi:10.1214/07-AOAS126
[25] Erosheva, E., Fienberg, S. and Lafferty, J. (2004). Mixed-membership models of scientific publications. Proc. Natl. Acad. Sci. USA 101 5220-5227.
[26] Feng, Y., Habing, B. T. and Huebner, A. (2014). Parameter estimation of the reduced rum using the em algorithm. Appl. Psychol. Meas. 38 137-150.
[27] Galyardt, A. (2015). Interpreting mixed membership models: Implications of Erosheva’s representation theorem. In Handbook of Mixed Membership Models and Their Applications. Chapman & Hall/CRC Handb. Mod. Stat. Methods 39-65. CRC Press, Boca Raton, FL.
[28] George, A. C., Robitzsch, A., Kiefer, T., GroSS, J. and Ünlü, A. (2016). The R package CDM for cognitive diagnosis models. J. Stat. Softw. 74 1-24.
[29] Gu, Y. and Xu, G. (2019). The sufficient and necessary condition for the identifiability and estimability of the DINA model. Psychometrika 84 468-483. · Zbl 1431.62536 · doi:10.1007/s11336-018-9619-8
[30] Gu, Y. and Xu, G. (2020). Partial identifiability of restricted latent class models. Ann. Statist. 48 2082-2107. · Zbl 1473.62044 · doi:10.1214/19-AOS1878
[31] Gu, Y., Liu, J., Xu, G. and Ying, Z. (2018). Hypothesis testing of the \(Q\)-matrix. Psychometrika 83 515-537. · Zbl 1422.62347 · doi:10.1007/s11336-018-9629-6
[32] Haberman, S. (1995). Book review of “Statistical applications using fuzzy sets,” by K.G. Manton, M.A. Woodbury, and L.S. Corder. J. Amer. Statist. Assoc. 90 1131-1133.
[33] Hansen, M., Cai, L., Monroe, S. and Li, Z. (2016). Limited-information goodness-of-fit testing of diagnostic classification item response models. Br. J. Math. Stat. Psychol. 69 225-252. · Zbl 1406.91357
[34] Hartz, S. M. (2002). A Bayesian Framework for the Unified Model for Assessing Cognitive Abilities: Blending Theory with Practicality. ProQuest LLC, Ann Arbor, MI. Thesis (Ph.D.)-Univ. Illinois at Urbana-Champaign.
[35] Hartz, S. and Roussos, L. (2008). The fusion model for skills diagnosis: Blending theory with practicality. ETS Res. Rep. Ser. 2008 i-57.
[36] Henson, R. A., Templin, J. L. and Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika 74 191-210. · Zbl 1243.62140 · doi:10.1007/s11336-008-9089-5
[37] Hong, H., Wang, C., Lim, Y. S. and Douglas, J. (2015). Efficient models for cognitive diagnosis with continuous and mixed-type latent variables. Appl. Psychol. Meas. 39 31-43. · doi:10.1177/0146621614524981
[38] Junker, B. W. and Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Appl. Psychol. Meas. 25 258-272. · doi:10.1177/01466210122032064
[39] Kass, R. E. and Raftery, A. E. (1995). Bayes factors. J. Amer. Statist. Assoc. 90 773-795. · Zbl 0846.62028 · doi:10.1080/01621459.1995.10476572
[40] Köhn, H.-F. and Chiu, C.-Y. (2017). A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests. Psychometrika 82 112-132. · Zbl 1360.62527 · doi:10.1007/s11336-016-9536-7
[41] Köhn, H.-F. and Chiu, C.-Y. (2018). How to build a complete Q-matrix for a cognitively diagnostic test. J. Classification 35 273-299. · Zbl 1422.62222 · doi:10.1007/s00357-018-9255-0
[42] Liu, Y., Douglas, J. A. and Henson, R. A. (2007). Testing person fit in cognitive diagnosis. In Annual Meeting of the National Council on Measurement in Education (NCME), Chicago, IL, April.
[43] Liu, J., Xu, G. and Ying, Z. (2012). Data-driven learning of Q-matrix. Appl. Psychol. Meas. 36 548-564. · doi:10.1177/0146621612456591
[44] Liu, J., Xu, G. and Ying, Z. (2013). Theory of self-learning \(Q\)-matrix. Bernoulli 19 1790-1817. · Zbl 1294.68118 · doi:10.3150/12-BEJ430
[45] Ma, W. and de la Torre, J. (2019). Gdina: The generalized dina model framework. R package version, 2.3.2. Retrieved from https://CRAN.R-project.org/package=GDINA.
[46] Manton, K. G., Woodbury, M. A. and Tolley, H. D. (1994). Statistical Applications Using Fuzzy Sets. Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics. Wiley, New York. A Wiley-Interscience Publication. · Zbl 0811.62003
[47] Reckase, M. (2009). Multidimensional Item Response Theory 150. Springer, Berlin.
[48] Rupp, A. A. and Templin, J. L. (2008). Effects of Q-matrix misspecification on parameter estimates and misclassification rates in the DINA model. Educ. Psychol. Meas. 68 78-98. · doi:10.1177/0013164407301545
[49] Rupp, A. A., Templin, J. L. and Henson, R. A. (2010). Diagnostic Measurement: Theory, Methods, and Applications. Guilford Press, New York.
[50] Shang, Z., Erosheva, E. A. and Xu, G. (2021). Supplement to “Partial-mastery cognitive diagnosis models.” https://doi.org/10.1214/21-AOAS1439SUPP
[51] Stout, W., Henson, R., DiBello, L. and Shear, B. (2019). The reparameterized unified model system: A diagnostic assessment modeling approach. In Handbook of Diagnostic Classification Models 47-79. Springer, Berlin.
[52] Tatsuoka, K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In Diagnostic Monitoring of Skill and Knowledge Acquisition 453-488.
[53] Tatsuoka, C. (2002). Data analytic methods for latent partially ordered classification models. J. R. Stat. Soc. Ser. C. Appl. Stat. 51 337-350. · Zbl 1111.62381 · doi:10.1111/1467-9876.00272
[54] Tatsuoka, K. K. (2009). Cognitive Assessment: An Introduction to the Rule Space Method. Routledge, New York.
[55] Templin, J. and Bradshaw, L. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika 79 317-339. · Zbl 1288.62189 · doi:10.1007/s11336-013-9362-0
[56] Templin, J. L. and Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychol. Methods 11 287-305. · doi:10.1037/1082-989X.11.3.287
[57] Templin, J. L., Henson, R. A., Templin, S. E. and Roussos, L. (2008). Robustness of hierarchical modeling of skill association in cognitive diagnosis models. Appl. Psychol. Meas. 32 559-574. · doi:10.1177/0146621607300286
[58] von Davier, M. (2008). A general diagnostic model applied to language testing data. Br. J. Math. Stat. Psychol. 61 287-307. · doi:10.1348/000711007X193957
[59] Xu, G. (2017). Identifiability of restricted latent class models with binary responses. Ann. Statist. 45 675-707. · Zbl 1371.62010 · doi:10.1214/16-AOS1464
[60] Xu, G. and Shang, Z. (2018). Identifying latent structures in restricted latent class models. J. Amer. Statist. Assoc. 113 1284-1295. · Zbl 1402.62347 · doi:10.1080/01621459.2017.1340889
[61] Xu, G. and Zhang, S. (2016). Identifiability of diagnostic classification models. Psychometrika 81 625-649. · Zbl 1345.62166 · doi:10.1007/s11336-015-9471-z
[62] Zhao, S., Engelhardt, B. E., Mukherjee, S. and Dunson, D. B. (2018). Fast moment estimation for generalized latent Dirichlet models. J. Amer. Statist. Assoc. 113 1528-1540 · Zbl 1409.62119 · doi:10.1080/01621459.2017.1341839
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.