Abstract
Conversation-based assessments (CBAs) make use of dialogue systems to create situations that can be used to gather evidence of the decisions learners make on performance-based assessments such as computer‐based scenarios, simulations, or other forms of technology rich tasks. CBAs are adaptive in nature providing subsequent questions, feedback and eliciting additional information from learners about the given topic. However, the adaptivity can always be improved with additional evidence of the learner’s knowledge, skills, and other attributes (KSAs) which can be encompassed in a learner model. Learner models based on CBA’s data can then be used to support and improve adaptive sequencing of questions and conversations as well as adaptive feedback. This information can also aid in selecting appropriate conversations and additional tasks. These improvements may result in more engaging and relevant conversations and tasks. In this paper, we discuss some of the challenges and opportunities for learner modeling in dialogue systems.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abyaa, A., Idrissi, M.K., Bennani, S.: Learner modelling: systematic review of the literature from the last 5 years. Educ. Technol. Res. Dev. 67, 1–39 (2019)
Adamson, D., Dyke, G., Jang, H.J., Rosé, C.P.: Towards an agile approach to adapting dynamic collaboration support to student needs. Int. J. AI Educ. 24(1), 91–121 (2014)
Aleven, V., McLaughlin, E.A., Glenn, R.A., Koedinger, K.R.: Instruction based on adaptive learning technologies. In: Mayer, R.E., Alexander, P.A. (eds.) Handbook of Research on Learning and Instruction, 2nd edn., pp. 522–560. Routledge, New York (2016)
Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4, 167–207 (1995)
Andrews-Todd, J., Forsyth, C., Steinberg, J., Rupp, A.A.: Identifying profiles of collaborative problem solvers in an online electronics environment. In: Boyer, K.E., Yudelson, M. (eds.) Proceedings of the 11th International Conference on Educational Data Mining, pp. 239–245. International Educational Data Mining Society, Buffalo (2018)
Baker, R.S., D’Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to be frustrated than bored: the incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. Int. J. Hum. Comput. Stud. 68(4), 223–241 (2010)
Blessing, S.B., Gilbert, S.B., Ourada, S., Ritter, S.: Authoring model-tracing cognitive tutors. Int. J. Artif. Intell. Educ. 19(2), 189–210 (2009)
Bull, S.: There are open learner models about! IEEE Trans. Learn. Technol. 13(2), 425–448 (2020)
Bull, S., Brna, P., Pain, H.: Extending the scope of the student model. User Model User-Adap. Inter. 5, 45–65 (1995)
Conati, C., Kardan, S.: Student modeling: supporting personalized instruction, from problem solving to exploratory open-ended activities. AI Mag. 34(3), 13–26 (2013)
Conati, C., Porayska-Pomsta, K., Mavrikis, M.: AI in education needs interpretable machine learning: lessons from Open Learner Modelling. arXiv (2018). http://arxiv.org/abs/1807.00154
Desmarais, M.C., d Baker, R.S.J.: A review of recent advances in learner and skill modeling in intelligent learning environments. User Model. User-Adap. Inter. 22(1–2), 9–38 (2012). https://doi.org/10.1007/s11257-011-9106-8
Dimitrova, V.: STyLE-OLM: interactive open learner modelling. Int. J. Artif. Intell. Educ. 13(1), 35–78 (2003)
Dimitrova, V., Brna, P.: From interactive open learner modelling to intelligent mentoring: STyLE-OLM and beyond. Int. J. Artif. Intell. Educ. 26(1), 332–349 (2015). https://doi.org/10.1007/s40593-015-0087-3
D’Mello, S.K., Graesser, A.: Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Model. User-Adap. Inter. 20(2), 147–187 (2010). https://doi.org/10.1007/s11257-010-9074-4
D’Mello, S., Graesser, A.: Dynamics of affective states during complex learning. Learn. Instr. 22(2), 145–157 (2012)
Driscoll, D.M., Craig, S.D., Gholson, B., Ventura, M., Hu, X.: Vicarious learning. Effects of overhearing dialog and monologue-like virtual tutoring sessions. J. Exp. Psychol.: Hum. Learn. Mem. 6, 588–598 (2003)
Forsyth, C.M., Andrews-Todd, J., Steinberg, J.: Are you really a team player?: profiles of collaborative problem solvers in an online environment. In: Rafferty, A.N., Whitehill, J., Cavalli-Sforza, V., Romero, C. (eds.) Proceedings of the13th International Conference on Educational Data Mining (EDM 2020), pp. 403–408 (2020)
Forsyth, C.M., Graesser, A.C., Millis, K.: Predicting learning in a multi-component serious game. Technol. Knowl. Learn. 25, 251–277 (2020)
Forsyth, C.M., Graesser, A.C., Pavlik, P., Millis, K., Samei, B.: Discovering theoretically grounded predictors of shallow vs. deep- level learning. In: Stamper, J., Pardos, Z., Mavrikis, M., McLaren, B.M. (eds.) Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014), pp. 229–232 (2014)
Forsyth, C.M., Peters, S., Moon, J., Napolitano, D.: Assessing scientific inquiry based on multiple sources of evidence. Presented at the Annual Meeting of the American Educational Research Association, Toronto, Canada, April 2019
Forsyth, C.M., Peters, S., Zapata-Rivera, D., Lentini, J., Graesser, A., Cai, Z.: Interactive score reporting: an AutoTutor-based system for teachers. In: André, E., Baker, R., Hu, X., Rodrigo, M.M.T., du Boulay, B. (eds.) AIED 2017. LNCS (LNAI), vol. 10331, pp. 506–509. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61425-0_51
Graesser, A.C.: Conversations with AutoTutor help students learn. Int. J. Artif. Intell. Educ. 26, 124–132 (2016)
Graesser, A.C., Forsyth, C., Lehman, B.: Two heads are better than one: learning from agents in conversational trialogues. Teach. Coll. Rec. 119, 1–20 (2017)
Graesser, A.C., Person, N.K.: Question asking during tutoring. Am. Educ. Res. J. 31, 104–137 (1994)
Graesser, A.C., Person, N.K., Harter, D.: The tutoring research group: teaching tactics and dialogue in AutoTutor. Int. J. Artif. Intell. Educ. 12, 257–279 (2001)
Greer, J., McCalla, G. (eds.): Student Models: The Key to Individualized Educational Systems. Springer, New York (1994)
Gunning, D.: Explainable Artificial Intelligence (XAI). Defense Advanced Research Projects Agency (DARPA) (2017)
Hao, J., Zapata-Rivera, D., Graesser, A.C., Cai, Z., Hu, X., Goldberg, B.: Towards an intelligent tutor for teamwork: responding to human sentiments. In: Sottilare, R., Graesser, A., Hu, X., Sinatra, A.M. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 6 - Team Tutoring, pp. 151–160. Army Research Laboratory, Orlando (2018). ISBN 978-0-9977257-4-2
Hsiao, I.H., Brusilovsky, P.: Guiding and motivating students through open social student modeling: lessons learned. Teach. Coll. Rec. 119(3), 1–42 (2017)
Kay, J., Zapata-Rivera, D., Conati, C.: The GIFT of scrutable learner models: why and how. In: Sinatra, R.A.M., Graesser, A.C., Hu, X., Goldberg, B., Hampton, A.J. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 8 – Data Visualization, pp. 25–40. U.S. Army CCDC - Soldier Center, Orlando (2020)
Katz, S., Albacete, P., Chounta, I.-A., Jordan, P., McLaren, B.M., Zapata-Rivera, D.: Linking dialogue with student modelling to create an adaptive tutoring system for conceptual physics. Int. J. Artif. Intell. Educ. 31(3), 397–445 (2021). https://doi.org/10.1007/s40593-020-00226-y
Kerly, A., Ellis, R., Bull, S.: CALMsystem: a conversational agent for learner modelling. In: Ellis, R., Allen, T., Petridis, M. (eds.) Applications and Innovations in Intelligent Systems XV, pp. 89–102. Springer, London (2008). https://doi.org/10.1007/978-1-84800-086-5_7
Latané, B., Williams, K., Harkins, S.: Many hands make light the work: the causes and consequences of social loafing. J. Pers. Soc. Psychol. 37(6), 822–832 (1979)
Long, Y., Aleven, V.: Enhancing learning outcomes through self-regulated learning support with an open learner model. User Model. User-Adap. Inter. 27(1), 55–88 (2017)
Lopez, A.A., Guzman-Orth, D., Zapata-Rivera, D., Forsyth, C.M., Luce, C.: Examining the accuracy of a conversation-based assessment in interpreting English learners’ written responses (Research Report No. RR-21-03). Educational Testing Service (2021).https://doi.org/10.1002/ets2.12315
Loukina, A., Madnani, N., Zechner, K.: The many dimensions of algorithmic fairness in educational applications. In: Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, Florence, Italy, pp. 1–10 (2019)
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. CoRR, abs/1908.09635 (2019). http://arxiv.org/abs/1908.09635
Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A., Halpern, D.: Operation ARIES!: a serious game for teaching scientific inquiry. In: Ma, M., Oikonomou, A., Jain, L.C. (eds.) Serious Games and Edutainment Applications, pp. 169–195. Springer, London (2011). https://doi.org/10.1007/978-1-4471-2161-9_10
Mislevy, R.J., Almond, R.G., Lukas, J.F.: A brief introduction to evidence‐centered design. ETS Res. Rep. Ser. 2003(1), i-29 (2003)
Mislevy, R.J., Riconscente, M.M.: Evidence-centered assessment design. In: Handbook of Test Development, pp. 75–104. Routledge (2011)
Mitrovic, A.: Fifteen years of constraint-based tutors: what we have achieved and where we are going. User Model. User-Adap. Inter. 22(1–2), 39–72 (2012). https://doi.org/10.1007/s11257-011-9105-9
Mitrovic, A., Martin, B., Suraweera, P.: Intelligent tutors for all: constraint-based modeling methodology, systems and authoring. IEEE Intell. Syst. 22, 38–45 (2007)
Pavlik, P.I., Brawner, K., Olney, A., Mitrovic, A.: A review of student models used in intelligent tutoring systems. In: Sollitare, R.Z., Graesser, A.C., Hu, X., Holden, H. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 1 - Learner Modeling, pp. 39–68. U.S. Army Research, Orlando (2013)
Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019)
Rosé, C.P., McLaughlin, E.A., Liu, R., Koedinger, K.R.: Explanatory learner models: why machine learning (alone) is not the answer. Br. J. Edu. Technol. 50(6), 2943–2958 (2019)
Schaldenbrand, P., et al.: Computer-supported human mentoring for personalized and equitable math learning. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds.) AIED 2021. LNCS (LNAI), vol. 12749, pp. 308–313. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-78270-2_55
Shahrour, G., Bull, S.: Interaction preferences and learning in an inspectable learner model for language. In: Artificial Intelligence in Education, pp. 659–661. IOS Press (2009)
Shute, V.J.: SMART: student modeling approach for responsive tutoring. User Model. User-Adap. Inter. 5, 1–44 (1995). https://doi.org/10.1007/BF01101800
Shute, V.J., Zapata-Rivera, D.: Adaptive educational systems. In: Durlach, P. (ed.) Adaptive Technologies for Training and Education, pp. 7–27. Cambridge University Press, New York (2012)
Somyürek, S., Brusilovsky, P., Guerra, J.: Supporting knowledge monitoring ability: open learner modeling vs. open social learner modeling. Res. Pract. Technol. Enhanc. Learn. 15(1), 1–24 (2020). https://doi.org/10.1186/s41039-020-00137-5
Sottilare, R.A., Brawner, K.W., Sinatra, A.M., Johnston, J.H.: An Updated Concept for a Generalized Intelligent Framework for Tutoring (GIFT). US Army Research Laboratory, Orlando (2017)
Sottilare, R., Barr, A., Robson, R., Hu, X., Graesser, A.: Exploring the opportunities and benefits of standards for adaptive instructional systems (AISs). In: Proceedings of the Adaptive Instructional Systems Workshop in the Industry Track of the 14th International Intelligent Tutoring Systems, pp. 49–53 (2018)
Tang, L.M., Kay, J.: Scaffolding for an OLM for long-term physical activity goals. In: Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization, pp. 147–156 (2018)
Thomson, D., Mitrovic, A.: Preliminary evaluation of a negotiable student model in a constraint-based ITS. Res. Pract. Technol. Enhanc. Learn. 5(01), 19–33 (2010)
VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A., Rose, C.P.: When are tutorial dialogues more effective than reading? Cogn. Sci. 3, 3–62 (2007)
Vincent-Lancrin, S., van der Vlies, R.: Trustworthy artificial intelligence (AI) in education: promises and challenges. OECD Education Working Papers, No. 218, OECD Publishing, Paris (2020). https://doi.org/10.1787/a6c90fa9-en
Zapata-Rivera, D.: Open student modeling research and its connections to educational assessment. Int. J. Artif. Intell. Educ. 31(3), 380–396 (2020). https://doi.org/10.1007/s40593-020-00206-2
Zapata-Rivera, D., Brawner, K., Jackson, G.T., Katz, I.R.: Reusing evidence in assessment and intelligent tutors. In: Sottilare, R., Graesser, A., Hu, X., Goodwin, G. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 5 - Assessment Methods, pp. 125–136. U.S. Army Research Laboratory, Orlando (2017). ISBN 978-0-9893923-9-6
Zapata-Rivera, J.D., Greer, J.: Interacting with Bayesian student models. Int. J. Artif. Intell. Educ. 14(2), 127–163 (2004)
Zapata-Rivera, D., Greer, J.E.: Exploring various guidance mechanisms to support interaction with inspectable learner models. In: Cerri, S.A., Gouardères, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 442–452. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-47987-2_47
Zapata-Rivera, D., Jackson, T., Katz, I.R.: Authoring conversation-based assessment scenarios. In: Sottilare, R.A., Graesser, A.C., Hu, X., Brawner, K. (eds.) Design Recommendations for Intelligent Tutoring Systems Volume 3: Authoring Tools and Expert Modeling Techniques, pp. 169–178. U.S. Army Research Laboratory (2015)
Zapata-Rivera, D., Jackson, T., Liu, L., Bertling, M., Vezzu, M., Katz, I.: Assessing science inquiry skills using trialogues. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 625–626. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_84
Zapata-Rivera, D., Hansen, E., Shute, V.J., Underwood, J.S., Bauer, M.: Evidence-based approach to interacting with open student models. Int. J. Artif. Intell. Educ. 17(3), 273–303 (2007)
Zapata-Rivera, D., Lehman, B., Sparks, J.R.: Learner modeling in the context of caring assessments. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2020. LNCS, vol. 12214, pp. 422–431. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50788-6_31
Zapata-Rivera, D., Liu, L., Chen, L., Hao, J., von Davier, A.A.: Assessing science inquiry skills in an immersive, conversation-based scenario. In: Kei Daniel, B. (ed.) Big Data and Learning Analytics in Higher Education, pp. 237–252. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-06520-5_14
Zapata-Rivera, D., Liu, L., Katz, I.R., Vezzu, M.: Exploring the use of game elements in the development of innovative assessment tasks for science. Cogn. Technol. 18(1), 43–50 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Rightsholder, under exclusive licence to Springer Nature Switzerland AG, part of Springer Nature
About this paper
Cite this paper
Zapata-Rivera, D., Forsyth, C.M. (2022). Learner Modeling in Conversation-Based Assessment. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. HCII 2022. Lecture Notes in Computer Science, vol 13332. Springer, Cham. https://doi.org/10.1007/978-3-031-05887-5_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-05887-5_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05886-8
Online ISBN: 978-3-031-05887-5
eBook Packages: Computer ScienceComputer Science (R0)