Skip to main content
Springer Nature Link
Log in

Learner Modeling in Conversation-Based Assessment

  • Conference paper
  • First Online:
Adaptive Instructional Systems (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13332))

Included in the following conference series:

Abstract

Conversation-based assessments (CBAs) make use of dialogue systems to create situations that can be used to gather evidence of the decisions learners make on performance-based assessments such as computer‐based scenarios, simulations, or other forms of technology rich tasks. CBAs are adaptive in nature providing subsequent questions, feedback and eliciting additional information from learners about the given topic. However, the adaptivity can always be improved with additional evidence of the learner’s knowledge, skills, and other attributes (KSAs) which can be encompassed in a learner model. Learner models based on CBA’s data can then be used to support and improve adaptive sequencing of questions and conversations as well as adaptive feedback. This information can also aid in selecting appropriate conversations and additional tasks. These improvements may result in more engaging and relevant conversations and tasks. In this paper, we discuss some of the challenges and opportunities for learner modeling in dialogue systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
eBook
USD 64.99
Price excludes VAT (USA)
Softcover Book
USD 84.99
Price excludes VAT (USA)

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abyaa, A., Idrissi, M.K., Bennani, S.: Learner modelling: systematic review of the literature from the last 5 years. Educ. Technol. Res. Dev. 67, 1–39 (2019)

    Article  Google Scholar 

  2. Adamson, D., Dyke, G., Jang, H.J., Rosé, C.P.: Towards an agile approach to adapting dynamic collaboration support to student needs. Int. J. AI Educ. 24(1), 91–121 (2014)

    Google Scholar 

  3. Aleven, V., McLaughlin, E.A., Glenn, R.A., Koedinger, K.R.: Instruction based on adaptive learning technologies. In: Mayer, R.E., Alexander, P.A. (eds.) Handbook of Research on Learning and Instruction, 2nd edn., pp. 522–560. Routledge, New York (2016)

    Google Scholar 

  4. Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4, 167–207 (1995)

    Article  Google Scholar 

  5. Andrews-Todd, J., Forsyth, C., Steinberg, J., Rupp, A.A.: Identifying profiles of collaborative problem solvers in an online electronics environment. In: Boyer, K.E., Yudelson, M. (eds.) Proceedings of the 11th International Conference on Educational Data Mining, pp. 239–245. International Educational Data Mining Society, Buffalo (2018)

    Google Scholar 

  6. Baker, R.S., D’Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to be frustrated than bored: the incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. Int. J. Hum. Comput. Stud. 68(4), 223–241 (2010)

    Article  Google Scholar 

  7. Blessing, S.B., Gilbert, S.B., Ourada, S., Ritter, S.: Authoring model-tracing cognitive tutors. Int. J. Artif. Intell. Educ. 19(2), 189–210 (2009)

    Google Scholar 

  8. Bull, S.: There are open learner models about! IEEE Trans. Learn. Technol. 13(2), 425–448 (2020)

    Article  Google Scholar 

  9. Bull, S., Brna, P., Pain, H.: Extending the scope of the student model. User Model User-Adap. Inter. 5, 45–65 (1995)

    Article  Google Scholar 

  10. Conati, C., Kardan, S.: Student modeling: supporting personalized instruction, from problem solving to exploratory open-ended activities. AI Mag. 34(3), 13–26 (2013)

    Google Scholar 

  11. Conati, C., Porayska-Pomsta, K., Mavrikis, M.: AI in education needs interpretable machine learning: lessons from Open Learner Modelling. arXiv (2018). http://arxiv.org/abs/1807.00154

  12. Desmarais, M.C., d Baker, R.S.J.: A review of recent advances in learner and skill modeling in intelligent learning environments. User Model. User-Adap. Inter. 22(1–2), 9–38 (2012). https://doi.org/10.1007/s11257-011-9106-8

    Article  Google Scholar 

  13. Dimitrova, V.: STyLE-OLM: interactive open learner modelling. Int. J. Artif. Intell. Educ. 13(1), 35–78 (2003)

    Google Scholar 

  14. Dimitrova, V., Brna, P.: From interactive open learner modelling to intelligent mentoring: STyLE-OLM and beyond. Int. J. Artif. Intell. Educ. 26(1), 332–349 (2015). https://doi.org/10.1007/s40593-015-0087-3

    Article  Google Scholar 

  15. D’Mello, S.K., Graesser, A.: Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Model. User-Adap. Inter. 20(2), 147–187 (2010). https://doi.org/10.1007/s11257-010-9074-4

    Article  Google Scholar 

  16. D’Mello, S., Graesser, A.: Dynamics of affective states during complex learning. Learn. Instr. 22(2), 145–157 (2012)

    Article  Google Scholar 

  17. Driscoll, D.M., Craig, S.D., Gholson, B., Ventura, M., Hu, X.: Vicarious learning. Effects of overhearing dialog and monologue-like virtual tutoring sessions. J. Exp. Psychol.: Hum. Learn. Mem. 6, 588–598 (2003)

    Google Scholar 

  18. Forsyth, C.M., Andrews-Todd, J., Steinberg, J.: Are you really a team player?: profiles of collaborative problem solvers in an online environment. In: Rafferty, A.N., Whitehill, J., Cavalli-Sforza, V., Romero, C. (eds.) Proceedings of the13th International Conference on Educational Data Mining (EDM 2020), pp. 403–408 (2020)

    Google Scholar 

  19. Forsyth, C.M., Graesser, A.C., Millis, K.: Predicting learning in a multi-component serious game. Technol. Knowl. Learn. 25, 251–277 (2020)

    Article  Google Scholar 

  20. Forsyth, C.M., Graesser, A.C., Pavlik, P., Millis, K., Samei, B.: Discovering theoretically grounded predictors of shallow vs. deep- level learning. In: Stamper, J., Pardos, Z., Mavrikis, M., McLaren, B.M. (eds.) Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014), pp. 229–232 (2014)

    Google Scholar 

  21. Forsyth, C.M., Peters, S., Moon, J., Napolitano, D.: Assessing scientific inquiry based on multiple sources of evidence. Presented at the Annual Meeting of the American Educational Research Association, Toronto, Canada, April 2019

    Google Scholar 

  22. Forsyth, C.M., Peters, S., Zapata-Rivera, D., Lentini, J., Graesser, A., Cai, Z.: Interactive score reporting: an AutoTutor-based system for teachers. In: André, E., Baker, R., Hu, X., Rodrigo, M.M.T., du Boulay, B. (eds.) AIED 2017. LNCS (LNAI), vol. 10331, pp. 506–509. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61425-0_51

    Chapter  Google Scholar 

  23. Graesser, A.C.: Conversations with AutoTutor help students learn. Int. J. Artif. Intell. Educ. 26, 124–132 (2016)

    Article  Google Scholar 

  24. Graesser, A.C., Forsyth, C., Lehman, B.: Two heads are better than one: learning from agents in conversational trialogues. Teach. Coll. Rec. 119, 1–20 (2017)

    Article  Google Scholar 

  25. Graesser, A.C., Person, N.K.: Question asking during tutoring. Am. Educ. Res. J. 31, 104–137 (1994)

    Article  Google Scholar 

  26. Graesser, A.C., Person, N.K., Harter, D.: The tutoring research group: teaching tactics and dialogue in AutoTutor. Int. J. Artif. Intell. Educ. 12, 257–279 (2001)

    Google Scholar 

  27. Greer, J., McCalla, G. (eds.): Student Models: The Key to Individualized Educational Systems. Springer, New York (1994)

    Google Scholar 

  28. Gunning, D.: Explainable Artificial Intelligence (XAI). Defense Advanced Research Projects Agency (DARPA) (2017)

    Google Scholar 

  29. Hao, J., Zapata-Rivera, D., Graesser, A.C., Cai, Z., Hu, X., Goldberg, B.: Towards an intelligent tutor for teamwork: responding to human sentiments. In: Sottilare, R., Graesser, A., Hu, X., Sinatra, A.M. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 6 - Team Tutoring, pp. 151–160. Army Research Laboratory, Orlando (2018). ISBN 978-0-9977257-4-2

    Google Scholar 

  30. Hsiao, I.H., Brusilovsky, P.: Guiding and motivating students through open social student modeling: lessons learned. Teach. Coll. Rec. 119(3), 1–42 (2017)

    Article  Google Scholar 

  31. Kay, J., Zapata-Rivera, D., Conati, C.: The GIFT of scrutable learner models: why and how. In: Sinatra, R.A.M., Graesser, A.C., Hu, X., Goldberg, B., Hampton, A.J. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 8 – Data Visualization, pp. 25–40. U.S. Army CCDC - Soldier Center, Orlando (2020)

    Google Scholar 

  32. Katz, S., Albacete, P., Chounta, I.-A., Jordan, P., McLaren, B.M., Zapata-Rivera, D.: Linking dialogue with student modelling to create an adaptive tutoring system for conceptual physics. Int. J. Artif. Intell. Educ. 31(3), 397–445 (2021). https://doi.org/10.1007/s40593-020-00226-y

    Article  Google Scholar 

  33. Kerly, A., Ellis, R., Bull, S.: CALMsystem: a conversational agent for learner modelling. In: Ellis, R., Allen, T., Petridis, M. (eds.) Applications and Innovations in Intelligent Systems XV, pp. 89–102. Springer, London (2008). https://doi.org/10.1007/978-1-84800-086-5_7

    Chapter  Google Scholar 

  34. Latané, B., Williams, K., Harkins, S.: Many hands make light the work: the causes and consequences of social loafing. J. Pers. Soc. Psychol. 37(6), 822–832 (1979)

    Article  Google Scholar 

  35. Long, Y., Aleven, V.: Enhancing learning outcomes through self-regulated learning support with an open learner model. User Model. User-Adap. Inter. 27(1), 55–88 (2017)

    Article  Google Scholar 

  36. Lopez, A.A., Guzman-Orth, D., Zapata-Rivera, D., Forsyth, C.M., Luce, C.: Examining the accuracy of a conversation-based assessment in interpreting English learners’ written responses (Research Report No. RR-21-03). Educational Testing Service (2021).https://doi.org/10.1002/ets2.12315

  37. Loukina, A., Madnani, N., Zechner, K.: The many dimensions of algorithmic fairness in educational applications. In: Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, Florence, Italy, pp. 1–10 (2019)

    Google Scholar 

  38. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. CoRR, abs/1908.09635 (2019). http://arxiv.org/abs/1908.09635

  39. Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A., Halpern, D.: Operation ARIES!: a serious game for teaching scientific inquiry. In: Ma, M., Oikonomou, A., Jain, L.C. (eds.) Serious Games and Edutainment Applications, pp. 169–195. Springer, London (2011). https://doi.org/10.1007/978-1-4471-2161-9_10

    Chapter  Google Scholar 

  40. Mislevy, R.J., Almond, R.G., Lukas, J.F.: A brief introduction to evidence‐centered design. ETS Res. Rep. Ser. 2003(1), i-29 (2003)

    Google Scholar 

  41. Mislevy, R.J., Riconscente, M.M.: Evidence-centered assessment design. In: Handbook of Test Development, pp. 75–104. Routledge (2011)

    Google Scholar 

  42. Mitrovic, A.: Fifteen years of constraint-based tutors: what we have achieved and where we are going. User Model. User-Adap. Inter. 22(1–2), 39–72 (2012). https://doi.org/10.1007/s11257-011-9105-9

    Article  Google Scholar 

  43. Mitrovic, A., Martin, B., Suraweera, P.: Intelligent tutors for all: constraint-based modeling methodology, systems and authoring. IEEE Intell. Syst. 22, 38–45 (2007)

    Article  Google Scholar 

  44. Pavlik, P.I., Brawner, K., Olney, A., Mitrovic, A.: A review of student models used in intelligent tutoring systems. In: Sollitare, R.Z., Graesser, A.C., Hu, X., Holden, H. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 1 - Learner Modeling, pp. 39–68. U.S. Army Research, Orlando (2013)

    Google Scholar 

  45. Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019)

    Article  Google Scholar 

  46. Rosé, C.P., McLaughlin, E.A., Liu, R., Koedinger, K.R.: Explanatory learner models: why machine learning (alone) is not the answer. Br. J. Edu. Technol. 50(6), 2943–2958 (2019)

    Article  Google Scholar 

  47. Schaldenbrand, P., et al.: Computer-supported human mentoring for personalized and equitable math learning. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds.) AIED 2021. LNCS (LNAI), vol. 12749, pp. 308–313. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-78270-2_55

    Chapter  Google Scholar 

  48. Shahrour, G., Bull, S.: Interaction preferences and learning in an inspectable learner model for language. In: Artificial Intelligence in Education, pp. 659–661. IOS Press (2009)

    Google Scholar 

  49. Shute, V.J.: SMART: student modeling approach for responsive tutoring. User Model. User-Adap. Inter. 5, 1–44 (1995). https://doi.org/10.1007/BF01101800

    Article  Google Scholar 

  50. Shute, V.J., Zapata-Rivera, D.: Adaptive educational systems. In: Durlach, P. (ed.) Adaptive Technologies for Training and Education, pp. 7–27. Cambridge University Press, New York (2012)

    Chapter  Google Scholar 

  51. Somyürek, S., Brusilovsky, P., Guerra, J.: Supporting knowledge monitoring ability: open learner modeling vs. open social learner modeling. Res. Pract. Technol. Enhanc. Learn. 15(1), 1–24 (2020). https://doi.org/10.1186/s41039-020-00137-5

    Article  Google Scholar 

  52. Sottilare, R.A., Brawner, K.W., Sinatra, A.M., Johnston, J.H.: An Updated Concept for a Generalized Intelligent Framework for Tutoring (GIFT). US Army Research Laboratory, Orlando (2017)

    Google Scholar 

  53. Sottilare, R., Barr, A., Robson, R., Hu, X., Graesser, A.: Exploring the opportunities and benefits of standards for adaptive instructional systems (AISs). In: Proceedings of the Adaptive Instructional Systems Workshop in the Industry Track of the 14th International Intelligent Tutoring Systems, pp. 49–53 (2018)

    Google Scholar 

  54. Tang, L.M., Kay, J.: Scaffolding for an OLM for long-term physical activity goals. In: Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization, pp. 147–156 (2018)

    Google Scholar 

  55. Thomson, D., Mitrovic, A.: Preliminary evaluation of a negotiable student model in a constraint-based ITS. Res. Pract. Technol. Enhanc. Learn. 5(01), 19–33 (2010)

    Article  Google Scholar 

  56. VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A., Rose, C.P.: When are tutorial dialogues more effective than reading? Cogn. Sci. 3, 3–62 (2007)

    Article  Google Scholar 

  57. Vincent-Lancrin, S., van der Vlies, R.: Trustworthy artificial intelligence (AI) in education: promises and challenges. OECD Education Working Papers, No. 218, OECD Publishing, Paris (2020). https://doi.org/10.1787/a6c90fa9-en

  58. Zapata-Rivera, D.: Open student modeling research and its connections to educational assessment. Int. J. Artif. Intell. Educ. 31(3), 380–396 (2020). https://doi.org/10.1007/s40593-020-00206-2

    Article  Google Scholar 

  59. Zapata-Rivera, D., Brawner, K., Jackson, G.T., Katz, I.R.: Reusing evidence in assessment and intelligent tutors. In: Sottilare, R., Graesser, A., Hu, X., Goodwin, G. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 5 - Assessment Methods, pp. 125–136. U.S. Army Research Laboratory, Orlando (2017). ISBN 978-0-9893923-9-6

    Google Scholar 

  60. Zapata-Rivera, J.D., Greer, J.: Interacting with Bayesian student models. Int. J. Artif. Intell. Educ. 14(2), 127–163 (2004)

    Google Scholar 

  61. Zapata-Rivera, D., Greer, J.E.: Exploring various guidance mechanisms to support interaction with inspectable learner models. In: Cerri, S.A., Gouardères, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 442–452. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-47987-2_47

    Chapter  Google Scholar 

  62. Zapata-Rivera, D., Jackson, T., Katz, I.R.: Authoring conversation-based assessment scenarios. In: Sottilare, R.A., Graesser, A.C., Hu, X., Brawner, K. (eds.) Design Recommendations for Intelligent Tutoring Systems Volume 3: Authoring Tools and Expert Modeling Techniques, pp. 169–178. U.S. Army Research Laboratory (2015)

    Google Scholar 

  63. Zapata-Rivera, D., Jackson, T., Liu, L., Bertling, M., Vezzu, M., Katz, I.: Assessing science inquiry skills using trialogues. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 625–626. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_84

    Chapter  Google Scholar 

  64. Zapata-Rivera, D., Hansen, E., Shute, V.J., Underwood, J.S., Bauer, M.: Evidence-based approach to interacting with open student models. Int. J. Artif. Intell. Educ. 17(3), 273–303 (2007)

    Google Scholar 

  65. Zapata-Rivera, D., Lehman, B., Sparks, J.R.: Learner modeling in the context of caring assessments. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2020. LNCS, vol. 12214, pp. 422–431. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50788-6_31

    Chapter  Google Scholar 

  66. Zapata-Rivera, D., Liu, L., Chen, L., Hao, J., von Davier, A.A.: Assessing science inquiry skills in an immersive, conversation-based scenario. In: Kei Daniel, B. (ed.) Big Data and Learning Analytics in Higher Education, pp. 237–252. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-06520-5_14

    Chapter  Google Scholar 

  67. Zapata-Rivera, D., Liu, L., Katz, I.R., Vezzu, M.: Exploring the use of game elements in the development of innovative assessment tasks for science. Cogn. Technol. 18(1), 43–50 (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Educational Testing Service, Princeton, NJ, 08541, USA

    Diego Zapata-Rivera & Carol M. Forsyth

Authors
  1. Diego Zapata-Rivera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carol M. Forsyth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diego Zapata-Rivera .

Editor information

Editors and Affiliations

  1. Soar Technology, Inc., Orlando, FL, USA

    Robert A. Sottilare

  2. Fraunhofer FKIE, Wachtberg, Germany

    Jessica Schwarz

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Rightsholder, under exclusive licence to Springer Nature Switzerland AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zapata-Rivera, D., Forsyth, C.M. (2022). Learner Modeling in Conversation-Based Assessment. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. HCII 2022. Lecture Notes in Computer Science, vol 13332. Springer, Cham. https://doi.org/10.1007/978-3-031-05887-5_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-05887-5_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-05886-8

  • Online ISBN: 978-3-031-05887-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation