IntelliGrader: A Framework for Automatic Short Answer Grading, Inconsistency Check and Feedback in Educational Context - Conception, Implementation and Evaluation

Paradesi Sree Lakshmi, School of Computer Science and Applications, REVA University, Bengaluru, IndiaFollow
Jay B. Simha, REVA Academy for Corporate Excellence (RACE), REVA University, India
Rajeev Ranjan, School of Computer Science and Applications, REVA University, Bengaluru, India

Abstract

Automatic Short Answer Grading (ASAG), an escalating realm in natural language understanding, constitutes a focal point of research within the broader field of learning analytics. Over time, many ASAG solutions have been proposed to address the difficulties in teaching. However, no work addressed three crucial aspects of evaluation together, i.e., i) automatic evaluation of brief subjective/descriptive answers written in English, ii) identifying the evaluation inconsistency, and iii) provision of providing feedback about inconsistent evaluation to the evaluator. The current work proposes IntelliGrader, a comprehensive ASAG system that addresses the above-mentioned issues. Automated grading is accomplished through a model answer-based approach. The collaborative analysis of eight crucial features, incorporating statistical, word-word, keyword, lemmatized, term frequency-inverse document frequency, contextual, semantic, and summary resemblances amid model and student answers, are performed, utilizing state-of-the-art regressors. In contrast to other existing research, i) Unsupervised learning approaches were used to identify inconsistencies in evaluation, ii) Underwent rigorous validation on benchmark datasets ASAP, STITA, and a novel dataset IDEAS. Experimental results show the finest Root Mean Square Error of 0.09 on the STITA dataset and 0.19 on IDEAS for a specific question. IntelliGrader performs better than the systems presented in the literature. Experimental results regarding the inconsistency showed less inconsistency in model-predicted scores when compared with human evaluation, showing the model's accuracy. Finally, the identified inconsistency is provided as detailed feedback to the evaluator, which assists them in improving the evaluation process. We recommend using this as a tool to support evaluators, not to replace human judgment.

Recommended Citation

Sree Lakshmi, Paradesi; Simha, Jay B.; and Ranjan, Rajeev (2024) "IntelliGrader: A Framework for Automatic Short Answer Grading, Inconsistency Check and Feedback in Educational Context - Conception, Implementation and Evaluation," Karbala International Journal of Modern Science: Vol. 10 : Iss. 3 , Article 9.
Available at: https://doi.org/10.33640/2405-609X.3370