Abstract
The complexity and high construction cost of case bases make it very difficult, if not impossible, to evaluate a CBR system, especially a knowledge-intensive CBR system, using statistical evaluation methods on many case bases. In this paper, we propose an evaluation strategy, which uses both many simple case bases and a few complex case bases to evaluate a CBR system, and show how this strategy may satisfy different evaluation goals. The identified evaluation goals are classified into two categories: domain-independent and domain-dependent. For the evaluation goals in the first category, we apply the statistical evaluation method using many simple case bases (for example, UCI data sets); for evaluation goals in the second category, we apply different, relatively weak, evaluation methods on a few complex domain-specific case bases. We apply this combined evaluation strategy to evaluate our knowledge-intensive conversational CBR method as a case study.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Simon, H.A.: Artificial Intelligence: an Empirical Science. Artif. Intell. 77, 95–127 (1995)
Cohen, P.R., Howe, A.E.: How Evaluation Guides AI Research. AI Mag. 9, 35–43 (1988)
Cohen, P., Howe, A.: Toward AI Research Methodology: Three Case Studies in Evaluation. Systems, Man and Cybernetics, IEEE Transactions 19, 634–646 (1989)
Cohen, P.R.: Empirical Methods for Artificial Intelligence. MIT Press, Cambridge (1995)
McSherry, D.: Interactive Case-Based Reasoning in Sequential Diagnosis. Applied Intelligence 14, 65–76 (2001)
Aamodt, A., Plaza, E.: Case-Based Reasoning: Foundational Issues, Methodological Variations, and System Approaches. AI Communications 7, 39–59 (1994)
Kolodner, J.: Case-Based Reasoning. Morgan Kaufmann, San Francisco (1993)
Santamaria, J.C., Ram, A.: Systematic Evaluation of Design Decisions in CBR Systems. In: Proceedings of the AAAI Case-Based Reasoning Workshop, Seattle, Washington, pp. 23–29 (1994)
Díaz-Agudo, B., González-Calero, P.A.: An Architecture for Knowledge Intensive CBR Systems. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 37–48. Springer, Heidelberg (2000)
Aha, D.W.: Generalizing from Case Studies: A Case Study. In: Sleeman, D.H., Edwards, P. (eds.) Proceedings of the Ninth International Workshop on Machine Learning, Aberdeen, Scotland, UK, pp. 1–10. Morgan Kaufmann, San Francisco (1992)
Aamodt, A.: Knowledge-Intensive Case-Based Reasoning in Creek. In: Funk, P., González-Calero, P.A. (eds.) 7th European Conference on Case-Based Reasoning, Madrid, Spain, pp. 1–15 (2004)
Sørmo, F., Cassens, J., Aamodt, A.: Explanation in Case-Based Reasoning-Perspectives and Goals. Artificial Intelligence Review 24, 109–143 (2005)
Newman, D., Hettich, S., Blake, C., Merz, C.: UCI Repository of Machine Learning Databases (1998), http://www.ics.uci.edu/~mlearn/mlrepository.html
Doyle, M., Cunningham, P.: A Dynamic Approach to Reducing Dialog in On-line Decision Guides. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 49–60. Springer, Heidelberg (2000)
Tong, X., Öztürk, P., Gu, M.: Dynamic Feature Weighting in Nearest Neighbor Classifiers. In: Proceedings of the 3rd International Conference on Machine Learning and Cybe (ICMLC 2004), Shanghai, China, vol. 4, pp. 2406–2411 (2004)
Yang, Q., Wu, J.: Enhancing the Effectiveness of Interactive Case-Based Reasoning with Clustering and Decision Forests. Applied Intelligence 12, 49–64 (2001)
Bogaerts, S., Leake, D.: Facilitating CBR for Incompletely-Described Cases: Distance Metrics for Partial Problem Descriptions. In: Funk, P., González Calero, P.A. (eds.) ECCBR 2004. LNCS, vol. 3155, pp. 62–76. Springer, Heidelberg (2004)
Bareiss, R.: The Experimental Evaluation of a Case-Based Learning Apprentice. In: The proceedings of the Case-Based Reasoning Workshop, Pensacola Beach, Florida, pp. 162–167 (1989)
McLaren, B.M.: Extensionally Defining Principles and Cases in Ethics: an AI Model. Artificial Intelligence Journal 150, 145–181 (2003)
Aha, D.W., Breslow, L., Muñoz-Avila, H.: Conversational Case-Based Reasoning. Applied Intelligence: The International Journal of Artificial Intelligence, Neural Networks, and Complex Problem-Solving Technologies 14, 9 (2001)
McSherry, D.: Minimizing Dialog Length in Interactive Case-Based Reasoning. In: International Joint Conferences on Artificial Intelligence, pp. 993–998 (2001)
Gupta, K.M., Aha, D.W., Sandhu, N.: Exploiting Taxonomic and Causal Relations in Conversational Case Retrieval. In: European Conference on Case- Based Reasoning, Aberdeen, Scotland, UK, pp. 133–147 (2002)
Gu, M., Aamodt, A.: A Knowledge-Intensive Method for Conversational CBR. In: Muñoz-Ávila, H., Ricci, F. (eds.) ICCBR 2005. LNCS, vol. 3620, pp. 296–311. Springer, Heidelberg (2005)
Gu, M., Tong, X., Aamodt, A.: Comparing Similarity Calculation Methods in Conversational CBR. In: Zhang, D., Khoshgoftaar, T.M., Shyu, M.L. (eds.) Proceedings of the 2005 IEEE International Conference on Information Reuse and Integration, Hilton, Las Vegas, Nevada, USA, pp. 427–432 (2005)
Gu, M., Aamodt, A.: Dialog Learning in Conversational CBR. In: Proceedings of the 19th International FLAIRS Conference, Melbourne Beach, Florida. AAAI Press (to appear, 2006)
Gu, M., Bø, K.: Component retrieval using knowledge-intensive conversational CBR. In: Ali, M., Dapoigny, R. (eds.) IEA/AIE 2006. LNCS, vol. 4031, pp. 554–563. Springer, Heidelberg (2006)
Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann Series in Data Management Systems. Morgan Kaufmann, San Francisco (2005)
Aha, D.W., McSherry, D., Yang, Q.: Advances in Conversational Case-Based Reasoning. Knowledge Engineering Review 20, 7 (2006)
Göker, M.H., Thompson, C.A.: Personalized Conversational Case-Based Recommendation. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 99–111. Springer, Heidelberg (2000)
Shimazu, H.: Expertclerk: A Conversational Case-Based Reasoning Tool for Developing Salesclerk Agents in E-Commerce Webshops. Artificial Intelligence Review 18, 223–244 (2002)
Ferguson, A., Bridge, D.G.: Partial Orders and Indifference Relations: Being Purposefully Vague in Case-Based Retrieval. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS (LNAI), vol. 1898, pp. 74–85. Springer, Heidelberg (2000)
Gu, M., Aamodt, A., Tong, X.: Component Retrieval Using Conversational Case-Based Reasoning. In: Shi, Z., He, Q. (eds.) Intelligent Information Processing II. IFIP International Federation for Information Processing, vol. 163. Springer Science + Business Media Inc. (2004) (2004)
McSherry, D.: Explanation in Recommender Systems. Artificial Intelligence Review 24, 179–197 (2005)
Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Explaining compound critiques. Artificial Intelligence Review 24, 199–220 (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gu, M., Aamodt, A. (2006). Evaluating CBR Systems Using Different Data Sources: A Case Study. In: Roth-Berghofer, T.R., Göker, M.H., Güvenir, H.A. (eds) Advances in Case-Based Reasoning. ECCBR 2006. Lecture Notes in Computer Science(), vol 4106. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11805816_11
Download citation
DOI: https://doi.org/10.1007/11805816_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-36843-4
Online ISBN: 978-3-540-36846-5
eBook Packages: Computer ScienceComputer Science (R0)