An integrative evaluation framework for intelligent decision support systems. (English) Zbl 1159.90425
Summary: Evaluation of the overall effectiveness of decision support systems (DSS) has been a research topic since the early 1980s. As artificial intelligence methods have been incorporated into systems to create intelligent decision support systems (IDSS), researchers have attempted to quantify the value of the additional capabilities. Despite the useful and relevant insights generated by previous research, existing evaluation methodologies offer only a fragmented and incomplete view of IDSS value and the contribution of its technical infrastructure. This paper proposes an integrative, multiple criteria IDSS evaluation framework through a model that links the decision value of an IDSS to both the outcome from, and process of, decision making and down to specific components of the IDSS. The proposed methodology provides the designer and developer specific guidance on the intelligent tools most useful for a specific user with a particular decision problem. The proposed framework is illustrated by evaluating an actual IDSS that coordinates management of urban infrastructures.
MSC:
| 90B50 | Management decision making, including multiple objectives |
| 90C29 | Multi-objective and goal programming |
| 68T37 | Reasoning under uncertainty in the context of artificial intelligence |
Keywords:
decision support systems; evaluation framework; multiple criteria analysis; artificial intelligenceReferences:
| [1] | Adelman, L., Evaluating Decision Support and Expert Systems (1992), John Wiley & Sons, Inc.: John Wiley & Sons, Inc. USA |
| [2] | Akoka, J., A framework for decision support systems evaluation, Information and Management, 4, 133-141 (1981) |
| [3] | Anderson, M. C.; Banker, R.; Ravindran, S., Value implication of iInvestments in information technology, Management Science, 52, 9, 1359-1376 (2006) |
| [4] | Brynjolfsson, E.; Hitt, L., Beyond the productivity paradox: Computers are the catalyst for bigger changes, Communications of the ACM, 41, 8, 49-55 (1998) |
| [5] | Banker, R.; Kauffman, R. J., The evolution of research on information systems: A fiftieth-year survey of literature in Management Science, Management Science, 50, 3, 281-298 (2004) |
| [6] | Buede, D., Structuring value attributes, Interfaces, 16, 2, 52-62 (1986) |
| [7] | Carr, N., It doesn’t matter, Harvard Business Review, 41-49 (2003) |
| [8] | Chandler, J., A multiple criteria approach for evaluating information systems, MIS Quarterly, 6, 1, 61-74 (1982) |
| [9] | Chandrasekaran, B., Generic tasks in knowledge-based reasoning: High-level building blocks for expert system design, IEEE Expert, 23-30 (1986) |
| [10] | Chandrasekaran, B., Design problem solving, AI Magazine, 59-71 (1990) |
| [11] | Chandrasekaran, B.; Johnson, T.; Smith, J., Task-structure analysis for knowledge modeling, Communications of the ACM, 35, 9, 124-137 (1992) |
| [12] | Checkland, P., Systems Thinking, Systems Practice (2000), Wiley: Wiley Chichester, UK |
| [13] | Cohen, P., A survey of the eighth national conference on artificial intelligence: Pulling together or pulling apart, AI Magazine, 11, 4, 16-41 (1991) |
| [14] | Devraj, S.; Kohli, R., The IT Payoff: Measuring the Business Value Information Technology Investments (2002), Financial Times Prentice Hall: Financial Times Prentice Hall Upper Saddle River, NJ |
| [15] | Devraj, S.; Kohli, R., Performance impacts of information technology: Is actual usage the missing link?, Management Science, 49, 3, 273-289 (2003) |
| [16] | Finkelstein, R., 2000. A method for evaluation the IQ of intelligent systems. In: Proceedings of the 2000 Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD, August, pp. 46-66.; Finkelstein, R., 2000. A method for evaluation the IQ of intelligent systems. In: Proceedings of the 2000 Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD, August, pp. 46-66. |
| [17] | Forgionne, G., An AHP model of DSS effectiveness, European Journal of Information Systems, 95-106 (1999) |
| [18] | Forgionne, G., Decision-making support systems effectiveness: The process to outcome link, Information Knowledge-Systems Management, 2, 169-188 (2000) |
| [19] | Forgionne, G.; Kohli, R., A multiple criteria assessment of decision technology system journal qualities, Information Management, 38, 421-435 (2001) |
| [20] | Forgionne, G., Mora, M., Gupta, J., Gelman, O., 2005. Decision-making support systems. In: Encyclopedia of Information Science and Technology, Idea Group, USA, pp. 759-765.; Forgionne, G., Mora, M., Gupta, J., Gelman, O., 2005. Decision-making support systems. In: Encyclopedia of Information Science and Technology, Idea Group, USA, pp. 759-765. |
| [21] | French, R., The turing test: The first fifty years, Trends in Cognitive Sciences, 4, 3, 115-121 (2000) |
| [22] | (Gupta, N.; Forgionne, G.; Mora, M., Intelligent Decision-making Support Systems Foundations, Applications and Challenges (2006), Springer-Verlag: Springer-Verlag Germany) |
| [23] | Harker, P., 1988. The Art and Science of Decision Making: The Analytic Hierarchy Process. Working Paper 88-06-03, Decision Science Department, The Wharton School, University of Pennsylvania, Philadelphia, PA.; Harker, P., 1988. The Art and Science of Decision Making: The Analytic Hierarchy Process. Working Paper 88-06-03, Decision Science Department, The Wharton School, University of Pennsylvania, Philadelphia, PA. |
| [24] | Hitt, L.; Brynjolfsson, E., Productivity, business profitability, and consumer surplus: Three different measures of information technology value, MIS Quarterly, 20, 2, 121-142 (1996) |
| [25] | Keen, P., Value Analysis: Justifying decision support systems, MIS Quarterly, 5, 1, 1-15 (1981) |
| [26] | Kohli, R.; Devaraj, S., Measuring information technology payoff: A meta-analysis of structural variables in firm-level empirical research, Information Systems Research, 14, 2, 127-145 (2003) |
| [27] | Kurikose, A., Successful decision making start with DSS evaluation, Data Management, 23, 2, 24-29 (1985) |
| [28] | Linger, H., Burstein, F., 1997. Intelligent decision support in the context of the modern organisation. In: Proceedings of the 4th conference of the international society for decision support systems - ISDSS’97, Lausanne, Switzerland, July 21-22.; Linger, H., Burstein, F., 1997. Intelligent decision support in the context of the modern organisation. In: Proceedings of the 4th conference of the international society for decision support systems - ISDSS’97, Lausanne, Switzerland, July 21-22. |
| [29] | Manyard, S.; Burstein, F.; Arnott, D., A multi-faceted decision support system evaluation approach, Journal of Decision Systems, 10, 3-4, 395-428 (2001) |
| [30] | Messina, E., Meystel, A., Reeker, L., 2001. Measuring performance and intelligence of intelligent systems. White Paper of the 2001 Performance Metrics for Intelligent Systems Workshop, Mexico, DF, September 4, pp. 1-13.; Messina, E., Meystel, A., Reeker, L., 2001. Measuring performance and intelligence of intelligent systems. White Paper of the 2001 Performance Metrics for Intelligent Systems Workshop, Mexico, DF, September 4, pp. 1-13. |
| [31] | Mora, M.; Forgionne, G.; Cervantes, F.; Garrido, L.; Gupta, J. N.D.; Gelman, O., Toward a comprehensive framework for the design and evaluation of intelligent decision-making support systems (i-DMSS), Journal of Decision Systems, 14, 3, 321-344 (2005) |
| [32] | Newell, A.; Simon, H. A., Human Problem Solving (1972), Prentice-Hall: Prentice-Hall Englewood Cliffs, NJ |
| [33] | Newell, A.; Simon, H. A., Computer science as empirical inquiry, Communications of the ACM, 19, 3, 113-126 (1976) |
| [34] | Newell, A., The knowledge level, AI Magazine, 1-33 (1981) |
| [35] | Phillips-Wren, G.; Hahn, E.; Forgionne, G., A multiple criteria framework for the evaluation of decision support systems, Omega, 32, 4, 323-332 (2004) |
| [36] | (Phillips-Wren, G.; Jain, L., Intelligent Decision Support Systems in Agent-Mediated Environments (2005), IOS Press: IOS Press Amsterdam) · Zbl 1083.68600 |
| [37] | Phillips-Wren, G.; Mora, M.; Forgionne, G.; Garrido, L.; Gupta, J. N.D., Multi-criteria evaluation of intelligent decision making support systems, (Gupta, J. N.D.; Forgionne, G.; Mora, M., Intelligent Decision-Making Support Systems (i-DMSS): Foundations, Applications and Challenges (2006), Springer), 3-24 |
| [38] | Phillips-Wren, G., Mora, M., Forgionne, G., Gupta, J.N.D., 2006b. Evaluation of decision-making support systems (DMSS): An integrated dmss and AI approach. In: Adam, F., Humphreys, P. (Eds.), Creativity and Innovation in Decision Making and Decision Support (Proceedings of CIDMDS 2006), London, UK, 29 June-1 July.; Phillips-Wren, G., Mora, M., Forgionne, G., Gupta, J.N.D., 2006b. Evaluation of decision-making support systems (DMSS): An integrated dmss and AI approach. In: Adam, F., Humphreys, P. (Eds.), Creativity and Innovation in Decision Making and Decision Support (Proceedings of CIDMDS 2006), London, UK, 29 June-1 July. |
| [39] | Quintero, A.; Konaré, D.; Pierre, S., Prototyping an intelligent decision support system for improving urban infrastructures management, European Journal of Operational Research, 162, 3, 654-672 (2005) · Zbl 1067.90085 |
| [40] | Rosenthal-Sabroux, C.; Zaraté, P., Artificial intelligence tools for decision support systems, European Journal of Operational Research, 103, 275-276 (1997) · Zbl 0942.90552 |
| [41] | Saaty, T. L., A scaling method for priorities in hierarchical structures, Journal of Mathematical Psychology, 234-281 (1977) · Zbl 0372.62084 |
| [42] | Saaty, T. L., How to make a decision: The analytic hierarchy process, Interfaces, 24, 6, 19-43 (1986) |
| [43] | Saaty, T.; Vargas, L., Decision Making in Economic, Political, Social and Technological Environments with The Analytic Hierarchy Process (1994), RWS Publications: RWS Publications Pittsburgh, PA |
| [44] | Santhanam, R.; Guimaraes, T., Assessing the quality of institutional DSS, European Journal of Information Systems, 4, 159-170 (1995) |
| [45] | Sharda, R.; Barr, S.; McDonnel, J., Decision support systems effectiveness: A review and an empirical test, Management Science, 34, 2, 139-159 (1988) |
| [46] | Simon, H., The New Science of Management Decision (1960), Harper and Row: Harper and Row New York, NY |
| [47] | Simon, H. A., Two heads are better than one: The collaboration between AI and OR, Interfaces, 17, 4, 8-15 (1987) |
| [48] | Simon, H. A.; Dantzing, G.; Hogart, R.; Plott, C.; Raiffa, H.; Schelling, T.; Shepsle, K.; Thaler, R.; Tversky, A.; Winter, S., Decision making and problem solving, Interfaces, 17, 5, 11-31 (1987) |
| [49] | Sprague, R., A framework for the development of decision support systems, MIS Quarterly, 4, 4, 1-26 (1980) |
| [50] | Sun, Y.; Kantor, P., Cross-evaluation: A new model for information system evaluation, Journal of the American Society for Information Science and Technology, 57, 5, 614-628 (2006) |
| [51] | Turing, A., Computing machinery and intelligence, Mind, 59, 433-460 (1950) |
| [52] | Wang, Y.; Forgionne, G., A decision-theoretic approach to the evaluation of information retrieval systems, Information Processing and Management, 24, 863-874 (2006) |
| [53] | Wind, Y.; Saaty, T. L., Marketing applications of the analytic hierarchy process, Management Sciences, 26, 7, 641-658 (1980) |
| [54] | Zhang, J.; Pu, P., Performance evaluation of consumer decision support systems, International Journal of E-Business Research, 2, 3, 38-45 (2006) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
