Improved dendritic learning: activation function analysis. (English) Zbl 07885228
Summary: This study conducted a thorough evaluation of an improved dendritic learning (DL) framework, focusing specifically on its application in power load forecasting. The objective was to optimise the activation functions within the synapses and somas of DL to enhance their adaptability across diverse real-world scenarios. Through a rigorous analysis involving 25 experiments across five activation functions (sigmoid, hyperbolic tangent (tanh), rectified linear unit (ReLU), leaky ReLU, and exponential linear unit (ELU)), we elucidated their impacts on both regression and classification performance. Notably, the leaky ReLU-tanh combination demonstrated exceptional mean performance and effectiveness across 14 benchmark datasets from the University of California Irvine Machine Learning Repository, surpassing alternative combinations. When applied to power load forecasting, this combination outperformed other models, particularly transformer and LSTM. These findings underscore the significant advantages of the leaky ReLU-tanh-based DL framework in accurately predicting electricity load in smart grids, as evidenced by the lowest mean absolute error (39.27), root mean squared error (29.13), and mean absolute percentage error (2.84).
References:
| [1] | Ouyang, C.; Yu, F.; Hao, Y.; Tang, Y.; Jiang, Y., Build interval-valued time series forecasting model with interval cognitive map trained by principle of justifiable granularity, Inf. Sci., 652, Article 119756 pp., 2024 |
| [2] | Liu, J.; Zhao, T.; Cao, J.; Li, P., Interval type-2 fuzzy neural networks with asymmetric mfs based on the twice optimization algorithm for nonlinear system identification, Inf. Sci., 629, 123-143, 2023 · Zbl 1536.93501 |
| [3] | Liu, H.; Liu, M.; Li, D.; Zheng, W.; Yin, L.; Wang, R., Recent advances in pulse-coupled neural networks with applications in image processing, Electronics, 11, 2022 |
| [4] | Triantafyllopoulos, A.; Schuller, B. W.; İymen, G.; Sezgin, M.; He, X.; Yang, Z.; Tzirakis, P.; Liu, S.; Mertes, S.; André, E.; Fu, R.; Tao, J., An overview of affective speech synthesis and conversion in the deep learning era, Proc. IEEE, 111, 1355-1381, 2023 |
| [5] | Wu, C.; Li, X.; Guo, Y.; Wang, J.; Ren, Z.; Wang, M.; Yang, Z., Natural language processing for smart construction: current status and future directions, Autom. Constr., 134, Article 104059 pp., 2022 |
| [6] | Li, F.; Wang, C., Develop a multi-linear-trend fuzzy information granule based short-term time series forecasting model with k-medoids clustering, Inf. Sci., 629, 358-375, 2023 |
| [7] | Nunes, J. D.; Carvalho, M.; Carneiro, D.; Cardoso, J. S., Spiking neural networks: a survey, IEEE Access, 10, 60738-60764, 2022 |
| [8] | Bose, P.; Kasabov, N. K.; Bruzzone, L.; Hartono, R. N., Spiking neural networks for crop yield estimation based on spatiotemporal analysis of image time series, IEEE Trans. Geosci. Remote Sens., 54, 6563-6573, 2016 |
| [9] | Egrioglu, E.; Baş, E.; Chen, M.-Y., Recurrent dendritic neuron model artificial neural network for time series forecasting, Inf. Sci., 607, 572-584, 2022 |
| [10] | Taylor, W. R.; He, S.; Levick, W. R.; Vaney, D. I., Dendritic computation of direction selectivity by retinal ganglion cells, Science, 289, 2347-2350, 2000 |
| [11] | Gul, H. H.; Egrioglu, E.; Bas, E., Statistical learning algorithms for dendritic neuron model artificial neural network based on sine cosine algorithm, Inf. Sci., 629, 398-412, 2023 |
| [12] | Luo, X.; Wen, X.; Zhou, M.; Abusorrah, A.; Huang, L., Decision-tree-initialized dendritic neuron model for fast and accurate data classification, IEEE Trans. Neural Netw. Learn. Syst., 33, 4173-4183, 2022 |
| [13] | Clifford, C.; Ibbotson, M., Fundamental mechanisms of visual motion detection: models, cells and functions, Prog. Neurobiol., 68, 409-437, 2002 |
| [14] | Tamura, H.; Tang, Z.; Ishii, M., The neuron model considering difference of time of inputs and its movement direction selection function, IEEJ Trans. Electron. Inf. Syst., 122, 1094-1103, 2002 |
| [15] | Takeuchi, K., Calculate dendrites of the neuron to perceive a slope in the depth direction, 2010, University of Toyama, Master’s thesis |
| [16] | Gao, S.; Zhou, M.; Wang, Y.; Cheng, J.; Yachi, H.; Wang, J., Dendritic neuron model with effective learning algorithms for classification, approximation, and prediction, IEEE Trans. Neural Netw. Learn. Syst., 30, 601-614, 2019 |
| [17] | Zhang, Y.; Yang, Y.; Li, X.; Yuan, Z.; Todo, Y.; Yang, H., A dendritic neuron model optimized by meta-heuristics with a power-law-distributed population interaction network for financial time-series forecasting, Mathematics, 11, 2023 |
| [18] | Yu, Y.; Lei, Z.; Wang, Y.; Zhang, T.; Peng, C.; Gao, S., Improving dendritic neuron model with dynamic scale-free network-based differential evolution, IEEE/CAA J. Autom. Sin., 9, 99-110, 2022 |
| [19] | Bianchini, M.; Gori, M., Optimal learning in artificial neural networks: a review of theoretical results. Soft computing, Neurocomputing, 13, 313-346, 1996 |
| [20] | Ahmadianfar, I.; Bozorg-Haddad, O.; Chu, X., Gradient-based optimizer: a new metaheuristic optimization algorithm, Inf. Sci., 540, 131-159, 2020 · Zbl 1474.90517 |
| [21] | Oyelade, O. N.; Ezugwu, A. E.-S.; Mohamed, T. I.A.; Abualigah, L., Ebola optimization search algorithm: a new nature-inspired metaheuristic optimization algorithm, IEEE Access, 10, 16150-16177, 2022 |
| [22] | Gupta, S.; Abderazek, H.; Yıldız, B. S.; Yildiz, A. R.; Mirjalili, S.; Sait, S. M., Comparison of metaheuristic optimization algorithms for solving constrained mechanical design optimization problems, Expert Syst. Appl., 183, Article 115351 pp., 2021 |
| [23] | Hayyolalam, V.; Pourhaji Kazem, A. A., Black widow optimization algorithm: a novel meta-heuristic approach for solving engineering optimization problems, Eng. Appl. Artif. Intell., 87, Article 103249 pp., 2020 |
| [24] | Qian, X.; Tang, C.; Todo, Y.; Lin, Q.; Ji, J., Evolutionary dendritic neural model for classification problems, Complexity, 2020, 1-13, 2020 |
| [25] | Tang, Y.; Ji, J.; Zhu, Y.; Gao, S.; Tang, Z.; Todo, Y., A differential evolution-oriented pruning neural network model for bankruptcy prediction, Complexity, 2019, 2019 |
| [26] | Song, Z.; Tang, Y.; Ji, J.; Todo, Y., Evaluating a dendritic neuron model for wind speed forecasting, Knowl.-Based Syst., 201, Article 106052 pp., 2020 |
| [27] | Wang, Y.; Li, Y.; Song, Y.; Rong, X., The influence of the activation function in a convolution neural network model of facial expression recognition, Appl. Sci., 10, 1897, 2020 |
| [28] | Ji, J.; Tang, C.; Zhao, J.; Tang, Z.; Todo, Y., A survey on dendritic neuron model: mechanisms, algorithms and practical applications, Neurocomputing, 489, 390-406, 2022 |
| [29] | Yin, X.; Goudriaan, J.; Lantinga, E. A.; Vos, J.; Spiertz, H. J., A flexible sigmoid function of determinate growth, Ann. Bot., 91, 361-371, 2003 |
| [30] | Zheng, J.; Liu, C.; Huang, S.; He, Y., A novel adaptive dynamic ga combined with am to optimize ann for multi-output prediction: small samples enhanced in industrial processing, Inf. Sci., 644, Article 119285 pp., 2023 |
| [31] | Zhang, L.; Lim, C. P.; Yu, Y., Intelligent human action recognition using an ensemble model of evolving deep networks with swarm-based optimization, Knowl.-Based Syst., 220, Article 106918 pp., 2021 |
| [32] | Liu, Y.; Mao, T.; Zhou, D.-X., Approximation of functions from Korobov spaces by shallow neural networks, Inf. Sci., 670, Article 120573 pp., 2024 · Zbl 07854211 |
| [33] | Mo, Z.; Xiang, W., Maximum output discrepancy computation for convolutional neural network compression, Inf. Sci., Article 120367 pp., 2024 · Zbl 07845875 |
| [34] | Shen, H.; Wang, Z.; Zhang, J.; Zhang, M., L-net: a lightweight convolutional neural network for devices with low computing power, Inf. Sci., 660, Article 120131 pp., 2024 |
| [35] | Koch, C., Biophysics of Computation: Information Processing in Single Neurons, 2004, Oxford University Press |
| [36] | Gabbiani, F.; Krapp, H. G.; Koch, C.; Laurent, G., Multiplicative computation in a visual neuron sensitive to looming, Nature, 420, 320-324, 2002 |
| [37] | Kurowski, P.; Gawlak, M.; Szulczyk, P., Muscarinic receptor control of pyramidal neuron membrane potential in the medial prefrontal cortex (mpfc) in rats, Neuroscience, 303, 474-488, 2015 |
| [38] | Toaza, B.; Esztergár-Kiss, D., A review of metaheuristic algorithms for solving tsp-based scheduling optimization problems image 1, Appl. Soft Comput., 148, Article 110908 pp., 2023 |
| [39] | Wang, Y.; Sun, Z.; Zhang, H.; Zhou, Y.; Liu, S.; Xu, W., Dynamic survivability of two-layer networks: the role of interlayer coupling, Chaos Solitons Fractals, 180, Article 114571 pp., 2024 |
| [40] | Bache, K.; Lichman, M., Uci machine learning repository, 2013 |
| [41] | Box, G. E.P.; Jenkins, G. M., Time series analysis: forecasting and control, J. Time, 31, 2010 |
| [42] | Dey, R.; Salem, F. M., Gate-variants of gated recurrent unit (gru) neural networks, (2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), 2017), 1597-1600 |
| [43] | Yu, Y.; Si, X.; Hu, C.; Zhang, J., A review of recurrent neural networks: LSTM cells and network architectures, Neural Comput., 31, 1235-1270, 2019 · Zbl 1494.68236 |
| [44] | Siami-Namini, S.; Tavakoli, N.; Namin, A. S., The performance of lstm and bilstm in forecasting time series, (2019 IEEE International Conference on Big Data (Big Data), 2019), 3285-3292 |
| [45] | Han, K.; Wang, Y.; Chen, H.; Chen, X.; Guo, J.; Liu, Z.; Tang, Y.; Xiao, A.; Xu, C.; Xu, Y.; Yang, Z.; Zhang, Y.; Tao, D., A survey on vision transformer, IEEE Trans. Pattern Anal. Mach. Intell., 45, 87-110, 2023 |
| [46] | Zhou, H.; Zhang, S.; Peng, J.; Zhang, S.; Li, J.; Xiong, H.; Zhang, W., Informer: beyond efficient transformer for long sequence time-series forecasting, (Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021), 11106-11115 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
