Echo state network activation function based on bistable stochastic resonance. (English) Zbl 1498.62169
Summary: Stochastic resonance (SR) is a phenomenon wherein an information-carrying signal is enhanced via noise in a nonlinear system. This phenomenon enables living beings to adapt to noisy environments and use environmental noise to obtain useful information. A novel activation function of the echo state network (ESN) based on bistable SR is proposed in this study. Instead of using the tanh activation function – which is representative of the traditional threshold activation function – the bistable SR activation function is used to improve the noise adaptability of the ESN. Further, the proposed activation function provides a short-term memory (STM) ability that is not provided by the widely used threshold activation function, and thus, a physical reservoir can be designed using the proposed function. An STM task and a parity check task are used to verify the short-term memory and nonlinear ability of the bistable SR activation function. Further, two different prediction benchmarks prove that the proposed activation function can improve the noise adaptability of ESN. Finally, a visual recognition task is performed to demonstrate the potential of the SR activation function for physical reservoir computing.
MSC:
| 62M10 | Time series, auto-correlation, regression, etc. in statistics (GARCH) |
Keywords:
stochastic resonance; noisy adaptability; echo state network; short-term memory; activation function; reservoir computingReferences:
| [1] | Cao, W.; Wang, X.; Ming, Z.; Gao, J., A review on neural networks with random weights, Neurocomputing, 275, 278-287 (2018) |
| [2] | Anwar, SM; Majid, M.; Qayyum, A.; Awais, M.; Alnowami, M.; Khan, MK., Medical image analysis using convolutional neural networks: a review, J Med Syst, 42, 1-13 (2018) |
| [3] | Tealab, A., Time series forecasting using artificial neural networks methodologies: A systematic review, Future Comput Inf J, 3, 334-340 (2018) |
| [4] | Giles, CL; Kuhn, GM; Williams, RJ., Dynamic recurrent neural networks: Theory and applications, IEEE Trans Neural Netw, 5, 153-156 (1994) |
| [5] | Siegelmann, HT; Sontag, ED., Turing computability with neural nets, Appl Math Lett, 4, 77-80 (1991) · Zbl 0749.68032 |
| [6] | Jaeger H. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the“ echo state network” approach: GMD-Forschungszentrum Informationstechnik Bonn; 2002. |
| [7] | Jaeger H. The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report. 2001;148:13. |
| [8] | Maass, W.; Natschläger, T.; Markram, H., Real-time computing without stable states: A new framework for neural computation based on perturbations, Neural Comput, 14, 2531-2560 (2002) · Zbl 1057.68618 |
| [9] | Verstraeten, D.; Schrauwen, B.; d’Haene, M.; Stroobandt, D., An experimental unification of reservoir computing methods, Neural Netw, 20, 391-403 (2007) · Zbl 1132.68605 |
| [10] | Jaeger, H.; Haas, H., Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science, 304, 78-80 (2004) |
| [11] | Chouikhi, N.; Ammar, B.; Rokbani, N.; Alimi, AM., PSO-based analysis of Echo State Network parameters for time series forecasting, Appl Soft Comput, 55, 211-225 (2017) |
| [12] | Han, M.; Xu, M., Predicting multivariate time series using subspace echo state network, Neural Proces Lett, 41, 201-209 (2015) |
| [13] | Shi, Z.; Han, M., Support vector echo-state machine for chaotic time-series prediction, IEEE Trans Neural Netw, 18, 359-372 (2007) |
| [14] | Alexandre, LA; Embrechts, MJ; Linton, J., Benchmarking reservoir computing on time-independent classification tasks, (2009 International Joint Conference on Neural Networks (2009), IEEE), 89-93 |
| [15] | Embrechts MJ, Alexandre LA, Linton JD. Reservoir computing for static pattern recognition. ESANN2009. |
| [16] | Meftah, B.; Lézoray, O.; Benyettou, A., Novel approach using echo state networks for microscopic cellular image segmentation, Cognit Comput, 8, 237-245 (2016) |
| [17] | Xia, Y.; Jelfs, B.; Van Hulle, MM; Príncipe, JC; Mandic, DP., An augmented echo state network for nonlinear adaptive filtering of complex noncircular signals, IEEE Trans Neural Netw, 22, 74-83 (2010) |
| [18] | Bauduin, M.; Massar, S.; Horlin, F., Non-linear satellite channel equalization based on a low complexity echo state network, (2016 Annual Conference on Information Science and Systems (CISS): IEEE (2016)), 99-104 |
| [19] | Sun, L.; Jin, B.; Yang, H.; Tong, J.; Liu, C.; Xiong, H., Unsupervised EEG feature extraction based on echo state network, Inf Sci, 475, 1-17 (2019) |
| [20] | Schuman, CD; Potok, TE; Patton, RM; Birdwell, JD; Dean, ME; Rose, GS, arXiv preprint (2017) |
| [21] | Appeltant, L.; Soriano, MC; Van der Sande, G.; Danckaert, J.; Massar, S.; Dambre, J., Information processing using a single dynamical node as complex system, Nat Commun, 2, 1-6 (2011) |
| [22] | Watt, S.; Kostylev, M., Reservoir Computing Using a Spin-Wave Delay-Line Active-Ring Resonator Based on Yttrium-Iron-Garnet Film, Phys Rev Appl, 13, Article 034057 pp. (2020) |
| [23] | Duport, F.; Schneider, B.; Smerieri, A.; Haelterman, M.; Massar, S., All-optical reservoir computing, Opt Express, 20, 22783-22795 (2012) |
| [24] | Zhao, C.; Li, J.; Liu, L.; Koutha, LS; Liu, J.; Yi, Y., Novel spike based reservoir node design with high performance spike delay loop, (Proceedings of the 3rd ACM International Conference on Nanoscale Computing and Communication2016 (2016)), 1-5 |
| [25] | McCulloch, WS; Pitts, W., A logical calculus of the ideas immanent in nervous activity, Bull Math Biophys, 5, 115-133 (1943) · Zbl 0063.03860 |
| [26] | Guo, Y.; Wang, F.; Chen, B.; Xin, J., Robust echo state networks based on correntropy induced loss function, Neurocomputing, 267, 295-303 (2017) |
| [27] | Jaeger H. Short term memory in echo state networks: GMD-Forschungszentrum Informationstechnik; 2001. |
| [28] | Holzmann, G.; Hauser, H., Echo state networks with filter neurons and a delay&sum readout, Neural Netw, 23, 244-256 (2010) · Zbl 1396.68094 |
| [29] | Lukoševicius M, Popovici D, Jaeger H, Siewert U, Park R. Time warping invariant echo state networks. International University Bremen, Technical Report. 2006. |
| [30] | Jaeger, H.; Lukoševičius, M.; Popovici, D.; Siewert, U., Optimization and applications of echo state networks with leaky-integrator neurons, Neural Netw, 20, 335-352 (2007) · Zbl 1132.68554 |
| [31] | Hänggi, P., Stochastic resonance in biology how noise can enhance detection of weak signals and help improve biological information processing, ChemPhysChem, 3, 285-290 (2002) |
| [32] | Liao, Z.; Jin, S.; Kuwahata, A.; Sekino, M.; Tabata, H., Coherent detection stochastic resonance assisted biomagnetometer for measuring magnetocardiography at room temperature, Appl Phys Express (2021) |
| [33] | Liu, H.; Yang, J.; Sanjuán, MA, Improvement in the stochastic resonance in the Duffing oscillator subjected to a Poisson white noise excitation, Eur Phys J Plus, 136, 1-16 (2021) |
| [34] | Singh, M.; Verma, A.; Sharma, N., Optimized multistable stochastic resonance for the enhancement of pituitary microadenoma in MRI, IEEE J Biomed Health Inf, 22, 862-873 (2017) |
| [35] | Li, F.; Duan, F.; Chapeau-Blondeau, F.; Abbott, D., Signal estimation and filtering from quantized observations via adaptive stochastic resonance, Phys Rev E, 103, Article 052108 pp. (2021) |
| [36] | Neiman, A.; Sung, W., Memory effects on stochastic resonance, Phys Lett A, 223, 341-347 (1996) · Zbl 1037.82527 |
| [37] | Bartussek, R.; Hänggi, P.; Jung, P., Stochastic resonance in optical bistable systems, Phys Rev E, 49, 3930 (1994) |
| [38] | Vemuri, G.; Roy, R., Stochastic resonance in a bistable ring laser, Phys Rev A, 39, 4668 (1989) |
| [39] | Wellens, T.; Buchleitner, A., Bistability and stochastic resonance in an open quantum system, Chem Phys, 268, 131-149 (2001) |
| [40] | Harmer, GP; Davis, BR; Abbott, D., A review of stochastic resonance: circuits and measurement, IEEE Trans Instrum Meas, 51, 299-309 (2002) |
| [41] | Liao, Z.; Ma, K.; Tang, S.; Sarker, MS; Yamahara, H.; Tabata, H., Phase locking of ultra-low power consumption stochastic magnetic bits induced by colored noise, Chaos Solitons Fractals., 151, Article 111262 pp. (2021) |
| [42] | Liao, Z.; Ma, K.; Tang, S.; Sarker, MS; Yamahara, H.; Tabata, H., Influence of levy noise on subthreshold synchronization of spintronic stochastic neurons, Results Phys, 27, Article 104475 pp. (2021) |
| [43] | Ikemoto, S.; DallaLibera, F.; Hosoda, K., Noise-modulated neural networks as an application of stochastic resonance, Neurocomputing, 277, 29-37 (2018) |
| [44] | Ikemoto, S., Noise-modulated neural networks for selectively functionalizing sub-networks by exploiting stochastic resonance, Neurocomputing, 448, 1-9 (2021) |
| [45] | Liu, J.; Duan, F.; Chapeau-Blondeau, F.; Abbott, D., Distributed Bayesian vector estimation using noise-optimized low-resolution sensor observations, Digit Signal Process, Article 103224 pp. (2021) |
| [46] | Duan, L.; Duan, F.; Chapeau-Blondeau, F.; Abbott, D., Stochastic resonance in Hopfield neural networks for transmitting binary signals, Phys Lett A, 384, Article 126143 pp. (2020) · Zbl 1448.62140 |
| [47] | Liu, X.; Duan, L.; Duan, F.; Chapeau-Blondeau, F.; Abbott, D., Enhancing threshold neural network via suprathreshold stochastic resonance for pattern classification, Phys Lett A, 403, Article 127387 pp. (2021) · Zbl 07409898 |
| [48] | Longtin, A., Stochastic resonance in neuron models, J Stat Phys, 70, 309-327 (1993) · Zbl 1002.92503 |
| [49] | Wang, JS; Han, S.; Guo, QP, Echo state networks based predictive model of vinyl chloride monomer convention velocity optimized by artificial fish swarm algorithm, Soft Comput, 18, 457-468 (2014) |
| [50] | Deihimi, A.; Solat, A., Optimized echo state networks using a big bang-big crunch algorithm for distance protection of series-compensated transmission lines, Int J Electr Power Energy Syst, 54, 408-424 (2014) |
| [51] | Shi, G.; Liu, D.; Wei, Q., Energy consumption prediction of office buildings based on echo state networks, Neurocomputing, 216, 478-488 (2016) |
| [52] | Bala, A.; Ismail, I.; Ibrahim, R.; Sait, SM; Oliva, D., An Improved Grasshopper Optimization Algorithm Based Echo State Network for Predicting Faults in Airplane Engines, IEEE Access, 8, 159773-159789 (2020) |
| [53] | Schaetti, N.; Salomon, M.; Couturier, R., Echo state networks-based reservoir computing for mnist handwritten digits recognition, (2016 IEEE Intl Conference on Computational Science and Engineering (CSE) and IEEE Intl Conference on Embedded and Ubiquitous Computing (EUC) and 15th Intl Symposium on Distributed Computing and Applications for Business Engineering (DCABES): IEEE (2016)), 484-491 |
| [54] | Najibi, E.; Rostami, H. SCESN; SPESN, SWESN, Three recurrent neural echo state networks with clustered reservoirs for prediction of nonlinear and chaotic time series, Appl Intell, 43, 460-472 (2015) |
| [55] | Jaeger, H., Adaptive nonlinear system identification with echo state networks, Adv Neural Inf Process Syst, 609-616 (2003) |
| [56] | Tikhonov AN, Arsenin VY. Solutions of ill-posed problems. New York. 1977:1-30. · Zbl 0354.65028 |
| [57] | Siewert U, Wustlich W. Echo-state networks with band-pass neurons: Towards generic time-scale-independent reservoir structures. 2007. |
| [58] | McNamara, B.; Wiesenfeld, K., Theory of stochastic resonance, Phys Rev A, 39, 4854 (1989) |
| [59] | Collins, J.; Chow, CC; Imhoff, TT., Aperiodic stochastic resonance in excitable systems, Phys Rev E, 52, R3321 (1995) |
| [60] | Lu, S.; He, Q.; Wang, J., A review of stochastic resonance in rotating machine fault detection, Mech Syst Sig Process, 116, 230-260 (2019) |
| [61] | Cheng, W.; Xu, X.; Ding, Y.; Sun, K., Stochastic resonance in a single-well potential and its application in rolling bearing fault diagnosis, Rev Sci Instrum, 91, Article 064701 pp. (2020) |
| [62] | Rehan, M.; Hong, K-S, LMI-based robust adaptive synchronization of FitzHugh-Nagumo neurons with unknown parameters under uncertain external electrical stimulation, Phys Lett A, 375, 1666-1670 (2011) · Zbl 1242.92013 |
| [63] | Tang, J.; Jia, Y.; Yi, M.; Ma, J.; Li, J., Multiplicative-noise-induced coherence resonance via two different mechanisms in bistable neural models, Phys Rev E, 77, Article 061905 pp. (2008) |
| [64] | Yuenyong, S., Fast and effective tuning of echo state network reservoir parameters using evolutionary algorithms and template matrices, (2015 International Computer Science and Engineering Conference (ICSEC): IEEE (2015)), 1-6 |
| [65] | Dale, M.; Dewhirst, J.; O’Keefe, S.; Sebald, A.; Stepney, S.; Trefzer, MA, (The role of structure and complexity on reservoir computing quality. International Conference on Unconventional Computation and Natural Computation (2019), Springer), 52-64 · Zbl 1525.68052 |
| [66] | Bertschinger, N.; Natschläger, T., Real-time computation at the edge of chaos in recurrent neural networks, Neural Comput, 16, 1413-1436 (2004) · Zbl 1102.68530 |
| [67] | Furuta, T.; Fujii, K.; Nakajima, K.; Tsunegi, S.; Kubota, H.; Suzuki, Y., Macromagnetic simulation for reservoir computing utilizing spin dynamics in magnetic tunnel junctions, Phys Rev Appl, 10, Article 034063 pp. (2018) |
| [68] | Guo, Y.; Yang, X.; Wang, Y.; Wang, F.; Chen, B., Online Robust Echo State Broad Learning System, Neurocomputing (2021) |
| [69] | Phillips, JD., Deterministic chaos and historical geomorphology: a review and look forward, Geomorphology, 76, 109-121 (2006) |
| [70] | Xue, Y.; Yang, L.; Haykin, S., Decoupled echo state networks with lateral inhibition, Neural Netw, 20, 365-376 (2007) · Zbl 1132.68610 |
| [71] | Sheng, C.; Zhao, J.; Wang, W., Map-reduce framework-based non-iterative granular echo state network for prediction intervals construction, Neurocomputing, 222, 116-126 (2017) |
| [72] | Kusetogullari, H.; Yavariabdi, A.; Cheddad, A.; Grahn, H.; Hall, J., Ardis: a swedish historical handwritten digit dataset, Neural Comput Appl, 1-14 (2019) |
| [73] | Cohen, G.; Afshar, S.; Tapson, J.; Van Schaik, A. EMNIST, Extending MNIST to handwritten letters, (International Joint Conference on Neural Networks (IJCNN): IEEE; 2017 (2017)), 2921-2926 |
| [74] | ONeill M. Standard reference data program NIST, http://www.codeproject.com/kb/library/NeuralNetRecognition.aspx; 2006 [accessed 15 September 2021]. |
| [75] | Appeltant, L.; Van der Sande, G.; Danckaert, J.; Fischer, I., Constructing optimized binary masks for reservoir computing with delay systems, Sci Rep, 4, 1-5 (2014) |
| [76] | Liao, Z.; Wang, Z.; Yamahara, H.; Tabata, H., Low-power-consumption physical reservoir computing model based on overdamped bistable stochastic resonance system, Neurocomputing, 468, 137-147 (2021) |
| [77] | Héeroux, JB; Numata, H.; Kanazawa, N.; Nakano, D., Optoelectronic reservoir computing with VCSEL, (2018 International Joint Conference on Neural Networks (IJCNN): IEEE (2018)), 1-6 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
