Enhancing threshold neural network via suprathreshold stochastic resonance for pattern classification. (English) Zbl 07409898
Summary: Hard-threshold nonlinearities are of significant interest for neural-network information processing due to their simplicity and low-cost implementation. They however lack an important differentiability property. Here, hard-threshold nonlinearities receiving assistance from added noise are pooled into a large-scale summing array to approximate a neuron with a noise-smoothed activation function. Differentiability that facilitates gradient-based learning is restored for such neurons, which are assembled into a feed-forward neural network. The added noise components used to smooth the hard-threshold responses have adjustable parameters that are adaptively optimized during the learning process. The converged non-zero optimal noise levels establish a beneficial role for added noise in operation of the threshold neural network. In the retrieval phase the threshold neural network operating with non-zero optimal added noise, is tested for data classification and for handwritten digit recognition, which achieves state-of-the-art performance of existing backpropagation-trained analog neural networks, while requiring only simpler two-state binary neurons.
Keywords:
threshold neural network; suprathreshold stochastic resonance; backpropagation; pattern classification; noise-smoothed activation functionSoftware:
PRMLTReferences:
| [1] | Hanson, S., A stochastic version of the delta rule, Physica D, 42, 1-3, 265-272 (1990) |
| [2] | Toms, D., Training binary node feedforward neural networks by back propagation of error, Electron. Lett., 26, 21, 1745-1746 (1990) |
| [3] | Bartlett, P.; Downs, T., Using random weights to train multilayer networks of hard-limiting units, IEEE Trans. Neural Netw., 3, 2, 202-210 (1992) |
| [4] | Corwin, E.; Logar, A.; Oldham, W., An iterative method for training multilayer networks with threshold functions, IEEE Trans. Neural Netw., 5, 3, 507-508 (1994) |
| [5] | Wilson, E.; Rock, S., Gradient-based parameter optimization for systems containing discrete-valued functions, Int. J. Robust Nonlinear Control, 12, 9, 1009-1028 (2002) · Zbl 1006.93518 |
| [6] | Huang, G.; Zhu, Q.; Mao, K.; Siew, C.; Saratchandran, P.; Sundararajan, N., Can threshold networks be trained directly?, IEEE Trans. Circuits Syst. II, Express Briefs, 53, 3, 187-191 (2006) |
| [7] | Frazier-Logue, N.; Hanson, S., The stochastic delta rule: faster and more accurate deep learning through adaptive weight noise, Neural Comput., 32, 5, 1018-1032 (2020) · Zbl 1468.68181 |
| [8] | Ikemoto, S.; Dallalibera, F.; Hosoda, K., Noise-modulated neural networks as an application of stochastic resonance, Neurocomputing, 277, 2, 29-37 (2018) |
| [9] | Plappert, M.; Houthooft, R.; Dhariwal, P.; Sidor, S.; Chen, R.; Chen, X.; Asfour, T.; Abbeel, P.; Andrychowicz, M., Parameter space noise for exploration, (6-th International Conference on Learning Representations (ICLR). 6-th International Conference on Learning Representations (ICLR), Vancouver, BC, Canada (2018)) |
| [10] | Stocks, N. G., Suprathreshold stochastic resonance in multilevel threshold systems, Phys. Rev. Lett., 84, 11, 2310-2313 (2000) |
| [11] | Stocks, N., Information transmission in parallel threshold arrays: suprathreshold stochastic resonance, Phys. Rev. E, 63, 4, Article 041114 pp. (2001) |
| [12] | McDonnell, M. D.; Stocks, N. G.; Pearce, C. E.M.; Abbott, D., Stochastic Resonance: From Suprathreshold Stochastic Resonance to Stochastic Signal Quantization (2008), Cambridge University Press: Cambridge University Press New York · Zbl 1157.94003 |
| [13] | Zhu, J.; Kong, C.; Liu, X., Subthreshold and suprathreshold vibrational resonance in the Fitzhugh-Nagumo neuron model, Phys. Rev. E, 94, 3, Article 032208 pp. (2016) |
| [14] | Rousseau, D.; Chapeau-Blondeau, F., Suprathreshold stochastic resonance and signal-to-noise ratio improvement in arrays of comparators, Phys. Lett. A, 321, 5-6, 280-290 (2004) · Zbl 1118.81431 |
| [15] | Gershman, A.; Böhme, J., A pseudo-noise approach to direction finding, Signal Process., 71, 1, 1-13 (1998) · Zbl 1010.94509 |
| [16] | Uhlich, S., Bayes risk reduction of estimators using artificial observation noise, IEEE Trans. Signal Process., 63, 20, 5535-5545 (2015) · Zbl 1394.94611 |
| [17] | Duan, F.; Pan, Y.; Chapeau-Blondeau, F.; Abbott, D., Noise benefits in combined nonlinear Bayesian estimators, IEEE Trans. Signal Process., 67, 17, 4611-4623 (2019) · Zbl 07123384 |
| [18] | Hari, V.; Anand, G.; Premkumar, A.; Madhukumar, A., Design and performance analysis of a signal detector based on suprathreshold stochastic resonance, Signal Process., 92, 7, 1745-1757 (2012) |
| [19] | Liu, J.; Hu, B.; Wang, Y., Optimum adaptive array stochastic resonance in noisy grayscale image restoration, Phys. Lett. A, 383, 13, 1457-1465 (2019) · Zbl 1470.94015 |
| [20] | Liu, R.; Kang, Y., Stochastic resonance in underdamped periodic potential systems with alpha stable Lévy noise, Phys. Lett. A, 382, 25, 1656-1664 (2018) |
| [21] | Duan, F.; Chapeau-Blondeau, F.; Abbott, D., Weak signal detection: condition for noise induced enhancement, Digit. Signal Process., 23, 5, 1585-1591 (2013) |
| [22] | Guo, D.; Perc, M.; Liu, T.; Yao, D., Functional importance of noise in neuronal information processing, Europhys. Lett., 124, 5, Article 50001 pp. (2018) |
| [23] | Li, W.; Hirota, K.; Dai, Y.; Jia, Z., An improved fully convolutional network based on post-processing with global variance equalization and noise-aware training for speech enhancement, J. Adv. Comput. Intell. Intell. Inform., 25, 1, 130-137 (2021) |
| [24] | Lu, S.; Qian, G.; He, Q.; Liu, F.; Liu, Y.; Wang, Q., In situ motor fault diagnosis using enhanced convolutional neural network in an embedded system, IEEE Sens. J., 20, 15, 8287-8296 (2020) |
| [25] | Kosko, B.; Audhkhasi, K.; Osoba, O., Noise can speed backpropagation learning and deep bidirectional pretraining, Neural Netw., 129, 359-384 (2020) |
| [26] | Fu, Y.; Kang, Y.; Chen, G., Stochastic resonance based visual perception using spiking neural networks, Front. Comput. Neurosci., 14, 24 (2020) |
| [27] | Bishop, C., Pattern Recognition and Machine Learning (2006), Springer: Springer New York · Zbl 1107.68072 |
| [28] | Haykin, S., Neural Networks and Learning Machines (2009), Prentice Hall: Prentice Hall New York |
| [29] | LeCun, Y.; Bottou, L., Gradient-based learning applied to document recognition, Proc. IEEE, 86, 11, 2278-2324 (1998) |
| [30] | Neelakantan, A.; Vilnis, L.; Sutskever, V.; Kaiser, L.; Kurach, K.; Martens, J., Adding gradient noise improves learning for very deep networks, (4-th International Conference on Learning Representation. 4-th International Conference on Learning Representation, San Juan, Puerto Rico (2016)), 1-11 |
| [31] | Blanchard, S.; Rousseau, D.; Chapeau-Blondeau, F., Noise enhancement of signal transduction by parallel arrays of nonlinear neurons with threshold and saturation, Neurocomputing, 71, 333-341 (2007) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
