Abstract
NeuroEvolution is the application of Evolutionary Algorithms to the training of Artificial Neural Networks. Currently the vast majority of NeuroEvolutionary methods create homogeneous networks of user defined transfer functions. This is despite NeuroEvolution being capable of creating heterogeneous networks where each neuron’s transfer function is not chosen by the user, but selected or optimised during evolution. This paper demonstrates how NeuroEvolution can be used to select or optimise each neuron’s transfer function and empirically shows that doing so significantly aids training. This result is important as the majority of NeuroEvolutionary methods are capable of creating heterogeneous networks using the methods described.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Homogeneous ANNs are ANNs where each neuron’s TF is identical. Heterogeneous ANNs are ANNs where each neuron’s TF is not identical.
Sometimes refereed to as TWEANNs—Topology and Weight Evolving Artificial Neural Networks.
An open source cross platform CGP/CGPANN library has also been recently released for those interested in using CGP or CGPANN [38].
The logistic sigmoid function is often simply referred to as the sigmoid function in the ANN literature. In fact the term sigmoid function refers to any function which is ‘S’ shaped. The logistic sigmoid function is therefore a specific type of sigmoid function along with other functions including the Gompertz function.
The ‘1’ in ‘Cancer1’ refers to the permutation of the dataset; see [29].
\((1+4)\)-ES has been used previously for CGPANN and is also used here for CNE for simplicity. Additionally as a greedy strategy is used with no crossover maintaining genetic diversity with large populations becomes less significant.
Where probabilistic mutation changes each gene to a new valid value with a given probability.
In the case where training and testing results are given the training result is used for comparisons. This is because no steps to combat over training were made.
References
Angeline P, Saunders G, Pollack J (1994) An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans Neural Netw 5(1):54–65
Augusteijn MF, Harrington TP (2004) Evolving transfer functions for artificial neural networks. Neural Comput Appl 13(1):38–46
Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layer-wise training of deep networks. Adv Neural Fnfo Process Syst 19:153
Cantú-Paz E, Kamath C (2005) An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Trans Syst Man Cybern Part B Cybern 35(5):915–927
Chalup SK, Wiklendt L (2007) Variations of the two-spiral task. Connect Sci 19(2):183–199
Chebira A, Madani K (2003) Advances in soft computing, vol 19, chap. A Neural network based approach for sensors issued data fusion. Physica, Wien, pp 155–160
Cliff D, Harvey I, Husbands P (1992) Incremental evolution of neural network architectures for adaptive behaviour. In: Proceedings of the European symposium on artificial neural networks (ESANN’93), pp 39–44
Duch W, Jankowski N (1999) Survey of neural transfer functions. Neural Comput Surv 2(1):163–212
Duch W, Jankowski N (2001) Transfer functions: hidden possibilities for better neural networks. In: ESANN, pp 81–94
Duch W, Jankowski N, Maszczyk T (2012) Make it cheap: learning with o (nd) complexity. In: The 2012 international joint conference on neural networks (IJCNN). IEEE, pp 1–4
Floreano D, Dürr P, Mattiussi C (2008) Neuroevolution: from architectures to learning. Evolut Intell 1(1):47–62
Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the international conference on artificial intelligence and statistics (AISTATS10). Society for artificial intelligence and statistics
Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366
Khan MM, Ahmad MA, Khan MG, Miller JF (2013) Fast learning neural networks using Cartesian Genetic Programming. Neurocomputing 121:274–289
Koutník J, Gomez F, Schmidhuber J (2010) Evolving neural networks in compressed weight space. In: Proceedings of the conference on genetic and evolutionary computation (GECCO-10), pp 619–626
Larochelle H, Bengio Y, Louradour J, Lamblin P (2009) Exploring strategies for training deep neural networks. J Mach Learn Res 10:1–40
Liu Y, Yao X (1996) Evolutionary design of artificial neural networks with different nodes. In: Proceedings of IEEE international conference on evolutionary computation, 1996, pp 670–675. IEEE
Mangasarian OL, Setiono R, Wolberg WH (1990) Large-scale numerical optimization. In: Coleman TF, Li Y (eds) Pattern recognition via linear programming: theory and application to medical diagnosis. SIAM, Philadelphia, PA, pp 22–31
Manning T, Walsh P (2013) Improving the performance of CGPANN for breast cancer diagnosis using crossover and radial basis functions. In: Evolutionary computation, machine learning and data mining in bioinformatics. Springer, Berlin, pp 165–176
McCulloch W, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biol 5(4):115–133
Miller JF (2001) What bloat? Cartesian genetic programming on Boolean problems. In: 2001 Genetic and evolutionary computation conference late breaking papers, pp 295–302
Miller JF (2011) Cartesian genetic programming. Springer, Berlin
Miller JF, Smith S (2006) Redundancy and computational efficiency in Cartesian genetic programming. IEEE Trans Evolut Comput 10(2):167–174
Miller JF, Thomson P (2000) Cartesian genetic programming. In: Proceedings of the third European conference on genetic programming (EuroGP), vol 1820. Springer, Berlin, pp 121–132
Montana D, VanWyk E, Brinn M, Montana J, Milligan S (2009) Evolution of internal dynamics for neural network nodes. Evolut Intel 1(4):233–251
Park J, Sandberg IW (1991) Universal approximation using radial-basis-function networks. Neural Comput 3(2):246–257
Poli R (1996) Discovery of symbolic, neuro-symbolic and neural networks with parallel distributed genetic programming. Cognitive Science Research Papers University of Birmingham CSRP
Poli R (1999) Parallel distributed genetic programming. New ideas in optimization, advanced topics in computer science, pp 403–431
Prechelt L (1994) Proben1: a set of neural network benchmark problems and benchmarking rules. Fakultät für Informatik, Univ. Karlsruhe, Karlsruhe, Germany. Tech Rep 21:94
Richard K, B., John, M., Nico l N, S (1990) Evolving networks: using the genetic algorithm with connectionist learning. Technical report of cognitive computer science research group, Computer Science and Engineering Department (C-014), University of California at San Diego
Rumelhart DE, Hintont GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536
Schmidt M, Lipson H (2007) Comparison of tree and graph encodings as function of problem complexity. In: Proceedings of the 9th annual conference on genetic and evolutionary computation. ACM, New York, pp 1674–1679
Silva S, Costa E (2009) Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories. Genetic Progr Evolvable Mach 10(2):141–179
Smolensky P (1986) Parallel distributed processing: explorations in the microstructure of cognition, chap. Information processing in dynamical systems: foundations of harmony theory. MIT Press, Cambridge, pp 194–281
Stanley K, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evolut Comput 10(2):99–127
Teller A, Veloso M (1996) Symbolic visual learning. In: Ikeuchi K, Veloso M (eds) PADO: a new learning architecture for object recognition. Oxford University Press, pp 81–116
Thrun S, Bala J, Bloedorn E, Bratko I, Cestnik B, Cheng J, De Jong K, Dzeroski S, Fahlman S, Fisher D et al (1991) The monk’s problems a performance comparison of different learning algorithms. Technical report, Carnegie Mellon University
Turner AJ (2014) Cartesian genetic programming library. http://cgplibrary.co.uk/
Turner AJ, Miller JF (2013) Cartesian genetic programming encoded artificial neural networks: a comparison using three benchmarks. In: Proceedings of the conference on genetic and evolutionary computation (GECCO-13), pp 1005–1012
Turner AJ, Miller JF (2013) The importance of topology evolution in NeuroEvolution: a case study using Cartesian genetic programming of artificial neural networks. In: Research and development in intelligent systems XXX. Springer, Berlin, pp 213–226
Turner AJ, Miller JF (2014) Cartesian genetic programming: why no bloat? Genetic programming: 17th European conference, vol 8599., EuroGP-2014, LNCS. Springer, Berlin, pp 193–204
Turner AJ, Miller JF (2014) Recurrent Cartesian genetic programming. In: 13th International conference on parallel problem solving from nature (PPSN 2014), LNCS, vol 8672, pp 476–486
Vargha A, Delaney HD (2000) A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J Educ Behav Stat 25(2):101–132
Vassilev VK, Miller JF (2000) The advantages of landscape neutrality in digital circuit evolution. In: Proceedings on international conference on evolvable systems, LNCS, vol 1801. Springer, Berlin, pp 252–263
Weingaertner D, Tatai VK, Gudwin RR, Von Zuben FJ (2002) Hierarchical evolution of heterogeneous neural networks. In: Proceedings of the 2002 congress on evolutionary computation, 2002. CEC’02, vol 2. IEEE, pp 1775–1780
Wieland A (1991) Evolving neural network controllers for unstable systems. In: IJCNN-91-Seattle international joint conference onNeural networks, 1991, vol 2. IEEE, pp 667–673
Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evolut Comput 1(1):67–82
Yao X (1993) A review of evolutionary artificial neural networks. Int J Intell Syst 8(4):539–567
Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447
Yu T, Miller JF (2001) Neutrality and the evolvability of boolean function landscape. In: Genetic programming, LNCS. Springer, Berlin, pp 204–217
Acknowledgments
We would like to thank the reviewers for their feedback which helped make a much stronger paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Turner, A.J., Miller, J.F. NeuroEvolution: Evolving Heterogeneous Artificial Neural Networks. Evol. Intel. 7, 135–154 (2014). https://doi.org/10.1007/s12065-014-0115-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12065-014-0115-5