Optimizing hyperparameters in Hopfield neural networks using evolutionary search. (English) Zbl 07921753
Summary: The major problem facing users of Hopfield neural networks is the automatic choice of hyperparameters depending on the optimisation problem. This work introduces an automatic method to overcome this problem based on an original mathematical model minimizing the energy function. This methods ensures the feasibility of optimal solution obtained by decomposing the set of the feasible solutions. We illustrate the proposed model in the context of six well-known NP-hard problems: meeting scheduling problem, Kohonen network problem, portfolio selection problem, traveling salesman problem, task assignment problem, and max-stable problem. To show the effectiveness of our model, we use particle swarm and genetic algorithms to solve several instances of the last three problems. Numerical results show the good performance of the proposed approach compared to random tuning hyperparameters methods. Indeed, our approach permits an improvement of 49.75% for traveling salesman problem, 5.92% for task assignment problem, and 29.41% for max-stable problem.
MSC:
| 90Bxx | Operations research and management science |
Keywords:
neural networks; Hopfield neural networks; hyperparameters selection; genetic algorithms; particle swarm optimization; combinatorial optimization problemsSoftware:
PSPSOReferences:
| [1] | Zhao, J., A note on Hopfield neural network stability, IAENG Int. J. Comput. Sci., 42, 4, 332-336, 2015 |
| [2] | Hopfield, JJ; Tank, DW, “Neural” computation of decisions in optimization problems, Biol. Cybern., 52, 3, 141-152, 1985 · Zbl 0572.68041 |
| [3] | Hopfield, JJ, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci., 79, 8, 2554-2558, 1982 · Zbl 1369.92007 |
| [4] | Wen, UP; Lan, KM; Shih, HS, A review of Hopfield neural networks for solving mathematical programming problems, Eur. J. Oper. Res., 198, 3, 675-687, 2009 · Zbl 1176.90617 |
| [5] | Smith, KA, Neural networks for combinatorial optimization: a review of more than a decade of research, INFORMS J. Comput., 11, 1, 15-34, 1999 · Zbl 1034.90528 |
| [6] | Rbihou, S., Haddouch, K.: Comparative study between a neural network, approach metaheuristic and exact method for solving traveling salesman problem. In: 2021 Fifth International Conference on Intelligent Computing in Data Sciences (ICDS)). IEEE, pp. 1-5 (2021) |
| [7] | Yogeesha, CB; Pujeri, RV, A comparative study of geometric Hopfield network and ant colony algorithm to solve travelling salesperson problem, Int. J. Adv. Comput. Res., 4, 3, 843, 2014 |
| [8] | Fernández, A.; Gómez, S., Portfolio selection using neural networks, Comput. Oper. Res., 34, 4, 1177-1191, 2007 · Zbl 1102.91322 |
| [9] | Talaván, PM; Yáñez, J., Parameter setting of the Hopfield network applied to TSP, Neural Netw., 15, 3, 363-373, 2002 |
| [10] | Satake, T.; Morikawa, K.; Nakamura, N., Neural network approach for minimizing the makespan of the general job-shop, Int. J. Prod. Econ., 33, 1-3, 67-74, 1994 |
| [11] | Yang, L.; Shami, A., On hyperparameter optimization of machine learning algorithms: theory and practice, Neurocomputing, 415, 295-316, 2020 |
| [12] | El Moutaouakil, K., Touhafi, A.: A new recurrent neural network fuzzy mean square clustering method. In: 2020 5th International Conference on Cloud Computing and Artificial Intelligence: Technologies and Applications (CloudTech), Marrakesh, Morocco, pp. 1-5 (2020) |
| [13] | El Moutaouakil, K.; Yahyaouy, A.; Chellak, S., An optimized gradient dynamic-neuro-weighted-fuzzy clustering method: application in the nutrition field, Int. J. Fuzzy Syst., 24, 3731-3744, 2022 |
| [14] | El Moutaouakil, K., OPT-RNN-DBSVM: OPTimal recurrent neural network and density-based support vector machine, Mathematics, 11, 16, 3555, 2023 |
| [15] | El Moutaouakil, K., Multi-objective optimization for controlling the dynamics of the diabetic population, Mathematics, 11, 13, 2957, 2023 |
| [16] | El Moutaouakil, K.; Baizri, H.; Chellak, S., Optimal fuzzy deep daily nutrients requirements representation: application to optimal Morocco diet problem, Math. Model. Comput., 9, 607-615, 2022 |
| [17] | El Moutaouakil, K.; Roudani, M.; El Ouissari, A., Optimal entropy genetic fuzzy-C-means SMOTE (OEGFCM-SMOTE), Knowl. Based Syst., 262, 2023 |
| [18] | El Moutaouakil, K., Multi-objectives optimization and convolution fuzzy C-means: control of diabetic population dynamic, RAIRO-Oper. Res., 56, 5, 3245-3256, 2022 · Zbl 1507.90162 |
| [19] | El Moutaouakil, K.; Saliha, C.; Hicham, B.; Mouna, C., Intelligent local search optimization methods to optimal Morocco regime, IntechOpen, 2023 · doi:10.5772/intechopen.105600 |
| [20] | El Moutaouakil, K., et al.: Metaheuristics optimization algorithm to an optimal Moroccan diet. In: 2021 7th Annual International Conference on Network and Information Systems for Computers (ICNISC). IEEE (2021) |
| [21] | Lu, P.; Ye, L.; Zhao, Y.; Dai, B.; Pei, M.; Tang, Y., Review of meta-heuristic algorithms for wind power prediction: methodologies, applications and challenges, Appl. Energy, 301, 117446, 2021 |
| [22] | Wei, H.; Tang, XS; Liu, H., A genetic algorithm (GA)-based method for the combinatorial optimization in contour formation, Appl. Intell., 43, 1, 112-131, 2015 |
| [23] | Kramer, O.: Genetic algorithms. In: Genetic Algorithm Essentials. Springer, Cham, pp. 11-19 (2017) · Zbl 1390.68003 |
| [24] | Gad, AG, Particle swarm optimization algorithm and its applications: a systematic review, Arch. Comput. Methods Eng., 29, 5, 2531-2561, 2022 |
| [25] | Sun, J.; Lai, CH; Wu, XJ, Particle Swarm Optimisation: Classical and Quantum Perspectives, 2016, Boca Raton: CRC Press, Boca Raton |
| [26] | Rezaee Jordehi, A.; Jasni, J., Particle swarm optimisation for discrete optimisation problems: a review, Artif. Intell. Rev., 43, 243-258, 2015 |
| [27] | Venter, G.; Sobieszczanski-Sobieski, J., Particle swarm optimization, AIAA J., 41, 8, 1583-1589, 2003 |
| [28] | Poli, R.; Kennedy, J.; Blackwell, T., Particle swarm optimization: an overview, Swarm Intell., 1, 33-57, 2007 |
| [29] | Talaván, PM; Yáñez, J., The generalized quadratic knapsack problem. A neuronal network approach, Neural Netw., 19, 4, 416-428, 2006 · Zbl 1100.68098 |
| [30] | Joudar, NE; En-Naimani, Z.; Ettaouil, M., Using continuous Hopfield neural network for solving a new optimization architecture model of probabilistic self organizing map, Neurocomputing, 344, 82-91, 2019 |
| [31] | Joya, G.; Atencia, MA; Sandoval, F., Hopfield neural networks for optimization: study of the different dynamics, Neurocomputing, 43, 1-4, 219-237, 2002 · Zbl 1016.68076 |
| [32] | Wang, L., On the dynamics of discrete-time, continuous-state Hopfield neural networks, IEEE Trans. Circuits Syst. II Analog Digit. Signal Process., 45, 6, 747-749, 1998 |
| [33] | Kang, L.; Chen, RS; Xiong, N.; Chen, YC; Hu, YX; Chen, CM, Selecting hyper-parameters of Gaussian process regression based on non-inertial particle swarm optimization in Internet of Things, IEEE Access, 7, 59504-59513, 2019 |
| [34] | Tan, KC; Tang, H.; Ge, SS, On parameter settings of Hopfield networks applied to traveling salesman problems, IEEE Trans. Circuits Syst. I Regul. Pap., 52, 5, 994-1002, 2005 · Zbl 1374.82025 |
| [35] | Ettaouil, M.; Loqman, C.; Hami, Y.; Haddouch, K., Task assignment problem solved by continuous Hopfield network, Int. J. Comput. Sci. Issues (IJCSI), 9, 2, 206, 2012 |
| [36] | Rbihou, S., Joudar, N. E., En-Naimani, Z., Haddouch, K.: Using Crank-Nicolson scheme for continuous Hopfield network equilibrium. In: International Conference on Artificial Intelligence and Industrial Applications. Springer, Cham, pp. 201-210 (2023) |
| [37] | Haddouch, K.; Elmoutaoukil, K.; Ettaouil, M., Solving the weighted constraint satisfaction problems via the neural network approach, Int. J. Interact. Multimed. Artif. Intell., 4, 1, 56-60, 2016 |
| [38] | Talaván, PM; Yáñez, J., The graph coloring problem: a neuronal network approach, Eur. J. Oper. Res., 191, 1, 100-111, 2008 · Zbl 1147.68611 |
| [39] | El Alaoui, M.; El Moutaouakil, K.; Ettaouil, M., A multi-step method to calculate the equilibrium point of the continuous Hopfield networks: application to the max-stable problem, Int. J. Comput. Sci. Inf. Secur. (IJCSIS), 14, 6, 216-221, 2016 |
| [40] | Bouhouch, A., Chakir, L., El Qadi, A.: Scheduling meeting solved by neural network and min-conflict heuristic. In: 2016 4th IEEE International Colloquium on Information Science and Technology (CiSt). IEEE, pp. 773-778 (2016) |
| [41] | Lin, JS; Liu, M.; Huang, NF, The shortest path computation in MOSPF protocol using an annealed Hopfield neural network with a new cooling schedule, Inf. Sci., 129, 1-4, 17-30, 2000 · Zbl 0981.68744 |
| [42] | Ahn, CW; Ramakrishna, RS; Kang, CG; Choi, IC, Shortest path routing algorithm using Hopfield neural network, Electron. Lett., 37, 19, 1176-1178, 2001 |
| [43] | Ettaouil, M.; Loqman, C.; Haddouch, K.; Hami, Y., Maximal constraint satisfaction problems solved by continuous Hopfield networks, WSEAS Trans. Comput., 12, 2, 29-40, 2013 |
| [44] | Ettaouil, M.; Lazaar, M.; Elmoutaouakil, K.; Haddouch, K., A new algorithm for optimization of the Kohonen network architectures using the continuous Hopfield networks, WSEAS Trans. Comput., 12, 4, 155-163, 2013 |
| [45] | Ettaouil, M.; Loqman, C.; Haddouch, K., Job shop scheduling problem solved by the continuous Hopfield networks, J. Adv. Res. Comput. Sci., 2, 1, 31-47, 2010 |
| [46] | Comert, S.; Yazgan, H.; Turk, G., Hopfield neural network based on clustering algorithms for solving green vehicle routing problem, Int. J. Ind. Eng. Comput., 13, 4, 573-586, 2022 |
| [47] | Talaván, PM; Yáñez, J., A continuous Hopfield network equilibrium points algorithm, Comput. Oper. Res., 32, 8, 2179-2196, 2005 · Zbl 1068.90021 |
| [48] | Singh, P.; Kamthane, AR; Tanksale, AN, Metaheuristics for the distance constrained generalized covering traveling salesman problem, Opsearch, 58, 3, 575-609, 2021 · Zbl 1543.90311 |
| [49] | García, L.; Talaván, PM; Yáñez, J., Improving the Hopfield model performance when applied to the traveling salesman problem, Soft. Comput., 21, 14, 3891-3905, 2017 · Zbl 1381.68281 |
| [50] | Jain, E.; Dahiya, K.; Verma, V., A priority based unbalanced time minimization assignment problem, Opsearch, 57, 1, 13-45, 2020 · Zbl 07182108 |
| [51] | Haddouch, K., El Moutaouakil, K.: New starting point of the continuous hopfield network. In: International Conference on Big Data, Cloud and Applications. Springer, Cham, pp. 379-389 (2018) |
| [52] | El Alaoui, M.; Ettaouil, M., An adaptive hybrid approach: combining neural networks and simulated annealing to calculate the equilibrium point in max-stable problem, IAENG Int. J. Comput. Sci., 48, 4, 893-898, 2021 |
| [53] | Mahajan, RP, A quantum neural network approach for portfolio selection, Int. J. Comput. Appl., 29, 4, 47-54, 2011 |
| [54] | Senhaji, K., El Moutaouakil, K., Ettaouil, M.: Portfolio selection problem: new multicriteria approach for the mean-semivariance model. In: 2016 3rd International Conference on Logistics Operations Management (GOL). IEEE, pp. 1-6 (2016) |
| [55] | El Moutaouakil, K.; Ahourag, A.; Chakir, S.; Kabbaj, Z.; Chellack, S.; Cheggour, M.; Baizri, H., Hybrid firefly genetic algorithm and integral fuzzy quadratic programming to an optimal Moroccan diet, Math. Model. Comput., 10, 2, 338-350, 2023 |
| [56] | Han, S.S., May, G.S.: Optimization of neural network structure and learning parameters using genetic algorithms. In: Proceedings Eighth IEEE International Conference on Tools with Artificial Intelligence). IEEE, pp. 200-206 (1996) |
| [57] | Sheppard, C., Genetic Algorithms with Python, S. l, 2017, Austin: Smashwords Edition, Austin |
| [58] | Aszemi, NM; Dominic, PDD, Hyperparameter optimization in convolutional neural network using genetic algorithms, Int. J. Adv. Comput. Sci. Appl., 10, 6, 269-278, 2019 |
| [59] | Haidar, A.; Field, M.; Sykes, J.; Carolan, M.; Holloway, L., PSPSO: a package for parameters selection using particle swarm optimization, SoftwareX, 15, 100706, 2021 |
| [60] | Reinelt, G., TSPLIB-A traveling salesman problem library, ORSA J. Comput., 3, 4, 376-384, 1991 · Zbl 0775.90293 |
| [61] | Elloumi, S.: The task assignment problem, a library of instances. http://cedric.cnam.fr/oc/TAP/TAP.html (2004) |
| [62] | The Second DIMACS Implementation Challenge. ftp://dimacs.rutgers.edu/pub/challenge/graph/benchmarks/ clique/ |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
