Neural network precision tuning. (English) Zbl 1537.68038
Parker, David (ed.) et al., Quantitative evaluation of systems. 16th international conference, QEST 2019, Glasgow, UK, September 10–12, 2019. Proceedings. Cham: Springer. Lect. Notes Comput. Sci. 11785, 129-143 (2019).
Summary: Minimizing the precision in which the neurons of a neural network compute is a desirable objective to limit the resources needed to execute it. This is specially important for neural networks used in embedded systems. Unfortunately, neural networks are very sensitive to the precision in which they have been trained and changing this precision generally degrades the quality of their answers. In this article, we introduce a new technique to tune the precision of neural networks in such a way that the optimized network computes in a lower precision without modifying the quality of the outputs of more than a percentage chosen by the user. From a technical point of view, we generate a system of linear constraints among integer variables that we can solve by linear programming. The solution to this system is the new precision of the neurons. We present experimental results obtained by using our method.
For the entire collection see [Zbl 1419.68016].
For the entire collection see [Zbl 1419.68016].
MSC:
| 68Q07 | Biologically inspired models of computation (DNA computing, membrane computing, etc.) |
| 65G99 | Error analysis and interval analysis |
| 65Y04 | Numerical algorithms for computer arithmetic, etc. |
| 68T05 | Learning and adaptive systems in artificial intelligence |
Keywords:
formal methods; floating-point arithmetic; static analysis; dynamic analysis; linear programming; numerical accuracyReferences:
| [1] | ANSI/IEEE: IEEE Standard for Binary Floating-point Arithmetic (2008) |
| [2] | Chiang, W., Baranowski, M., Briggs, I., Solovyev, A., Gopalakrishnan, G., Rakamaric, Z.: Rigorous floating-point mixed-precision tuning. In: POPL, pp. 300-315. ACM (2017) · Zbl 1380.68116 |
| [3] | Damouche, N., Martel, M.: Mixed precision tuning with salsa. In: Proceedings of the 8th International Joint Conference on Pervasive and Embedded Computing and Communication Systems, PECCS 2018, pp. 185-194. SciTePress (2018) |
| [4] | Damouche, N., Martel, M., Chapoutot, A.: Improving the numerical accuracy of programs by automatic transformation. STTT 19(4), 427-448 (2017). https://doi.org/10.1007/s10009-016-0435-0 |
| [5] | Darulova, E., Horn, E., Sharma, S.: Sound mixed-precision optimization with rewriting. In: Proceedings of the 9th ACM/IEEE International Conference on Cyber-Physical Systems, ICCPS, pp. 208-219. IEEE Computer Society/ACM (2018) |
| [6] | Darulova, E., Kuncak, V.: Sound compilation of reals. In: POPL 2014, pp. 235-248. ACM (2014) · Zbl 1284.68393 |
| [7] | Dutta, S., Jha, S., Sankaranarayanan, S., Tiwari, A.: Output range analysis for deep feedforward neural networks. In: Dutle, A., Muñoz, C., Narkawicz, A. (eds.) NFM 2018. LNCS, vol. 10811, pp. 121-138. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77935-5_9 |
| [8] | Gao, X., Bayliss, S., Constantinides, G.A.: SOAP: structural optimization of arithmetic expressions for high-level synthesis. In: International Conference on Field-Programmable Technology, pp. 112-119. IEEE (2013) |
| [9] | Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., Vechev, M.T.: AI2: safety and robustness certification of neural networks with abstract interpretation. In: 2018 IEEE Symposium on Security and Privacy, SP, pp. 3-18. IEEE (2018) |
| [10] | Goubault, E.: Static analysis by abstract interpretation of numerical programs and systems, and FLUCTUAT. In: Logozzo, F., Fähndrich, M. (eds.) SAS 2013. LNCS, vol. 7935, pp. 1-3. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38856-9_1 |
| [11] | Graphics, M.: Algorithmic C Datatypes, software version 2.6 edn. (2011). http://www.mentor.com/esl/catapult/algorithmic |
| [12] | Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97-117. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_5 · Zbl 1494.68167 |
| [13] | Lam, M.O., Hollingsworth, J.K., de Supinski, B.R., LeGendre, M.P.: Automatically adapting programs for mixed-precision floating-point computation. In: Supercomputing, ICS 2013, pp. 369-378. ACM (2013) |
| [14] | Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: 6th International Conference on Learning Representations, ICLR 2018 (2018). OpenReview.net |
| [15] | Martel, M.: Floating-point format inference in mixed-precision. In: Barrett, C., Davies, M., Kahsai, T. (eds.) NFM 2017. LNCS, vol. 10227, pp. 230-246. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57288-8_16 |
| [16] | Martel, M., Najahi, A., Revy, G.: Code size and accuracy-aware synthesis of fixed-point programs for matrix multiplication. In: Pervasive and Embedded Computing and Communication Systems, pp. 204-214. SciTePress (2014) |
| [17] | Muller, J.M., et al.: Handbook of Floating-Point Arithmetic. Birkhäuser, Boston (2010) · Zbl 1197.65001 |
| [18] | Nguyen, C., et al.: Floating-point precision tuning using blame analysis. In: International Conference on Software Engineering (ICSE). ACM (2016) |
| [19] | Panchekha, P., Sanchez-Stern, A., Wilcox, J.R., Tatlock, Z.: Automatically improving accuracy for floating point expressions. In: PLDI 2015, pp. 1-11. ACM (2015) |
| [20] | Rubio-Gonzalez, C., et al.: Precimonious: tuning assistant for floating-point precision. In: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 27:1-27:12. ACM (2013) |
| [21] | Singh, G., Gehr, T., Mirman, M., Püschel, M., Vechev, M.T.: Fast and effective robustness certification. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, pp. 10825-10836 (2018) |
| [22] | Singh, G., Gehr, T., Püschel, M., Vechev, M.T.: An abstract domain for certifying neural networks. PACMPL 3(POPL), 41:1-41:30 (2019) |
| [23] | Solovyev, A., Jacobsen, C., Rakamarić, Z., Gopalakrishnan, G.: Rigorous estimation of floating-point round-off errors with symbolic taylor expansions. In: Bjørner, N., de Boer, F. (eds.) FM 2015. LNCS, vol. 9109, pp. 532-550. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19249-9_33 · Zbl 1427.65064 |
| [24] | Wang, S., Pei, K., Whitehouse, J., Yang, J., Jana, S.: Formal security analysis of neural networks using symbolic intervals. In: 27th USENIX Security Symposium, USENIX Security 2018, pp. 1599-1614. USENIX Association (2018) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
