FFNSL: Feed-forward neural-symbolic learner. (English) Zbl 07694494
Summary: Logic-based machine learning aims to learn general, interpretable knowledge in a data-efficient manner. However, labelled data must be specified in a structured logical form. To address this limitation, we propose a neural-symbolic learning framework, called Feed-Forward Neural-Symbolic Learner (FFNSL), that integrates a logic-based machine learning system capable of learning from noisy examples, with neural networks, in order to learn interpretable knowledge from labelled unstructured data. We demonstrate the generality of FFNSL on four neural-symbolic classification problems, where different pre-trained neural network models and logic-based machine learning systems are integrated to learn interpretable knowledge from sequences of images. We evaluate the robustness of our framework by using images subject to distributional shifts, for which the pre-trained neural networks may predict incorrectly and with high confidence. We analyse the impact that these shifts have on the accuracy of the learned knowledge and run-time performance, comparing FFNSL to tree-based and pure neural approaches. Our experimental results show that FFNSL outperforms the baselines by learning more accurate and interpretable knowledge with fewer examples.
MSC:
| 68T05 | Learning and adaptive systems in artificial intelligence |
Keywords:
neural-symbolic learning; inductive logic programming; logic-based machine learning; distributional shiftReferences:
| [1] | Abdar, M.; Pourpanah, F.; Hussain, S.; Rezazadegan, D.; Liu, L.; Ghavamzadeh, M.; Fieguth, P.; Cao, X.; Khosravi, A.; Acharya, UR; Makarenkov, V.; Nahavandi, S., A review of uncertainty quantification in deep learning: Techniques, applications and challenges, Information Fusion, 76, 243-297 (2021) · doi:10.1016/j.inffus.2021.05.008 |
| [2] | Amodei, D., Olah, C., Steinhardt, J., Christiano, P. F., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety . http://arxiv.org/abs/1606.06565 |
| [3] | Bellodi, E., & Riguzzi, F. (2013). Structure learning of probabilistic logic programs by searching the clause space. Theory and Practice of Logic Programming, 15. |
| [4] | Besold, T., Garcez, A., Bader, S., Bowman, H., Domingos, P., Hitzler, P., Kühnberger, K.-U., Lamb, L., Lowd, D., Lima, P., de Penning, L., Pinkas, G., Poon, H., & Zaverucha, G. (2017). Neural-symbolic learning and reasoning: A survey and interpretation. http://arxiv.org/abs/1711.03902 |
| [5] | Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight uncertainty in neural network. In International conference on machine learning (pp. 1613-1622). |
| [6] | Cohen, W. W. (2016). Tensorlog: A differentiable deductive database. http://arxiv.org/abs/1605.06523 |
| [7] | Dai, W.-Z., & Muggleton, S. (2021). Abductive knowledge induction from raw data. In: Z.-H. Zhou (Ed.), Proceedings of the thirtieth international joint conference on artificial intelligence, IJCAI-21 (pp. 1845-1851). doi:10.24963/ijcai.2021/254. |
| [8] | Dai, W.-Z., Xu, Q., Yu, Y., & Zhou, Z.-H. (2019). Bridging machine learning and logical reasoning by abductive learning. Advances in Neural Information Processing Systems, 32. |
| [9] | Dantsin, E.; Eiter, T.; Gottlob, G.; Voronkov, A., Complexity and expressive power of logic programming, ACM Computing Surveys (CSUR), 33, 3, 374-425 (2001) · doi:10.1145/502807.502810 |
| [10] | De Raedt, L., Dries, A., Thon, I., Van den Broeck, G., & Verbeke, M. (2015). Inducing probabilistic relational rules from probabilistic examples. In Proceedings of 24th international joint conference on artificial intelligence (IJCAI) (Vol. 2015-January, pp. 1835-1842). IJCAI-INT JOINT CONF ARTIF INTELL, United States. |
| [11] | De Raedt, L., Kimmig, A., Toivonen, H. (2007). Problog: A probabilistic prolog and its application in link discovery. In IJCAI (Vol. 7, pp. 2462-2467). |
| [12] | Donadello, I., Serafini, L., & d’Avila Garcez, A. S. (2017). Logic tensor networks for semantic image interpretation. In Proceedings of the twenty-sixth international joint conference on artificial intelligence (pp. 1596-1602). IJCAI, California, USA. |
| [13] | Evans, R.; Grefenstette, E., Learning explanatory rules from noisy data, Journal of Artificial Intelligence Research, 61, 1-64 (2018) · Zbl 1426.68235 · doi:10.1613/jair.5714 |
| [14] | Flaminio, T.; Marchioni, E., T-norm based logics with an independent involutive negation, Fuzzy Sets and Systems, 157, 3125-3144 (2006) · Zbl 1114.03015 · doi:10.1016/j.fss.2006.06.016 |
| [15] | Garcez, A.d., & Lamb, L.C. (2020). Neurosymbolic AI: the 3rd wave. http://arxiv.org/abs/2012.05876 |
| [16] | Gelfond, M.; Kahl, Y., Knowledge representation, reasoning, and the design of intelligent agents: The answer-set programming approach (2014), Cambridge: Cambridge University Press, Cambridge · doi:10.1017/CBO9781139342124 |
| [17] | Gilpin, L., Bau, D., Yuan, B., Bajwa, A., Specter, M., & Kagal, L. (2018). Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) (pp. 80-89). |
| [18] | Hughes, D. P., & Salathé, M. (2015). An open access repository of images on plant health to enable the development of mobile disease diagnostics through machine learning and crowdsourcing. CoRR abs/1511.08060. |
| [19] | Hüllermeier, E.; Waegeman, W., Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods, Machine Learning, 110, 3, 457-506 (2021) · Zbl 07432810 · doi:10.1007/s10994-021-05946-3 |
| [20] | Karp, R. M. (1972). Reducibility among combinatorial problems. In Complexity of Computer Computations (pp. 85-103). · Zbl 1467.68065 |
| [21] | Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020). Concept bottleneck models. In International conference on machine learning (pp. 5338-5348). |
| [22] | Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016). Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1675-1684). |
| [23] | Law, M. (2018). Inductive learning of answer set programs. PhD thesis, Imperial College London. |
| [24] | Law, M., Russo, A., & Broda, K. (2019). Logic-based learning of answer set programs. In Reasoning Web. Explainable Artificial Intelligence - 15th International Summer School 2019, Bolzano, Italy, September 20-24, 2019, Tutorial Lectures (pp. 196-231). · Zbl 07817423 |
| [25] | Law, M., Russo, A., Bertino, E., Broda, K., & Lobo, J. (2020). Fastlas: scalable inductive logic programming incorporating domain-specific optimisation criteria. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, pp. 2877-2885). |
| [26] | Law, M.; Russo, A.; Broda, K., Inductive learning of answer set programs from noisy examples, Advances in Cognitive Systems, 7, 57-76 (2018) |
| [27] | Law, M.; Russo, A.; Broda, K., The complexity and generality of learning answer set programs, Artificial Intelligence, 259, 110-146 (2018) · Zbl 1445.68206 · doi:10.1016/j.artint.2018.03.005 |
| [28] | LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P., Gradient-based learning applied to document recognition, Proceedings of the IEEE, 86, 11, 2278-2324 (1998) · doi:10.1109/5.726791 |
| [29] | López-Cifuentes, A.; Escudero-Viñolo, M.; Bescós, J., Álvaro García-Martín: Semantic-aware scene recognition, Pattern Recognition, 102 (2020) · doi:10.1016/j.patcog.2020.107256 |
| [30] | Mackay, DJC, Probable networks and plausible predictions—A review of practical bayesian methods for supervised neural networks, Network: Computation in Neural Systems, 6, 3, 469-505 (1995) · Zbl 0834.68098 · doi:10.1088/0954-898X_6_3_011 |
| [31] | Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T., & De Raedt, L. (2018). Deepproblog: Neural probabilistic logic programming. In Advances in neural information processing systems (pp. 3749-3759). · Zbl 1520.68022 |
| [32] | Metcalfe, G., Olivetti, N., & Gabbay, D. M. (2008). Proof Theory for Fuzzy Logics (Vol. 36). Springer. · Zbl 1168.03002 |
| [33] | Minervini, P., Bosnjak, M., Rocktäschel, T., Riedel, S., & Grefenstette, E. (2020). Differentiable reasoning on large knowledge bases and natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 (pp. 5182-5190). |
| [34] | Minervini, P., Riedel, S., Stenetorp, P., Grefenstette, E., & Rocktäschel, T. (2020). Learning reasoning strategies in end-to-end differentiable proving. In Proceedings of the 37th international conference on machine learning, ICML 2020, 13-18 July 2020, Virtual event (pp. 6938-6949). |
| [35] | Molnar, C. (2019). Interpretable Machine Learning, Online. |
| [36] | Muggleton, S., Inductive logic programming, New Generation Computing, 8, 4, 295-318 (1991) · Zbl 0712.68022 · doi:10.1007/BF03037089 |
| [37] | Muggleton, SH; Lin, D.; Tamaddoni-Nezhad, A., Meta-interpretive learning of higher-order dyadic datalog: Predicate invention revisited, Machine Learning, 100, 1, 49-73 (2015) · Zbl 1346.68119 · doi:10.1007/s10994-014-5471-y |
| [38] | Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J.V., Lakshminarayanan, B., & Snoek, J. (2019). Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In 33rd conference on neural information processing systems (NeurIPS) (pp. 13969-13980). |
| [39] | Pearce, T., Brintrup, A., & Zhu, J. (2021). Understanding Softmax confidence and uncertainty. http://arxiv.org/abs/2106.04972 |
| [40] | Quattoni, A., & Torralba, A. (2009). Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition (pp. 413-420). IEEE. |
| [41] | Rasmussen, C. E.(2003). Gaussian processes in machine learning. In Summer School on Machine Learning (pp. 63-71) · Zbl 1120.68436 |
| [42] | Richardson, M.; Domingos, P., Markov logic networks, Machine Learning, 62, 1-2, 107-136 (2006) · Zbl 1470.68221 · doi:10.1007/s10994-006-5833-1 |
| [43] | Riegel, R., Gray, A., Luus, F., Khan, N., Makondo, N., Akhalwaya, I. Y., Qian, H., Fagin, R., Barahona, F., Sharma, U., et al. (2020). Logical neural networks . http://arxiv.org/abs/2006.13155 |
| [44] | Rocktäschel, T., & Riedel, S. (2017). End-to-end differentiable proving. In Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, December 4-9, 2017, Long Beach, CA, USA (pp. 3788-3800). |
| [45] | Sen, P., de Carvalho, B. W., Riegel, R., & Gray, A. (2021). Neuro-symbolic inductive logic programming with logical neural networks. http://arxiv.org/abs/2112.03324 |
| [46] | Sensoy, M., Kaplan, L., & Kandemir, M. (2018). Evidential deep learning to quantify classification uncertainty. In Advances in neural information processing systems (pp. 3179-3189). |
| [47] | Sensoy, M., Kaplan, L., Cerutti, F., & Saleki, M. (2020). Uncertainty-aware deep classifiers using generative models. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, pp. 5620-5627). |
| [48] | Serafini, L., & d’Avila Garcez, A. S. (2016). Logic tensor networks: Deep learning and logical reasoning from data and knowledge. http://arxiv.org/abs/1606.04422 |
| [49] | Stahl, I. (1993). Predicate invention in ilp-an overview. In European conference on machine learning (pp. 311-322). |
| [50] | Tsamoura, E., Hospedales, T., & Michael, L. (2021). Neural-symbolic integration: A compositional perspective. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, pp. 5051-5060). |
| [51] | Tuckey, D., Broda, K., & Russo, A. (2020). Towards structure learning under the credal semantics. In: C. Dodaro, G. A. Elder, W. Faber, J. Fandinno, M. Gebser, M. Hecher, E. LeBlanc, M. Morak, & J. Zangari (Eds.), International Conference on Logic Programming 2020 Workshop Proceedings Co-located with 36th International Conference on Logic Programming (ICLP 2020), Rende, Italy, September 18-19, 2020. CEUR Workshop Proceedings, vol. 2678. CEUR-WS.org, Italy. |
| [52] | Xu, J., Zhang, Z., Friedman, T., Liang, Y., & Broeck, G. (2018). A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning (pp. 5502-5511). PMLR. |
| [53] | Yang, Z., Ishay, A., & Lee, J. (2020). Neurasp: Embracing neural networks into answer set programming. In C. Bessiere (Ed.) Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI-20 (pp. 1755-1762). |
| [54] | Yang, F., Yang, Z., & Cohen, W. W. (2017). Differentiable learning of logical rules for knowledge base reasoning. In Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, December 4-9, 2017, Long Beach, CA, USA (pp. 2319-2328). |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
