Graph contrastive learning with min-max mutual information. (English) Zbl 07845882
Summary: Graph contrastive learning has achieved rapid development in learning representations from graph-structured data, which aims to maximize the mutual information between two representations learned from different augmented views of a graph. However, maximizing the mutual information between different views without any constraints may cause encoders to capture information irrelevant to downstream tasks, limiting the efficiency of graph contrastive learning methods. To tackle these issues, we propose a Graph Contrastive Learning method with Min-max mutual Information (GCLMI). Specifically, we conduct theoretical analysis to present our learning objective. It designs a min-max principle to constrain the mutual information among multiple views, including between a graph and each of its augmented views, as well as between different augmented views. Based on the learning objective, we further construct two augmented views by separating the feature and topology information of a graph to preserve different semantic information from the graph. Subsequently, we maximize the mutual information between each augmented view and the graph while minimizing the mutual information between two augmented views, to learn informative and diverse representations. Extensive experiments are conducted on a variety of graph datasets, and experimental results show that GCLMI achieves better or competitive performance compared with state-of-the-art methods.
Keywords:
graph neural networks; graph data augmentations; graph contrastive learning; graph representation learning; min-max mutual informationReferences:
| [1] | Shervashidze, N.; Schweitzer, P.; Van Leeuwen, E. J.; Mehlhorn, K.; Borgwardt, K. M., Weisfeiler-Lehman graph kernels, J. Mach. Learn. Res., 12, 1-23, 2011 |
| [2] | Borgwardt, K. M.; Ong, C. S.; Schönauer, S.; Vishwanathan, S.; Smola, A. J.; Kriegel, H.-P., Protein function prediction via graph kernels, Bioinformatics, 21, i47-i56, 2005 |
| [3] | Kriege, N. M.; Mutzel, P., Subgraph matching kernels for attributed graphs, (Proceedings of the International Conference on Machine Learning (ICML), 2012), 1-8 |
| [4] | Yanardag, P.; Vishwanathan, S., Deep graph kernels, (Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015), 1365-1374 |
| [5] | Wan, H.; Luan, X.; Stojanovic, V.; Liu, F., Self-triggered finite-time control for discrete-time Markov jump systems, Inf. Sci., 634, 101-121, 2023 · Zbl 1536.93777 |
| [6] | Zhou, C.; Tao, H.; Chen, Y.; Stojanovic, V.; Paszke, W., Robust point-to-point iterative learning control for constrained systems: A minimum energy approach, Int. J. Robust Nonlinear Control, 32, 10139-10161, 2022 · Zbl 1529.93023 |
| [7] | Shervashidze, N.; Vishwanathan, S.; Petri, T.; Mehlhorn, K.; Borgwardt, K., Efficient graphlet kernels for large graph comparison, (Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2009), 488-495 |
| [8] | Kipf, T. N.; iling, M., Semi-supervised classification with graph convolutional networks, (Proceedings of the International Conference on Learning Representations (ICLR), 2017), 1-14 |
| [9] | Veličković, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; Bengio, Y., Graph attention networks, (Proceedings of the International Conference on Learning Representations (ICLR), 2018), 1-12 |
| [10] | Hamilton, W. L.; Ying, R.; Leskovec, J., Inductive representation learning on large graphs, (Proceedings of the International Conference and Workshop on Neural Information Processing Systems (NeurIPS), 2017), 1025-1035 |
| [11] | Xu, K.; Hu, W.; Leskovec, J.; Jegelka, S., How powerful are graph neural networks?, (Proceedings of the International Conference on Learning Representations (ICLR), 2019), 1-17 |
| [12] | Yao, X.; Zhu, H.; Gu, M., Brain-inspired GCN: Modularity-based Siamese simple graph convolutional networks, Inf. Sci., 657, Article 119971 pp., 2024 |
| [13] | Xiao, Y.; Yuan, Q.; Jiang, K.; He, J.; Wang, Y.; Zhang, L., From degrade to upgrade: Learning a self-supervised degradation guided adaptive network for blind remote sensing image super-resolution, Inf. Fusion, 96, 297-311, 2023 |
| [14] | Rani, V.; Nabi, S. T.; Kumar, M.; Mittal, A.; Kumar, K., Self-supervised learning: A succinct review, Arch. Comput. Methods Eng., 30, 2761-2775, 2023 |
| [15] | Tao, H.; Qiu, J.; Chen, Y.; Stojanovic, V.; Cheng, L., Unsupervised cross-domain rolling bearing fault diagnosis based on time-frequency information fusion, J. Franklin Inst., 360, 1454-1477, 2023 · Zbl 1506.93064 |
| [16] | Yu, J.; Yin, H.; Xia, X.; Chen, T.; Li, J.; Huang, Z., Self-supervised learning for recommender systems: A survey, IEEE Trans. Knowl. Data Eng., 1-20, 2023 |
| [17] | Grover, A.; Leskovec, J., node2vec: Scalable feature learning for networks, (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016), 855-864 |
| [18] | Adhikari, B.; Zhang, Y.; Ramakrishnan, N.; Prakash, B. A., Sub2vec: Feature learning for subgraphs, (Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), 2018), 170-182 |
| [19] | Narayanan, A.; Chandramohan, M.; Venkatesan, R.; Chen, L.; Liu, Y.; Jaiswal, S., graph2vec: Learning distributed representations of graphs, 2017, arXiv preprint |
| [20] | Peng, Z.; Huang, W.; Luo, M.; Zheng, Q.; Rong, Y.; Xu, T.; Huang, J., Graph representation learning via graphical mutual information maximization, (Proceedings of the ACM Web Conference (WWW), 2020), 259-270 |
| [21] | Hassani, K.; Khasahmadi, A. H., Contrastive multi-view representation learning on graphs, (Proceedings of the International Conference on Machine Learning (ICML), 2020), 4116-4126 |
| [22] | Velickovic, P.; Fedus, W.; Hamilton, W. L.; Liò, P.; Bengio, Y.; Hjelm, R. D., Deep graph infomax, (Proceedings of the International Conference on Learning Representations (ICLR), 2019), 1-17 |
| [23] | Sun, F.-Y.; Hoffmann, J.; Verma, V.; Tang, J., Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization, (Proceedings of the International Conference on Learning Representations (ICLR), 2020), 1-16 |
| [24] | Zhang, Q.; Zhao, Z.; Zhou, H.; Li, X.; Li, C., Self-supervised contrastive learning on heterogeneous graphs with mutual constraints of structure and feature, Inf. Sci., 640, Article 119026 pp., 2023 |
| [25] | You, Y.; Chen, T.; Sui, Y.; Chen, T.; Wang, Z.; Shen, Y., Graph contrastive learning with augmentations, (Proceedings of the International Conference and Workshop on Neural Information Processing Systems (NeurIPS), vol. 33, 2020), 5812-5823 |
| [26] | You, Y.; Chen, T.; Shen, Y.; Wang, Z., Graph contrastive learning automated, (Proceedings of the International Conference on Machine Learning (ICML), 2021), 12121-12132 |
| [27] | Sun, M.; Xing, J.; Wang, H.; Chen, B.; Zhou, J., Mocl: Data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph, (Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2021), 3585-3594 |
| [28] | Lee, N.; Lee, J.; Park, C., Self-supervised graph representation learning via positive mining, Inf. Sci., 611, 476-493, 2022 |
| [29] | Xia, J.; Wu, L.; Chen, J.; Hu, B.; Li, S. Z., Simgrace: A simple framework for graph contrastive learning without data augmentation, (Proceedings of the ACM Web Conference (WWW), 2022), 1070-1079 |
| [30] | Hassani, K.; Khasahmadi, A. H., Learning graph augmentations to learn graph representations, 2022, arXiv preprint |
| [31] | Wang, Y.; Wang, W.; Liang, Y.; Cai, Y.; Hooi, B., Mixup for node and graph classification, (Proceedings of the ACM Web Conference (WWW), 2021), 3663-3674 |
| [32] | Feng, W.; Zhang, J.; Dong, Y.; Han, Y.; Luan, H.; Xu, Q.; Yang, Q.; Kharlamov, E.; Tang, J., Graph random neural networks for semi-supervised learning on graphs, (Proceedings of the International Conference and Workshop on Neural Information Processing Systems (NeurIPS), vol. 33, 2020), 22092-22103 |
| [33] | Jin, M.; Zheng, Y.; Li, Y.-F.; Gong, C.; Zhou, C.; Pan, S., Multi-scale contrastive Siamese networks for self-supervised graph representation learning, (Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2021), 1-7 |
| [34] | Qiu, J.; Chen, Q.; Dong, Y.; Zhang, J.; Yang, H.; Ding, M.; Wang, K.; Tang, J., GCC: Graph contrastive coding for graph neural network pre-training, (Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020), 1150-1160 |
| [35] | Jiao, Y.; Xiong, Y.; Zhang, J.; Zhang, Y.; Zhang, T.; Zhu, Y., Sub-graph contrast for scalable self-supervised graph representation learning, (Proceedings of IEEE International Conference on Data Mining (ICDM), 2020), 222-231 |
| [36] | Li, Z.; Liu, J.; Peng, Y.; Wen, C.-F., A novel method to information fusion in multi-source incomplete interval-valued data via conditional information entropy: Application to mutual information entropy based attribute reduction, Inf. Sci., Article 120011 pp., 2023 |
| [37] | Cheng, P.; Min, M. R.; Shen, D.; Malon, C.; Zhang, Y.; Li, Y.; Carin, L., Improving disentangled text representation learning with information-theoretic guidance, (Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020), 7530-7541 |
| [38] | Zhao, J.; Mao, W., Disentangled text representation learning with information-theoretic perspective for adversarial robustness, 2022, arXiv preprint |
| [39] | Pimentel, T.; Valvoda, J.; Maudslay, R. H.; Zmigrod, R.; Williams, A.; Cotterell, R., Information-theoretic probing for linguistic structure, 2020, arXiv preprint |
| [40] | Oord, A.v.d.; Li, Y.; Vinyals, O., Representation learning with contrastive predictive coding, 2018, arXiv preprint |
| [41] | Xie, Y.; Xu, Z.; Wang, Z.; Ji, S., Self-supervised learning of graph neural networks: A unified review, IEEE Trans. Pattern Anal. Mach. Intell., 45, 2412-2429, 2023 |
| [42] | Poole, B.; Ozair, S.; Van Den Oord, A.; Alemi, A.; Tucker, G., On variational bounds of mutual information, (Proceedings of the International Conference on Machine Learning (ICML), 2019), 5171-5180 |
| [43] | Han, S.; Lin, C.; Shen, C.; Wang, Q.; Guan, X., Interpreting adversarial examples in deep learning: A review, ACM Comput. Surv., 55, 1-38, 2023 |
| [44] | Xie, X.; Pu, Y.-F.; Wang, J., A fractional gradient descent algorithm robust to the initial weights of multilayer perceptron, Neural Netw., 158, 154-170, 2023 · Zbl 07752005 |
| [45] | Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L., Pytorch: An imperative style, high-performance deep learning library, (Proceedings of the International Conference and Workshop on Neural Information Processing Systems (NeurIPS), vol. 32, 2019), 1-12 |
| [46] | Fey, M.; Lenssen, J. E., Fast graph representation learning with PyTorch Geometric, (ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019), 1-9 |
| [47] | Morris, C.; Kriege, N. M.; Bause, F.; Kersting, K.; Mutzel, P.; Neumann, M., Tudataset: A collection of benchmark datasets for learning with graphs, (ICML Workshop on Graph Representation Learning and Beyond, 2020), 1-11 |
| [48] | Kipf, T. N.; Welling, M., Variational graph auto-encoders, 2016, arXiv preprint |
| [49] | Hu, W.; Liu, B.; Gomes, J.; Zitnik, M.; Liang, P.; Pande, V.; Leskovec, J., Strategies for pre-training graph neural networks, (Proceedings of the International Conference on Learning Representations (ICLR), 2020), 1-22 |
| [50] | Li, H.; Zhang, Z.; Wang, X.; Zhu, W., Disentangled graph contrastive learning with independence promotion, IEEE Trans. Knowl. Data Eng., 35, 7856-7869, 2023 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
