Optimal 1-Wasserstein distance for WGANs. (English) Zbl 07898706
Summary: The mathematical forces at work behind Generative Adversarial Networks raise challenging theoretical issues. Motivated by the important question of characterizing the geometrical properties of the generated distributions, we provide a thorough analysis of Wasserstein GANs (WGANs) in both the finite sample and asymptotic regimes. We study the specific case where the latent space is univariate and derive results valid regardless of the dimension of the output space. We show in particular that for a fixed sample size, the optimal WGANs are closely linked with connected paths minimizing the sum of the squared Euclidean distances between the sample points. We also highlight the fact that WGANs are able to approach (for the 1-Wasserstein distance) the target distribution as the sample size tends to infinity, at a given convergence rate and provided the family of generative Lipschitz functions grows appropriately. We derive in passing new results on optimal transport theory in the semi-discrete setting.
MSC:
| 60F99 | Limit theorems in probability theory |
| 68T05 | Learning and adaptive systems in artificial intelligence |
| 49Q22 | Optimal transportation |
Keywords:
optimal distribution; optimal transport theory; rate of convergence; shortest path; Wasserstein distance; Wasserstein generative adversarial networks (WGANs)References:
| [1] | Arjovsky, M., Chintala, S. and Bottou, L. (2017). Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning (D. Precup and Y.W. Teh, eds.) 70 214-223. PMLR. |
| [2] | Aurenhammer, F., Hoffmann, F. and Aronov, B. (1998). Minkowski-type theorems and least-squares clustering. Algorithmica 20 61-76. 10.1007/PL00009187 MathSciNet: MR1483422 · Zbl 0895.68135 |
| [3] | Biau, G., Sangnier, M. and Tanielian, U. (2021). Some theoretical insights into Wasserstein GANs. J. Mach. Learn. Res. 22 Paper No. 119, 45. MathSciNet: MR4279770 · Zbl 1541.68329 |
| [4] | Borji, A. (2019). Pros and cons of GAN evaluation measures. Comput. Vis. Image Underst. 179 41-65. |
| [5] | Deheuvels, P. (1986). On the influence of the extremes of an i.i.d. sequence on the maximal spacings. Ann. Probab. 14 194-208. Digital Object Identifier: 10.1214/aop/1176992622 Google Scholar: Lookup Link Euclid: euclid.aop/1176992622 zbMATH: 0594.60029 MathSciNet: MR0815965 · Zbl 0594.60029 · doi:10.1214/aop/1176992622 |
| [6] | Facco, E., d’Errico, M., Rodriguez, A. and Laio, A. (2017). Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Sci. Rep. 7 12140. 10.1038/s41598-017-11873-y |
| [7] | Fefferman, C., Mitter, S. and Narayanan, H. (2016). Testing the manifold hypothesis. J. Amer. Math. Soc. 29 983-1049. 10.1090/jams/852 MathSciNet: MR3522608 · Zbl 1359.62122 |
| [8] | Fournier, N. and Guillin, A. (2015). On the rate of convergence in Wasserstein distance of the empirical measure. Probab. Theory Related Fields 162 707-738. 10.1007/s00440-014-0583-7 MathSciNet: MR3383341 zbMATH: 1325.60042 · Zbl 1325.60042 |
| [9] | Geiß, D., Klein, R., Penninger, R. and Rote, G. (2013). Optimally solving a transportation problem using Voronoi diagrams. Comput. Geom. 46 1009-1016. 10.1016/j.comgeo.2013.05.005 MathSciNet: MR3061462 · Zbl 1282.49039 |
| [10] | Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y. (2014). Generative adversarial nets. In Advances in Neural Information Processing Systems (Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence and K.Q. Weinberger, eds.) 27 2672-2680. Curran Associates. |
| [11] | Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V. and Courville, A.C. (2017). Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems (I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan and R. Garnett, eds.) 30 5767-5777. Curran Associates. |
| [12] | Gulrajani, I., Raffel, C. and Metz, L. (2019). Towards GAN benchmarks which require generalization. In International Conference on Learning Representations. |
| [13] | Hartmann, V. and Schuhmacher, D. (2020). Semi-discrete optimal transport: A solution procedure for the unsquared Euclidean distance case. Math. Methods Oper. Res. 92 133-163. 10.1007/s00186-020-00703-z MathSciNet: MR4152920 · Zbl 1457.65014 |
| [14] | Hur, Y., Guo, W. and Liang, T. (2021). Reversible Gromov-Monge sampler for simulation-based inference. arXiv:2109.14090. · Zbl 07846456 |
| [15] | Kantorovič, L.V. and Rubinšteĭn, G.Š. (1958). On a space of completely additive functions. Vestn. Leningr. Univ. 13 52-59. MathSciNet: MR0102006 · Zbl 0082.11001 |
| [16] | Karras, T., Aila, T., Laine, S. and Lehtinen, J. (2018). Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations. |
| [17] | Karras, T., Laine, S. and Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition 4396-4405. MathSciNet: MR4351661 |
| [18] | Kodali, N., Abernethy, J., Hays, J. and Kira, Z. (2017). On convergence and stability of GANs. Available at arXiv:1705.07215. |
| [19] | Liang, T. (2021). How well generative adversarial networks learn distributions. J. Mach. Learn. Res. 22 Paper No. 228, 41. MathSciNet: MR4329807 · Zbl 07626743 |
| [20] | Lucic, M., Kurach, K., Michalski, M., Gelly, S. and Bousquet, O. (2018). Are GANs created equal? A large-scale study. In Advances in Neural Information Processing Systems (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett, eds.) 31 697-706. Curran Associates. |
| [21] | Luise, G., Pontil, M. and Ciliberto, C. (2020). Generalization properties of optimal transport GANs with latent distribution learning. Available at arXiv:2007.14641. |
| [22] | Mescheder, L., Geiger, A. and Nowozin, S. (2018). Which training methods for GANs do actually converge? In Proceedings of the 35th International Conference on Machine Learning (J. Dy and A. Krause, eds.) 80 3481-3490. PMLR. |
| [23] | Müller, A. (1997). Integral probability metrics and their generating classes of functions. Adv. in Appl. Probab. 29 429-443. 10.2307/1428011 MathSciNet: MR1450938 · Zbl 0890.60011 |
| [24] | Pratelli, A. (2007). On the equality between Monge’s infimum and Kantorovich’s minimum in optimal mass transportation. Ann. Inst. Henri Poincaré Probab. Stat. 43 1-13. 10.1016/j.anihpb.2005.12.001 MathSciNet: MR2288266 · Zbl 1121.49036 |
| [25] | Radford, A., Metz, L. and Chintala, S. (2016). Unsupervised representation learning with deep convolutional generative adversarial networks. In 4th International Conference on Learning Representations (Y. Bengio and Y. LeCun, eds.). |
| [26] | Schreuder, N., Brunel, V.-E. and Dalalyan, A.S. (2021). Statistical guarantees for generative models without domination. In Algorithmic Learning Theory. Proc. Mach. Learn. Res. (PMLR) 132 21. [place of publication not identified]. MathSciNet: MR4227353 |
| [27] | Singh, S., Uppal, A., Li, B., Li, C.L., Zaheer, M. and Poczos, B. (2018). Nonparametric density estimation under adversarial losses. In Advances in Neural Information Processing Systems (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett, eds.) 31 10225-10236. Curran Associates. |
| [28] | Stéphanovitch, A., Tanielian, U., Cadre, B., Klutchnikoff, N. and Biau, G. (2024). Supplement to “Optimal 1-Wasserstein distance for WGANs.” 10.3150/23-BEJ1701SUPP |
| [29] | Tanielian, U., Issenhuth, T., Dohmatob, E. and Mary, J. (2020). Learning disconnected manifolds: A no GAN’s land. In Proceedings of the 37th International Conference on Machine Learning (H. Daumé III and A. Singh, eds.) 119 9418-9427. PMLR. |
| [30] | Uppal, A., Singh, S. and Poczos, B. (2019). Nonparametric density estimation and convergence rates for GANs under Besov IPM losses. In Advances in Neural Information Processing Systems (H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox and R. Garnett, eds.) 32 9089-9100. Curran Associates. |
| [31] | Vaishnavh, N., Raffel, C. and Goodfellow, I.J. (2018). Theoretical insights into memorization in GANs. In Neural Information Processing Systems 2018 - Integration of Deep Learning Theories Workshop. |
| [32] | Villani, C. (2009). Optimal Transport: Old and New. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 338. Berlin: Springer. 10.1007/978-3-540-71050-9 MathSciNet: MR2459454 · Zbl 1156.53003 |
| [33] | Vondrick, C., Pirsiavash, H. and Torralba, A. (2016). Generating videos with scene dynamics. In Advances in Neural Information Processing Systems (D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon and R. Garnett, eds.) 29 613-621. Curran Associates. |
| [34] | Yu, L., Zhang, W., Wang, J. and Yu, Y. (2017). SeqGAN: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence 2852-2858. AAAI Press. |
| [35] | Zhou, Z., Liang, J., Song, Y., Yu, L., Wang, H., Zhang, W., Yu, Y. and Zhang, Z. (2019). Lipschitz generative adversarial nets. In Proceedings of the 36th International Conference on Machine Learning (K. Chaudhuri and R. Salakhutdinov, eds.) 97 7584-7593. PMLR. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
