Some numerical algorithms on Grassmann manifolds (i. e. sets of fixed-dimensional subspaces of a Euclidean space) exhibit a convergence property i. e. if the distance dist\(({\mathcal Y, S})\) between the initial point \({\mathcal Y}\) and the solution point \({\mathcal S}\) is smaller than some given number \(\delta\) then the sequence of iterates generated by the algorithm is guaranteed to converge to the solution \({\mathcal S}\). The problem is to determine the probability that a randomly chosen initial subspace \({\mathcal Y}\) satisfies the distance condition dist\(({\mathcal Y},{\mathcal S}) < \delta\). There are several definitions for dist\(({\mathcal Y},{\mathcal S})\) and the most important one is the projection \(2\)-norm which is related to the largest canonical (or principal) angle. Here, formulas for the probability density function and the corresponding probability distribution function of the largest canonical angle between two subspaces chosen from the uniform distribution on the Grassmann manifold of \(p\)-planes in \(\mathbb{R}^n\) are given.
The derived formulas include the gamma function and require an algorithm that evaluates the hypergeometric function \({}_2F_1\) with matrix argument. Recently, new algorithms were found that approximate the hypergeometric function of matrix argument by its expansion as a series of Jack functions. The MATLAB [The MathWorks, Inc., Natick, MA, MATLAB Reference Guide (1992)] implementations of these algorithms were used and comparisons were made with the probability functions estimated from samples of \(p\)-dimensional subspaces of \(\mathbb{R}^n\). An excellent agreement was achieved between the computed probability functions and those estimated from the sample.