Let G be a separable locally compact group and K a compact group of automorphisms of G. This paper is concerned with the question of when the subalgebra \(L^ 1_ K(G)\) of all K-invariant functions in \(L^ 1(G)\) is commutative. \(L^ 1_ K(G)\) is isomorphic to the algebra of all K- biinvariant functions in the \(L^ 1\)-algebra of the semi-direct product \(H=K\rtimes G\). Hence \(L^ 1_ K(G)\) is commutative if and only if (H,K) is a Gelfand pair, and it is well known that (H,K) is a Gelfand pair precisely when the trivial representation \(1_ K\) of K occurs at most once in the restriction \(\rho| K\) for every \(\rho\in \hat H\), the dual space of H.
The author employs Mackey’s theory of extending group representations to find an equivalent condition in terms of the multiplier representations of the stability subgroups \(K_{\pi}\) associated to \(\pi\in \hat G\). However, for his main result to make sense at all, he has to assume that all the stability groups \(K_{\pi}\), \(\pi\in \hat G\), are closed in K. This is known to be true if G is type I and to be false in general (compare Lemma 1 and the first example by R. J. Blattner [Pac. J. Math. 15, 1101-1113 (1965; Zbl 0143.362)]).
The author studies two interesting examples that are not affected by the mistake just mentioned. The first of them is related to Siegel domains and is defined to be a certain semi-direct product of \(K=SU(p)\times U(q)\) and a 2-step nilpotent simply connected Lie group \(G_{pq}\) whose center is the vector space \(H_ p\) of all complex hermitian matrices of order p and for which \(G_{pq}/H_ p=M_{pq}\), the vector space of all \(p\times q\) complex matrices (p\(\leq q)\). It is shown that \(L^ 1_ K(G_{pq})\) is commutative if and only if \(p\leq 2\).