Let \(K\) be an analytic subgroup of a connected, simply connected nilpotent Lie group \(G\). Given a unitary character \(\chi\) of \(K\), the orbit method tells us how to describe the canonical central decomposition of the monomial representation \(\tau = \text{ind}^ G_ K\chi\) [cf. L. Corwin, P. Greenleaf and G. Grélaud: Trans. Am. Math. Soc. 304, 549-583 (1987; Zbl 0629.22005)]. Now the authors continue their study for the case where \(\tau\) has finite multiplicities.
Let \(C^ \infty(G,\tau) = \{\psi\in C^ \infty(G): \psi(kx) = \chi(k)\psi(x)\) for all \(k \in K\), \(g\in C\}\) and consider the algebra \(\mathbb{D}_ \tau(K\setminus G)\) of differential operators on \(G\), taking \(C^ \infty(G,\tau)\) into itself and commuting there with the right action of \(\tau\). Let \(\mathfrak k\) (resp. \(\mathfrak g\)) be the Lie algebra of \(K\) (resp. \(G\)). It follows that there is an \(f\in {\mathfrak g}^*\) such that \(\chi(\text{exp }Y) = e^{2\pi if(Y)}\) for all \(Y\in {\mathfrak k}\). Suppose that \(\tau\) has finite multiplicities. Then the authors showed [in Commun. Pure Appl. Math. 45, 681-748 (1992)] the commutativity of \(\mathbb{D}_ \tau(K\setminus G)\), which was in fact isomorphic to a generating subalgebra of the field \(\mathbb{C}(\Omega_ \tau)^ K\) of \(\text{Ad}^*(K)\)-invariant rational functions on \(\Omega_ \tau = f+{\mathfrak k}^ \perp\subset {\mathfrak g}^*\). In this paper, they assume the existence of a subalgebra which polarizes generic \(\xi \in \Omega_ \tau\) and is normalized by \(\mathfrak k\). This condition enables them to take a suitable Malcev basis of \(\mathfrak g\), and to analyze the corresponding Malcev-Fourier transform in order to get the following main results. \(\mathbb{D}_ \tau(K\setminus G)\) is isomorphic to the algebra \(\mathbb{C}[\Omega_ \tau]^ K\) of \(\text{Ad}^*(K)\)-invariant polynomials on \(\Omega_ \tau\). Furthermore every nonzero element of \(\mathbb{D}_ \tau(K\setminus G)\) has a tempered fundamental solution. The paper includes various examples which are explicitly calculated.