Let \(R\) denote a domain in the complex plane with boundary \(B\) and closure \(X\). Let \(T\) be an operator on a complex Hilbert space \(\mathcal H\). If \(T\) has a normal \(B\)-dilation (i.e., there exist a Hilbert space \({\mathcal K}\supset{\mathcal H}\) and a normal operator \(N\) on \(\mathcal K\) such that \(f(T)=P_{\mathcal H}f(N)| _{\mathcal H}\) for every rational function \(f\) with poles off \(X\); \(P_{\mathcal H}\) denotes the orthogonal projection of \(\mathcal K\) onto \(\mathcal H\)), then \(X\) is a spectral set for \(T\) (i.e., the spectrum of \(T\) is contained in \(X\) and \(\| f(T)\| \leq\| f\| _R=\sup_{z\in R}| f(z)| \) for every rational function \(f\) with poles off \(X\)). As regarding the converse, J. Agler, J. Harland and B. Raphael [“Classical function theory, operator dilation theory and machine computations on multiply-connected domains” (per bibl.)] have recently given an example of a \(4\times 4\) matrix \(T\) and of a triply connected domain \(R\) such that its closure \(X\) is a spectral set for \(T\), but \(T\) does not have a normal \(B\)-dilation.
The main result of the paper under review shows that for any bounded triply connected domain \(B\) with analytic disjoint boundary components, there exists an operator \(T\) such that \(X\) (the boundary of \(B\)) is a spectral set for \(T\) and \(T\) does not have a normal \(B\)-dilation. The proof employs certain auxiliary techniques and results, which are of independent interest. A key role is played by the theory of minimal inner functions in connection to harmonic functions and analytic functions with positive real part [cf.S. D.Fisher, “Function theory on planar domains.A second course in complex analysis” (Pure and Appl.Math., Wiley & Sons Inc., New York) (1983; Zbl 0511.30022); H. Grunsky, “Lectures on theory of functions in multiply connected domains” (Stud.Math., Skript 4, Vandenhoeck & Ruprecht, Götingen) (1978; Zbl 0371.30015)]. We could also mention in this context the representations, in terms of theta functions, for reproducing Szegő kernels on \(R\) obtained by J. D.Fay [“Theta functions on Riemann surfaces” (Lect.Notes Math.352, Springer-Verlag, Berlin), (1973; Zbl 0281.30013)] and J. A.Ball and K. F.Clancey [Integral Equations Oper.Theory 25, 35–57 (1996; Zbl 0867.30038)]. In order to prove the existence of a finite-dimensional example, the authors use some arguments due to V. I.Paulsen in the last section of the paper.