The paper studies limiting configurations of anti-self-dual \(SO(3)\) instantons on \(M\times\mathbb{R}\), where \(M\) is a \(3\)-manifold with a cylindrical end, i.e. isometric to \(\Sigma\times[0,\infty)\) for a surface \(\Sigma\). In this case, sequences of instantons in the limit may bubble off instantons on \(\mathbb{R}^4\), on \(\Sigma\times\mathbb{C}\), and even holomorphic spheres and disks, when energy escapes to infinity. The limiting configuration can be described by an explicit combinatorial object (a bubble tree), which the author calls stable scaled instanton. The main result is a Gromov-Uhlenbeck type compactness theorem: under a regularity condition on \(M\), a sequence of anti-self-dual instantons with bounded energy has a subsequence converging to a stable scaled instanton.
The regularity condition is that the natural map from the moduli space of flat connections on \(M\) to the moduli space of flat connections \(R_\Sigma\) on \(\Sigma\) is an immersion with transverse double points. The condition is borrowed from Fukaya’s conjecture [K. Fukaya, Jpn. J. Math. (3) 13, No. 1, 1–65 (2018; Zbl 1394.57030)], which asserts existence of a suitable bounding cochain of the \(A_\infty\)-algebra associated to the (Lagrangian) submanifold of \(R_\Sigma\) immersed by the above map. Indeed, the author sees this work as a first step towards proving the conjecture. In its turn, Fukaya’s conjecture is a building block for proving the \(SO(3)\) Atiyah-Floer conjecture: there is an isomorphism between the instanton Floer homology of a \(3\)-manifold and the Lagrangian intersection Floer homology of its splitting by a surface. The latter homology requires Fukaya’s bounding cochain to be defined. The author also suggests that a modification of his argument will lead to a compactness theorem for the neck-stretching limit in the \(SO(3)\) instanton equation. This is the limit Atiyah used in his original heuristic argument supporting the Atiyah-Floer conjecture.
The proof is based on combining the isoperimetric inequality, the annulus lemma, and the boundary diameter estimate. The latter is of interest in its own right, and may lead to a simplified proof of compactness for the strip-shrinking limit of pseudo-holomorphic quilts, as the author suggests. The treatment of the compactness problem near the “boundary” of \(M\times\mathbb{R}\) is one of the main novelties of the paper.