Summary: The invariants in \(D = 4\), \(\mathcal{N} = 4\) supergravity are discussed up to the three-loop order (where one expects a general \(R^4\) structure). Because there is an anomaly in the rigid \(\mathrm{SL}(2, \mathbb{R})\) symmetry of this theory, the analysis of possible restrictions on three-loop divergences due to duality needs careful treatment. We show that this anomalous symmetry is still strong enough at the three-loop order to require duality invariance of candidate counterterms. Provided one makes the additional assumption that there exists a full 16-supercharge off-shell formulation of the theory, counterterms at \(L \geqslant 2\) loops would also have to be writable as full-superspace integrals. At the three-loop order such a duality-invariant full-superspace integral candidate counterterm exists, but its duality invariance is marginal in the sense that the full-superspace counter-Lagrangian is not itself duality-invariant. We show that, subject to the assumption that a full off-shell quantisation formalism exists, such marginal invariants are not allowable as counterterms.