An automorphism of a group \(G\) is said to be a power automorphism if it maps every subgroup of \(G\) onto itself. The set \(\text{PAut }G\) of all power automorphisms of \(G\) is an Abelian normal subgroup of the full automorphism group \(\operatorname{Aut} G\) of \(G\). An automorphism \(\alpha\) of \(G\) is called an I-automorphism if \(H^\alpha=H\) for each infinite subgroup \(H\) of \(G\). The structure of the group \(\text{IAut }G\) consisting of all I-automorphisms of the group \(G\) has been investigated by M. Curzio, S. Franciosi and the reviewer [Arch. Math. 54, No. 1, 4-13 (1990; Zbl 0664.20018)]. In this article the authors study groups \(G\) for which \(\text{IAut }G\neq\text{PAut }G\), proving that such groups must satisfy strong structural restrictions.