Summary: For a finite group \(G\), let \(Z(G)\) denote the center of \(G\) and \(\operatorname{cs}^\ast (G)\) be the set of non-trivial conjugacy class sizes of \(G\). In this paper, we show that if \(G\) is a finite group such that for some odd prime power \(q\geq 4\), \(\operatorname{cs}^\ast (G)=\operatorname{cs}^\ast (\mathrm{PGL}_2(q))\), then either \(G \cong \mathrm{PGL}_2(q) \times Z(G)\) or \(G\) contains a normal subgroup \(N\) and a non-trivial element \(t\in G\) such that \(N \cong \mathrm{PSL}_2(q) \times Z(G)\), \(t^2 \in N\) and \(G=N.\langle t\rangle\). This shows that the almost simple groups cannot be determined by their set of conjugacy class sizes (up to an abelian direct factor).